huggingface/text-generation-inference

Pull Requests Commits

fix: also show total memory after full warmup

drbh committed 1 year ago

e152cb02

fix: skip cuda graphs that will oom and improve free memory logging

drbh committed 1 year ago

8b4cd2a9

nix: add awq-inference-engine as server dependency (#2442)

danieldk committed 1 year ago

Verified 358ceb67

Adding eetq to flake. (#2438)

Narsil committed 1 year ago

Verified 310778e0

nix: add `text-generation-benchmark` to pure devshell (#2431)

danieldk committed 1 year ago

Verified 94744150

nix: add pure server to flake, add both pure and impure devshells (#2430)

danieldk committed 1 year ago

Verified f5f11b79

Prefix caching (#2402)

Narsil committed 1 year ago

Verified b70ae096

nix: update to CUDA 12.4 (#2429)

danieldk committed 1 year ago

Verified 38773453

All integration tests back everywhere (too many failed CI). (#2428)

Narsil committed 1 year ago

Verified e4201f44

doc: Add metrics documentation and add a 'Reference' section (#2230)

Hugoch committed 1 year ago

Verified 53729b74

Narsil committed 1 year ago

Verified cb0a2948

Further fixes. (#2426)

Narsil committed 1 year ago

Verified c7ab1810

Improve the Consuming TGI + Streaming docs. (#2412)

Vaibhavs10 committed 1 year ago

Verified 99b662f8

nix: try to reduce the number of Rust rebuilds (#2424)

danieldk committed 1 year ago

Verified 1411bfb9

Upgrading the tests to match the current workings. (#2423)

Narsil committed 1 year ago

Verified 1b0aa062

Fixing exl2 and other quanize tests again. (#2419)

Narsil committed 1 year ago

Verified 57b34958

nix: build router incrementally (#2422)

danieldk committed 1 year ago

Verified 9aaa12e7

More fixes trtllm (#2342)

mfuntowicz committed 1 year ago

Verified 3f385991

Upgrading exl2. (#2415)

Narsil committed 1 year ago

Verified f3b5c694

nix: partial incremental build of the router (#2416)

danieldk committed 1 year ago

Verified c5fff92b

fix: adds causal to attention params (#2408)

drbh committed 1 year ago

Verified 1cebccc7

add numa to improve cpu inference perf (#2330)

sywangyi committed 1 year ago

Verified 59922f9b

Adding more kernels to flake. (#2411)

Narsil committed 1 year ago

Verified cd9b15d1

nix: incremental build of the launcher (#2410)

danieldk committed 1 year ago

Verified 6f4bb4f2

fix: include create_exllama_buffers and set_device for exllama (#2407)

drbh committed 1 year ago

Verified 8a7749b8

Pr 2395 ci run (#2406)

drbh committed 1 year ago

Verified 9a7830bd

Updating the flake. (#2404)

Narsil committed 1 year ago

Verified 19ea85f8

fix: improve completions to send a final chunk with usage details (#2336)

drbh committed 1 year ago

Verified 30395b09

fix: allocate tmp based on sgmv kernel if available (#2345)

drbh committed 1 year ago

Verified 4c3f8a70

feat: validate template variables before apply and improve sliding wi… (#2403)

drbh committed 1 year ago

Verified 155f9c98

Older