huggingface/text-generation-inference

Pull Requests Commits

Fixing exl2 (by disabling cuda graphs)

Narsil committed 1 year ago

Verified 89707adb

Narsil committed 1 year ago

Verified b2d3948c

Fixing the other pathways.

Narsil committed 1 year ago

Verified 5c598cc7

Upgrading exl2.

Narsil committed 1 year ago

Verified c9047667

nix: partial incremental build of the router (#2416)

danieldk committed 1 year ago

Verified c5fff92b

fix: adds causal to attention params (#2408)

drbh committed 1 year ago

Verified 1cebccc7

add numa to improve cpu inference perf (#2330)

sywangyi committed 1 year ago

Verified 59922f9b

Adding more kernels to flake. (#2411)

Narsil committed 1 year ago

Verified cd9b15d1

nix: incremental build of the launcher (#2410)

danieldk committed 1 year ago

Verified 6f4bb4f2

fix: include create_exllama_buffers and set_device for exllama (#2407)

drbh committed 1 year ago

Verified 8a7749b8

Pr 2395 ci run (#2406)

drbh committed 1 year ago

Verified 9a7830bd

Updating the flake. (#2404)

Narsil committed 1 year ago

Verified 19ea85f8

fix: improve completions to send a final chunk with usage details (#2336)

drbh committed 1 year ago

Verified 30395b09

fix: allocate tmp based on sgmv kernel if available (#2345)

drbh committed 1 year ago

Verified 4c3f8a70

feat: validate template variables before apply and improve sliding wi… (#2403)

drbh committed 1 year ago

Verified 155f9c98

Keeping the benchmark somewhere (#2401)

Narsil committed 1 year ago

Verified 136bcc81

Add support for prefix caching to the v3 router (#2392)

danieldk committed 1 year ago

Verified 8deeaca4

Cpu dockerimage (#2367)

sywangyi committed 1 year ago

Verified b6bb1d51

Fixing import exl2 (#2399)

Narsil committed 1 year ago

Verified 84bc3d7b

Adding launcher to build. (#2397)

Narsil committed 1 year ago

Verified 730fa00e

Upgrade fbgemm (#2398)

Narsil committed 1 year ago

Verified 9c739651

nix: add router to the devshell (#2396)

danieldk committed 1 year ago

Verified 01a515de

Update flake for 9.0a capability in Torch (#2394)

danieldk committed 1 year ago

Verified 8dcc7d3f

feat: add guideline to chat request and template (#2391)

drbh committed 1 year ago

Verified 0d06aed0

Using an enum for flash backens (paged/flashdecoding/flashinfer) (#2385)

Narsil committed 1 year ago

Verified 7a48a847

flake: use rust-overlay (#2390)

danieldk committed 1 year ago

Verified 6e127dcc

Update documentation for Supported models (#2386)

Vaibhavs10 committed 1 year ago

Verified b2b9c427

flake: add fmt and clippy (#2389)

danieldk committed 1 year ago

Verified 977534bc

Using HF_HOME instead of CACHE to get token read in addition to models. (#2288)

Narsil committed 1 year ago

Verified 952b450a

Add experimental flake (#2384)

danieldk committed 1 year ago

Verified c6d5039c

Older