huggingface/text-generation-inference

Pull Requests Commits

Keeping the benchmark somewhere

danieldk committed 1 year ago

f230da8d

Test partially overlapping prefills.

danieldk committed 1 year ago

7865851c

Initial radix cache tests

danieldk committed 1 year ago

2a255ad7

Add radix cache free, improve allocate

danieldk committed 1 year ago

6486887b

First radix allocation bits

danieldk committed 1 year ago

9415b908

danieldk committed 1 year ago

07ede8d8

Block/node eviction

danieldk committed 1 year ago

5da69604

Update parent refcounts when inserting a child

danieldk committed 1 year ago

ce76f4cc

danieldk committed 1 year ago

aa1c96a7

danieldk committed 1 year ago

9caee1f3

Double linked data structures are still terrible in Rust.

danieldk committed 1 year ago

590fc2c5

Trie insertion/lookup

danieldk committed 1 year ago

cd4933cd

danieldk committed 1 year ago

743d3781

Port over block allocator interface (with token ids)

danieldk committed 1 year ago

5d482d4d

Use a block size of 1 for FlashInfer

danieldk committed 1 year ago

4562c160

Add FlashInfer support

danieldk committed 1 year ago

8fb8e1da

Unify attention output handling

danieldk committed 1 year ago

fe41e13b

Fix cache block size for flash decoding (#2351)

danieldk committed 1 year ago

Verified 22fb1be5

enable HuggingFaceM4/idefics-9b in intel gpu (#2338)

sywangyi committed 1 year ago

Verified 9ab99374

refactor usage stats (#2339)

ErikKaum committed 1 year ago

Verified 7451041e

Pr 2290 ci run (#2329)

drbh committed 1 year ago

Verified f7f61876

Handle GPTQ-Marlin loading in `GPTQMarlinWeightLoader` (#2300)

danieldk committed 1 year ago

Verified 34f7dcfd

Rebase TRT-llm (#2331)

Narsil committed 1 year ago

Verified 2b19d671

server quantize: store quantizer config in standard format (#2299)

danieldk committed 1 year ago

Verified 53aec273

fix: adjust test snapshots and small refactors (#2323)

drbh committed 1 year ago

Verified 0b95693f

patch-error-on-invalid-grammar (#2282)

ErikKaum committed 1 year ago

Verified 3d7f4f41

fix: reject grammars without properties (#2309)

drbh committed 1 year ago

Verified f15e808d

Install Marlin from standalone package (#2320)

danieldk committed 1 year ago

Verified 922732b2

Run ci api key (#2315)

ErikKaum committed 1 year ago

Verified 583d37a2

fix: fix buildkit config in ci

XciD committed 1 year ago

Verified fd2e0631

Older