huggingface/text-generation-inference

Pull Requests Commits

Choosing input/total tokens automatically based on available VRAM? (#2673)

Narsil committed 1 year ago

Verified 0c9b6cdd

Green main (#2697)

Narsil committed 1 year ago

Verified 2e4f4ba1

Avoiding timeout for bloom tests. (#2693)

Narsil committed 1 year ago

Verified 8a8794a6

chore: prepare 2.4.0 release (#2695)

OlivierDehaene committed 1 year ago

Verified a6b02da9

feat: add triton kernels to decrease latency of large batches (#2687)

OlivierDehaene committed 1 year ago

Verified 6f88bd93

Switch from fbgemm-gpu w8a8 scaled matmul to vLLM/marlin-kernels (#2688)

danieldk committed 1 year ago

Verified 0f346a32

Add support for stop words in TRTLLM (#2678)

mfuntowicz committed 1 year ago

Verified ba5fc7d9

Fixing mt0 test. (#2692)

Narsil committed 1 year ago

Verified db68bd05

Fixing rocm gptq by using triton code too (renamed cuda into triton). (#2691)

Narsil committed 1 year ago

Verified cece8635

[TENSORRT-LLM] - Implement new looper thread based backend (#2357)

mfuntowicz committed 1 year ago

Verified 43df056e

Fixing "deadlock" when python prompts for trust_remote_code by always (#2664)

Narsil committed 1 year ago

Verified ed87b464

Add support for FP8 KV cache scales (#2628)

danieldk committed 1 year ago

Verified eab07f74

Fix Phi 3.5 MoE tests (#2684)

danieldk committed 1 year ago

Verified 14a0df3a

flashinfer: reminder to remove contiguous call in the future (#2685)

danieldk committed 1 year ago

Verified 1b914f37

feat: allow any supported payload on /invocations (#2683)

OlivierDehaene committed 1 year ago

Verified 41c26237

hotfix: fix flashllama

OlivierDehaene committed 1 year ago

Verified 27ff1871

feat: natively support Granite models (#2682)

OlivierDehaene committed 1 year ago

Verified 03c9388b

Make moe-kernels and marlin-kernels mandatory in CUDA installs (#2632)

danieldk committed 1 year ago

Verified f58eb70e

Add `impureWithCuda` dev shell (#2677)

danieldk committed 1 year ago

Verified 9c9ef37c

break when there's nothing to read (#2582)

sywangyi committed 1 year ago

Verified 058d3061

Test Marlin MoE with `desc_act=true` (#2622)

danieldk committed 1 year ago

Verified 7f54b733

Make handling of FP8 scales more consisent (#2666)

danieldk committed 1 year ago

Verified 5e0fb468

CI job. Gpt awq 4 (#2665)

Narsil committed 1 year ago

Verified 153ff374

Break cycle between the attention implementations and KV cache (#2627)

danieldk committed 1 year ago

Verified 8ec57558

fix: prefer inplace softmax to avoid copy (#2661)

drbh committed 1 year ago

Verified 5f32dea1

fix tgi-entrypoint wrapper in docker file: exec instead of spawning a child process (#2663)

oOraph committed 1 year ago

Verified 1b97e084

Simplify the `attention` function (#2609)

danieldk committed 1 year ago

Verified 59ea38cb

Support `e4m3fn` KV cache (#2655)

danieldk committed 1 year ago

Verified 5bbe1ce0

feat: prefill chunking (#2600)

OlivierDehaene committed 1 year ago

Verified a6a0c97e

Fp8 e4m3_fnuz support for rocm (#2588)

mht-sharma committed 1 year ago

Verified 704a58c8

Newer Older