PR #48677 🚨Add an FP8 backend to `QuantizedCache`

🚨Add an FP8 backend to `QuantizedCache` #48677

vasqu merged 21 commits into huggingface:main from jiqing-feng:fp8-kv-cache

Add an FP8 backend to `QuantizedCache`

9789b126

Reorder the FP8 states for beam search

64cfc5d5

Use an early return in `Fp8QuantizedLayer.reorder_cache`

87ab14a5

vasqu requested a review from

vasqu 46 days ago

vasqu requested a review from

zucchini-nlp 46 days ago

Measure the quantized cache backends accurately

a7cc8e34

Merge branch 'main' into fp8-kv-cache

8882b422

Merge branch 'main' into fp8-kv-cache

2b7a9eb5

Reset the FP8 scales, and cover the FP8 backend in the quantized laye…

bc126c1e

jiqing-feng marked this pull request as ready for review 21 days ago

Merge branch 'main' into fp8-kv-cache

5eaeb32f

Merge branch 'main' into fp8-kv-cache

4fc4f00b

IlyasMoutawwakil commented on 2026-10-15

Address review on the FP8 quantized cache layer

4743c39f

Trim the comments down to one line

e9e889db

IlyasMoutawwakil commented on 2026-10-16

IlyasMoutawwakil approved these changes on 2026-10-16

Store the FP8 scale alongside the states it quantizes

05ef3728

Say what the `.contiguous()` in `lazy_initialization` is actually for

bc31eda6

vasqu commented on 2026-10-16

Address @vasqu's review

ed19267d

One-line the docstrings and comments that fit on one line

7018cc41

Merge branch 'main' into fp8-kv-cache

862a6436

Keep the FP8 state initialization out of the shared quantized layer

b4e172bc

vasqu commented on 2026-10-19

Split the quantized layer into a base and a group-wise class

02dc2e88

Note which subtree a future fp8 variation belongs to

b138c839

vasqu commented on 2026-10-20

jiqing-feng changed the title ~~Add an FP8 backend to `QuantizedCache`~~ 🚨Add an FP8 backend to `QuantizedCache` 6 days ago

Keep the group-wise signature on QuantizedLayer and name the new base

780cc9d4

vasqu approved these changes on 2026-10-21

Declare the quantized states in the base layer, and group the quantiz…

08c5bc7b

vasqu approved these changes on 2026-10-22

vasqu enabled auto-merge 5 days ago

vasqu merged 43cc0abd into main 5 days ago

Reviewers

vasqu

IlyasMoutawwakil

zucchini-nlp

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

transformers 🚨Add an FP8 backend to `QuantizedCache` #48677 Merged

🚨Add an FP8 backend to `QuantizedCache` #48677

transformers
🚨Add an FP8 backend to `QuantizedCache`
#48677

Merged