[Spec Decode] Add hidden states extraction system #33736
LucasWilkinson
merged
26 commits into
vllm-project:main
from
fynnsu:extract_hidden_states
gemini-code-assist
commented
on 2026-02-03
benchislett
commented
on 2026-02-05
benchislett
commented
on 2026-02-25
benchislett
commented
on 2026-02-25
benchislett
commented
on 2026-02-25
mgoin
approved these changes
on 2026-02-26
Add extract_hidden_states speculation method
27bad969
Add ExampleHiddenStatesConnector
806715ec
Cleanup / simplify components
ead1f8b2
Add example usage
ac5d6ae8
Fix shape issues
d1ebb82f
Add safetensors output to example script
e1a01ba3
Update example script
b9ab5cfb
Small fixes to ExtractHiddenStatesProposer
f332456f
Simplify custom attention components
1c396456
Update ExtractHiddenStatesModel for simpler CacheOnlyAttentionLayer
ab3181c2
Clean up ExampleHiddenStatesConnector
f04dcf6c
Add basic kv cache insertion function
ab1ec95f
Clean up ExtractHiddenStatesProposer implementation
b3e64b9d
Fix small issues
2e3b4f40
Add support for merging KVConnectorOutput
294d8426
Improve config handling
1ad24df4
Fix precommit issues
a624b724
Cleanup todos
28aa17a1
precommit
45e7e4a8
Fix docs issues
7c53b80e
Add tests for extract_hidden_states
ef326ad5
Fix ExampleHiddenStatesConnector handling of batched prefill
88d2fb47
Handle review comments
76386685
lint
06396f53
Add ExtractHiddenStatesModel to test model registry
4806c959
Update computation of padded tokens in extract_hidden_states
22682246
Reviewers
Assignees
No one assigned
Labels
documentation
new-model
speculative-decoding
ready
v1
llama
kv-connector
Milestone
No milestone
Login to write a write a comment.
Login via GitHub