vllm
[KV Connector] Generalize declarative connector stats (TypedKVConnectorStats)
#55969
Open
Overview
Commits
75
Changes
View On GitHub

[KV Connector] Generalize declarative connector stats (TypedKVConnectorStats) #55969

LucasWilkinson wants to merge 75 commits into main from feat/kv-connector-typed-stats
LucasWilkinson
MatthewBonanni Add HiSparse host-resident sparse MLA decode
a6fc69f8
MatthewBonanni Fix slot mapping for HiSparse hot cache groups
8ba010b4
MatthewBonanni Compose HiSparse with generic indexer offloading
3d6bb7ef
MatthewBonanni Align HiSparse with KV connector lifecycle
769fd192
MatthewBonanni Fix NIXL push worker shutdown signature
f09c1328
MatthewBonanni Move HiSparse connector beside KV connectors
3a28c504
MatthewBonanni Initialize HiSparse through connector registration
a0ff900f
MatthewBonanni [HiSparse] Compose with adaptive speculative decoding
ab1384e4
MatthewBonanni [HiSparse] Remove decode routing mirror test
e562b11f
MatthewBonanni [HiSparse] Import Triton runtime for cache invalidation
22fb0ffa
MatthewBonanni Replace HiSparse invalidation Triton kernel
ed938bd9
MatthewBonanni [HiSparse] Decouple attention warmup from model dummy runs
4ffd8795
MatthewBonanni Release HiSparse cache after CUDA graph profiling
71dad3f5
MatthewBonanni Fix HiSparse mixed single-token warmup
64ec39d4
MatthewBonanni Refresh HiSparse state during graph replay
f30e6cec
MatthewBonanni Expose GSM8K request timeout
21d4347e
MatthewBonanni Adapt HiSparse to current attention initialization
4d22696e
MatthewBonanni Update HiSparse replay test for masked MHA arguments
fe73d562
MatthewBonanni Cap HiSparse resident pages during full-ISL admission
9093e804
MatthewBonanni Keep scheduler alive for pending HiSparse spills
5bcf75c1
MatthewBonanni Resolve HiSparse conflicts with current main
4e30b3f5
LucasWilkinson Refresh HiSparse batch state before KV cache update
6ebd8183
LucasWilkinson Fix HiSparse KV corruption from aliased fused_norm_rope buffers
d7412f65
LucasWilkinson fix(hisparse): defer hot allocation to scheduler
58878bba
LucasWilkinson fix(hisparse): refresh capacity after resident reclaim
3efd1909
LucasWilkinson [HiSparse] Serve prefix-cache hits from GPU-resident shadow pages
1c9d137e
MatthewBonanni fix(hisparse): avoid eager mirrors for local offload
955d7377
MatthewBonanni perf(hisparse): batch host mirrors after forward
956ace47
MatthewBonanni perf(hisparse): use one TP writer for shared host cache
95e78b42
MatthewBonanni fix(hisparse): preserve host KV across aggregated prefills
eb7ab9ef
MatthewBonanni perf(hisparse): eliminate prefill host syncs
5feb2765
LucasWilkinson perf(hisparse): pipeline sparse MLA offload with DMA
a0d516ad
LucasWilkinson refactor(hisparse): write sparse indices request-major
108f2f18
LucasWilkinson fix
4eadde99
MatthewBonanni refactor(hisparse): use strided request-major views
d7d70e16
MatthewBonanni refactor(hisparse): remove mirror destination synchronization
fbe4ccd7
MatthewBonanni fix(hisparse): preserve host-only connector imports
609f057d
MatthewBonanni fix(hisparse): map speculative rows through request state
a01a959d
MatthewBonanni fix(hisparse): make async MTP host spills durable
24a32b7b
MatthewBonanni fix(hisparse): mirror async MTP speculative horizon
abcc83eb
MatthewBonanni fix(hisparse): mirror exact async MTP writes
78297b3e
MatthewBonanni refactor(hisparse): simplify exact MTP mirroring
d6ea30c6
MatthewBonanni refactor(hisparse): unify host mirror planning
13825f10
MatthewBonanni fix(hisparse): preserve async speculation metadata
58493cfe
MatthewBonanni fix(hisparse): mirror async draft KV writes
82efc454
LucasWilkinson fix(hisparse): resolve row mirrors from the resident slot mapping
a88d2bd3
MatthewBonanni fix(hisparse): isolate padded decode residency state
7a240b6f
MatthewBonanni test(hisparse): align DMA tests with resident mirrors
076fb796
MatthewBonanni Simplify HiSparse P/D host landing
6e75000f
MatthewBonanni Rebase HiSparse MLA decode onto current stack
bd8bb1a9
MatthewBonanni Coordinate NIXL transfer completion across workers
ea56bbdc
MatthewBonanni Fix HiSparse cache pool bookkeeping
cb0ea830
MatthewBonanni Harden HiSparse attention warmup paths
20672a3f
MatthewBonanni Exclude HiSparse auxiliary groups from prefix matching
1af088df
MatthewBonanni Bind HiSparse step state before forward
c4402b86
MatthewBonanni Submit HiSparse spills after each forward
5ecad9d0
MatthewBonanni Prune redundant HiSparse tests
f3888e5a
MatthewBonanni Fix HiSparse import capacity test setup
2a5cac67
MatthewBonanni Align HiSparse import capacity expectation
71566c96
MatthewBonanni Fix runner selection test fixture
6049665f
MatthewBonanni Complete runner selection test fixtures
5a6997fe
MatthewBonanni Drop inapplicable offloading layout case
29de40cb
MatthewBonanni Update sparse FlashInfer scale test fixture
0f3e7cdf
NickLucche Remove redundant NIXL block size initialization
e9e80ca1
MatthewBonanni Simplify NIXL receive failure accounting
f636fe95
MatthewBonanni Clean up HiSparse connector lifecycle
e420a7eb
MatthewBonanni Require HMA for HiSparse
3104762d
MatthewBonanni Restore mixed-memory descriptor cache initialization
e07675d5
MatthewBonanni Update FlashInfer warmup test for HiSparse
03851418
MatthewBonanni Initialize shared NIXL region test mapping
0394efd3
MatthewBonanni Propagate replay boundaries through HiSparse caching
cb443699
MatthewBonanni Fix HiSparse connector CI fixtures
9084aaee
MatthewBonanni Fix MLA cache spec test expectations
a46bd886
MatthewBonanni Fix HiSparse CUDA and E2E test fixtures
f3d4fa23
LucasWilkinson [KV Connector] Generalize declarative connector stats
51162f42
mergify mergify added kv-connector
mergify
mergify mergify added needs-rebase
Base automatically changed from feat/hisparse-mla-decode-main to main 10 days ago
mergify
mergify mergify added documentation
mergify mergify added ci/build
mergify mergify added deepseek
mergify mergify added glm
mergify mergify added minimax
mergify mergify added nvidia
mergify mergify added cpu
mergify mergify added speculative-decoding
mergify mergify added dflash
mergify mergify added mrv2
mergify mergify added scheduler
mergify mergify added kv-cache-manager
mergify mergify added torch.compile

Login to write a write a comment.

Login via GitHub

Reviewers
No reviews
Assignees
No one assigned
Labels
documentation speculative-decoding torch.compile needs-rebase ci/build deepseek cpu kv-connector nvidia mrv2 dflash scheduler kv-cache-manager glm minimax
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum