DeepSpeed
feat(rollout): segment-based kernel injection for hybrid-engine greedy decode
#8602
Open

feat(rollout): segment-based kernel injection for hybrid-engine greedy decode #8602

delock wants to merge 59 commits into master from gma/qwen_ki_expr
delock
delock delock force pushed from f1f9beb4 to b67e55c5 54 days ago
delock delock force pushed from 11231a7f to d41c9149 53 days ago
delock delock changed the title feat(rollout): HybridEngineRollout graph capture + segment-KI for Qwen3.5 hybrid — 67 tok/s (1.14× of vLLM) feat(rollout): segment-based kernel injection for hybrid-engine greedy decode 53 days ago
delock delock marked this pull request as ready for review 51 days ago
delock delock requested a review from tjruwase tjruwase 51 days ago
delock delock requested a review from tohtana tohtana 51 days ago
delock delock requested a review from hwchen2017 hwchen2017 51 days ago
delock delock requested a review from loadams loadams 51 days ago
chatgpt-codex-connector
chatgpt-codex-connector commented on 2026-09-28
hwchen2017 hwchen2017 requested a review from copilot-pull-request-reviewer copilot-pull-request-reviewer 51 days ago
copilot-pull-request-reviewer
copilot-pull-request-reviewer commented on 2026-09-28
nathon-lee
hwchen2017 hwchen2017 requested a review from copilot-pull-request-reviewer copilot-pull-request-reviewer 42 days ago
copilot-pull-request-reviewer
copilot-pull-request-reviewer commented on 2026-10-07
delock
nathon-lee
delock
nathon-lee
delock delock marked this pull request as draft 31 days ago
delock delock marked this pull request as ready for review 28 days ago
chatgpt-codex-connector
delock
delock
phoeenniixx
phoeenniixx commented on 2026-10-25
phoeenniixx
delock
delock perf(rollout): TP2 decode optimizations — GLU combo, custom-AR handsh…
0a6a0c55
delock fix: CI compliance — arch guards, hook literals, mock-safe guard, scr…
4bcc86b8
delock delock force pushed from 1811118c to 4bcc86b8 17 days ago
delock fix: CI compliance — remove torch.distributed import, fix ignore-cuda…
9e7e8dd3
delock delock force pushed from b0c0c441 to 9e7e8dd3 17 days ago
delock
delock feat(segki): arena/reshard handshake — segki_refresh drops derived-we…
9cc11f5a
delock fix: import RolloutRequest in the arena tests
8b811a03
delock test: ZeRO-3 trainer -> arena-sync -> segKI rollout end-to-end
4e384a20
delock docs: mark the refresh_weights contract provisional pending final han…
8111716c
delock
delock fix: gate arena tests' graph capture on real accelerators; CI compliance
bd736d3d
delock fix: route the arena e2e script through get_accelerator() instead of …
e3a46527
delock chore: drop dead accelerator graph-capture probes; use DeepSpeed lice…
5f44d619
delock
delock refactor(comm): adopt the custom_collective from #8997, drop the loca…
11525511
delock test: reference anchors for the post-original glue kernels
cef3ec90
delock refactor: drop the compile integration — segKI is hand-written kernels
47d426d9
anyadontfly
anyadontfly
anyadontfly commented on 2026-11-04
anyadontfly
anyadontfly commented on 2026-11-04
anyadontfly
anyadontfly commented on 2026-11-04
delock revert: restore the TP dataloader consistency check in engine.py
d154866c
delock Merge remote-tracking branch 'origin/master' into gma/qwen_ki_expr
1c7717ed
delock refactor(interface): all Python call sites route through the portable…
38ee22cd
delock test: route kernel-reference tests through grouped_linear + fix asser…
7a449232
delock refactor(reference): kernel_reference aligned to the 7 interface methods
1d3948e6
delock fix: reference test alignment — all call sites through grouped_linear…
e3563429
delock chore: drop accidentally committed pre-commit cache dirs
fb0351e4
delock fix(kernels): decode_attn honors the q row stride
9c24dba1
delock
delock fix(segment_ki): pack FLA recurrent bypass inputs; gate by parity probe
5712d44f
delock
delock fix(cache): bridge the HF >=5.19 GDN cache protocol on the static cache
169577e3
delock sync(custom_collective): align with the validated #8997 implementation
7b3ea179
delock fix(segment_ki): correct GDN TP layout — slice z/b/a per rank, valida…
a4edcac3
delock
anyadontfly
delock merge: adopt master's _token_selector (#8939), drop _sample_from_logits
2d834e9f
delock delock force pushed from 2b304f40 to 2d834e9f 10 days ago
delock fix: merge fallout — drop duplicate get_query_offset, unused import
a9a6ac1e
delock fix(rollout): sample the prefill first token and the b>1 decode steps
4fd31d6e
delock fix(cache): restore the transformers>=5.20 seq-length protocol on Dee…
39210bf4
delock test(cache): align seq-length expectations with the restored +1 protocol
300e5b8d
delock fix(rollout): route graph capture through the accelerator-gated variable
43b624fa
delock fix(rollout): define the accelerator-gated use_graph_capture; drop is…
5b70a0be
delock fix(rollout): adopt master's _continuous_next_tokens with token selector
16b6fc67
delock fix: drop leftover @staticmethod on _continuous_next_tokens
30ce8d77
delock fix: add module-level logger; defensive num_attention_heads access
d8340393
delock fix: _continuous_prefill_cache sampling param; drop greedy-only restr…
e53e2dc2
delock fix: add sampling param to _continuous_prefill; update call sites
6d273de1
delock fix(rollout): restore master's graph-capture gates lost in the merge
eff04b38
delock Merge branch 'master' into gma/qwen_ki_expr
e2ad9437
delock
delock Merge branch 'master' into gma/qwen_ki_expr
2553e95c
tohtana tohtana added needs-gpu-test
delock Merge branch 'master' into gma/qwen_ki_expr
aed28987
delock fix(comm): restrict the CPU shm inference all-reduce to world-wide gr…
2d8f4094
delock Merge remote-tracking branch 'origin/master' into gma/qwen_ki_expr
00353b7b
delock test: keep GDN in_proj z/b/a replicated in the arena e2e specs
d43f5a2e
delock fix(rollout): drop fossil do_sample kwarg duplicated by _hf_sampling_…
9b4b7206
jinyouzhi
jinyouzhi commented on 2026-11-13
jinyouzhi
jinyouzhi commented on 2026-11-13
jinyouzhi
jinyouzhi commented on 2026-11-13
jinyouzhi
tohtana tohtana removed needs-gpu-test
delock
jinyouzhi
delock Merge master: adopt the #8997 evolution of the custom-AR files
f0001ab9
delock perf(rollout): amortize the b>1 EOS check like the b=1 path
8b05554f
delock fix(rollout): MRoPE position shape + visible fallback when installs d…
c77736dd
delock fix(rollout): check EOS every step in the b>1 decode loop
90175e9a
delock
jinyouzhi
delock
delock perf(module_inject): bf16-native silu epilogue in the GLU reference
b5298ed9
delock Merge master: adopt #8786's adaptive prefill cost model
2603bd3d
delock delock force pushed from 23adb533 to 2603bd3d 3 days ago
delock fix(rollout): keep use_segki out of the legacy positional config slots
de059c11
tohtana
tohtana requested changes on 2026-11-16
delock test: drop the scratch-repo sys.path insert from the arena e2e script
7423b144
delock fix(rollout): repair refresh_weights and wire it into sync_weights
bd69fc79
delock fix(rollout): release the continuous graph across weight syncs (inter…
9bf48c86
delock
delock Merge branch 'master' into gma/qwen_ki_expr
01257b71
tohtana
delock
delock fix(refresh): reach the backend cat cache on GLU-only (dense) models
ed86967a
delock fix: yapf style + skip the GLU-cache test without a registered FusedG…
0aeb3db6
delock delock force pushed from 36d8a2ad to 0aeb3db6 13 hours ago
delock fix: the GLU-cache skip misread an instantiated NotImplementedBuilder
012bea77
tohtana
tohtana approved these changes on 2026-11-19

Login to write a write a comment.

Login via GitHub

Assignees
No one assigned
Labels
Milestone