[CB] GLM-5.3 faster than vLLM on 8x B300 (draft, to be split) #51065
Add the packed allocator
c3925735
Add the mla alocator
a6d51858
MLA threaded in + size correction
cff6590e
Oops import
23d841b9
Docs
957e7253
Test
5274436a
Rename
41b53a09
MLA allocator rewire
68c73219
[CB] Support MLA layers with a sparse attention indexer
620d891b
Keep the indexed allocator docstrings within 119 characters
962e3f17
[scratch, bench only] Absorbed sparse MLA on CB cache rows and per-qu…
d2374a58
[scratch, gap prototype] One all-reduce per MLP block under TP, behin…
14033960
[scratch, bench only] CB engine prototypes behind env vars: device-bu…
b6681f83
[scratch, bench only] PROTO_FP8_LINEAR prototypes for the dense FP8 l…
78aa5f25
[scratch, bench only] Env-gated glue kernel reductions: fused RMSNorm…
4774cd28
[scratch, bench only] CB engine prototype: block table path for MLA d…
d8bbca55
[scratch, bench only] Env-gated opaque cache writes of the prototype …
593f5992
Merge branch 'gap-comm-moe' into gap-combined
6cabd87d
[scratch, bench only] Skip indexer scoring when every context fits in…
df466618
Merge branch 'gap-glue' into gap-combined
ea405425
[scratch, bench only] Merge gap-engine into gap-combined: block table…
1f870304
[scratch, bench only] FUSE_COMPILE: regional torch.compile of the CB …
d9be0593
[scratch, bench only] FUSE_COMPILE under tensor parallel: per-layer l…
d21294a8
[scratch, bench only] Prefill batches take the per-request indexer pa…
9c0f7494
[scratch, gap prototype] FP8 experts through the hub kernel's fused M…
df275aa6
[scratch, bench only] TP inference all-reduce through flashinfer's MN…
09443847
[scratch, bench only] Routed FP8 experts at inference: flashinfer TRT…
66a1acc0
[scratch, bench only] FUSE_COMPILE: optional opaque grouped FP8 kerne…
b43bec1e
[scratch, bench only] CB engine prototypes, phase 2: one upload per d…
ca000814
[scratch, bench only] GLM-5 MTP module as layers[num_hidden_layers] (…
0bcf785b
[scratch, bench only] Prefill prototypes behind env flags: fused inde…
46f794f4
[scratch, bench only] CB_MTP: host-side check that no placeholder tok…
cf18a68c
[scratch, bench only] Block table decode path: pass the valid length …
94b99f32
[scratch, bench only] flashinfer all-reduce: world sizes the kernels …
6173480d
[scratch, bench only] flashinfer_fused: refuse flashinfer older than …
810ce882
[scratch, bench only] Decode experts kernels: weights-first tiles (na…
de902c87
[scratch, bench only] PROTO_PREFILL_NOSKIP=1: check-only flag, prefil…
ea59f2c1
[scratch, bench only] Decode experts: table for every batch size, PRO…
8f86045e
[scratch, bench only] CB sync IOs: the compute stream waits for the i…
f6eb9b9b
[scratch, bench only] CB_MTP: logits rows follow the request count; s…
d5d37ccd
[scratch, bench only] FIFO scheduler prototype: prefills apart from d…
c353cf45
[scratch, bench only] FUSE_COMPILE: per-region recompile limit (dynam…
e3647fb7
[scratch, bench only] Decode experts: down kernel walking 6 weight ti…
62171542
[scratch, gap2 train] Apply #50891 (release_memory / restore_memory) …
38f3772d
Merge branch 'gap2-engine' into gap2-train
1fb963c7
[scratch, bench only] Apply the src diff of #50891 (release and resto…
f7b42f79
Merge branch 'gap2-prefill' into gap2-combined
c6bcd391
Merge branch 'gap2-comm' into gap2-combined
38ad2ac8
Merge branch 'gap2-experts' into gap2-combined
66a13364
[scratch, bench only] fast TP all-reduce: only the CB engine thread m…
2788cddc
[scratch, bench only] Synchronous mode: issue the input uploads on th…
ada393b7
Merge branch 'gap2-engine' into gap2-combined
f66905a5
[scratch, bench only] FUSE_KERNELS: fused Triton kernels for the deco…
1b8378d0
[scratch, gap2 train] kv_b_proj of the absorbed attention follows PEF…
664e367d
[scratch, bench only] fast TP all-reduce: warm-up captures (ModelRunn…
eba276bd
Merge branch 'gap2-comm' into gap2-combined
45c06de7
Merge branch 'gap2-train' into gap2-combined
e5ff5038
Merge branch 'gap2-fuse' into gap3-compile (regional compile + fused …
34e8ecef
Merge gap2-mtp into gap3-mtp (MTP module + CB speculative decoding on…
673589fd
[scratch, bench only] FUSE_COMPILE on the stack S: block-table decode…
ff2a5eda
[scratch, bench only] CB_MTP=spec on the block table decode path (CB_…
d4208188
[scratch, bench only] Experts: TRT-LLM layout in place, no copy (PROT…
93416f05
[scratch, bench only] FUSE_COMPILE: cache views' row dim dynamic (no …
d3ce95b8
[scratch, gap3 train] PROTO_TF32=scoped: router logits and indexer sc…
ac4d503b
[scratch, bench only] FUSE_COMPILE / FUSE_KERNELS: per-cache state li…
f18c2fb9
Merge branch 'gap3-experts' into gap3-train
44d54d8f
gap3 gemm: fused q_a+kv_a(+wk) and gate+up GEMMs, bf16 router GEMM wi…
ff9a83d4
Merge branch 'gap3-gemm' into gap2-combined
330d9f87
Merge branch 'gap3-compile' into gap2-combined
b46c9a13
Merge branch 'gap3-mtp' into gap2-combined
1e6ee896
[scratch, bench only] Experts: TRT-LLM routing inside the kernel and …
87514d42
[scratch, bench only] CB_MTP_ASYNC=1: MTP speculative decoding under …
8c87e115
[scratch, gap3 train] TRAIN_FIX_KV_B: LoRA adapters on a tensor-paral…
301fb9c4
[scratch, bench only] FUSE_COMPILE / FUSE_KERNELS with CB_MTP=spec: b…
5da52d75
[scratch, bench only] MOE_SHARED_STREAM=1: shared expert on a side st…
8c5562a1
[scratch, bench only] FUSE_KERNELS with GEMM_FUSE_QKV_A: the fused la…
829f6fb9
[scratch, bench only] FUSE_COMPILE / FUSE_KERNELS under TP: the twins…
e442d679
Merge branch 'gap3-experts2' into gap2-combined
fb5d555a
Merge branch 'gap3-compile' into gap2-combined
2fce5aeb
Merge branch 'gap3-mtpa' into gap2-combined
ee642ed1
Merge branch 'gap3-shstream' into gap2-combined
bad0aa0e
[scratch, gap3 train] release_memory keeps only the blocks in use on …
df3e447f
Merge branch 'gap3-train' into gap2-combined
c863939c
[scratch, bench only] CB_MTP_K=k: k drafts per step on the block tabl…
4678b685
Merge remote-tracking branch 'origin/main' into cb-glm53-fast-path
791eee13
Assignees
No one assigned
Login to write a write a comment.
Login via GitHub