auto-round
87b617ea - fix: lower default MoE auto-decode threshold 256 -> 32

Commit
14 days ago
fix: lower default MoE auto-decode threshold 256 -> 32 The shared S4 DPAS grouped-GEMM (prefill path) already beats the scalar GEMV decode kernel by ~2x at 256 tokens (bs32) and only loses at the single-stream bs1 (8-token) extreme. Routing 256-token batches to decode was leaving ~2x on the table, so lower the auto-dispatch default and update coupled unified-dispatch tests and perf-test notes. Co-authored-by: a32543254 <53296245+a32543254@users.noreply.github.com>
Author
Parents
Loading