DeepSpeed
b5298ed9 - perf(module_inject): bf16-native silu epilogue in the GLU reference

Commit
2 days ago
perf(module_inject): bf16-native silu epilogue in the GLU reference The glu_epilogue fallback path (m > 32 decode and all prefill, where M = batch * prompt_len) computed silu via an fp32 decomposition: two upcasts of the [M, inter] gate/up tensors, five elementwise kernels, and a final downcast. Replacing it with F.silu in bf16 eliminates the cast round-trips and three kernels, and is bitwise-equal to HF's activation (the fp32 formula rounds differently -- max 0.5 at bf16 -- so m<=32, m>32 and prefill previously disagreed with HF and each other). Measured on L20 (Ada), Qwen3.5-4B, b=64 greedy decode, interleaved A/B (medians of 3): 1667.3 -> 1765.5 tok/s (+5.9%, non-overlapping ranges). The native op keeps the small-M fused kernel (m <= 32); on Ada its large-M path splits into row-chunked calls / an aten composite with the same fp32 round-trip, so the eager fallback with this epilogue is the fastest large-M implementation on this arch (1765 vs 1623 via the native dispatcher; Hopper keeps the concat-GEMM path). test_segment_ki_kernels.py (the only oracle consumer; compares with fp32 inputs under tolerance) passes: 25/25 on L20. Numerics note: the fp32 single-rounding formula stays documented in the docstring as the spec semantics; the executable path now matches HF instead of exceeding its precision. Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
Author
Parents
Loading