DeepSpeed
a0e86487 - Add subchannel FP8/INT8 quantized training for nn.Linear and AutoEP experts

Commit
2 days ago
Add subchannel FP8/INT8 quantized training for nn.Linear and AutoEP experts A new opt-in `quantization.subchannel` config section quantizes the three GEMMs of a linear layer (forward, dx, dw) independently: FP8 (e4m3/e5m2) or INT8 operands with one FP32 scale per block of the reduction dimension, optional stochastic rounding, and an optional fused butterfly Hadamard rotation inside each block. `block_size: 0` gives each operand row a single scale, which lets the FP8 GEMM apply scales once per output element and run faster than BF16 on wide layers. The GEMMs run on Triton kernels on CUDA devices of compute capability 8.9+ (8.0+ for INT8) and on a PyTorch reference implementation elsewhere. Parameters, gradients and optimizer state keep their usual precision. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: Zhipeng Wang <7927074+PKUWZP@users.noreply.github.com>
Author
Parents
Loading