DeepSpeed
7caa7a22 - Fused RMSNorm: multi-row programs and a one-pass backward for narrow norms (#8664)

Commit
3 days ago
Fused RMSNorm: multi-row programs and a one-pass backward for narrow norms (#8664) ## Summary Builds on the opt-in fused RMSNorm implementation merged in deepspeedai/DeepSpeed#8645. This branch has been rebased onto current upstream `master`: the PR diff contains only narrow-row scheduling/backward changes, boundary tests and documentation, not the former parent commits. The merged parent's per-instance opt-in, checkpoint opt-out, copy/DataParallel behavior, eager fallback and collection-safe CUDA test framework are preserved. The parent fused RMSNorm kernels run one row per program. For the 128-wide query and key norms of models like Qwen3, that leaves most of each program idle. The backward also reads `hidden` and the incoming gradient twice, once in the dx kernel and once in the weight-gradient partial kernel. For norms at most 128 wide, this change: - runs up to 4096 elements per program in the forward (32 rows at width 128), with smaller power-of-two row tiles for short inputs; - uses one backward kernel that reads each row once and writes both the input gradient and the per-block weight-gradient partials, which are then summed as before. Wider norms, including the hidden-size norms, keep the existing row kernels unchanged. The short-input cap addresses the Codex review in this existing PR, rather than opening a duplicate optimization PR. Both passes use `min(max_rows_per_tile, next_power_of_2(n_rows))`; a single width-128 or width-one row uses `BLOCK_M=1`, not 32 or 256 padded rows. Filled tiles keep their original scheduling. AI assistance was used for this follow-up, with author review before submission. ## Numerics The expression and its rounding points are the same as the parent row kernels'. Multi-row scheduling can change the FP32 reduction order for variance, `grad_rstd`, and weight-gradient partials. **Bitwise equality with eager is not claimed**. Forward, gamma-gradient and wide-path numerical thresholds are unchanged. The narrow-shape input-gradient comparison accepts each element at **at most 8 ULP or absolute error at most 1e-7**, while still requiring **at least 95% within one ULP**. This bounds near-zero cancellation errors without increasing the ULP limit for other elements. Identical fused runs give repeatable input and weight gradients at widths 128 and 2048, which is a separate determinism property. At width one and `eps > 0`, the pre-gamma input derivative is `eps / (x**2 + eps)**(3/2)`, not strictly zero. A tiny epsilon can make that value hard to distinguish from low-precision cancellation. The dedicated width-one test uses `eps=0.25` and an FP64 reference so the nonzero gradient is measurable, without relaxing the existing ULP thresholds. ## Tests The existing `test_fused_rms_norm.py` forward/backward test gains narrow shapes: - many rows with a partial last block: `(4099, 128)`; - the head-major query-norm shape: `(2, 300, 32, 128)`; - width 96 with masked columns, and widths 64 and 16 at the scheduling boundary; - widths 7 and 15 with masked columns and partial last blocks, under the same forward/dx/dgamma ULP thresholds. A width-one test checks the nonzero input derivative against FP64 and forward/gradient parity with eager. The repeated-run test checks fused input and weight gradients at widths 128 and 2048. Existing tests cover head-major non-contiguous inputs, transposed incoming gradients, and non-contiguous weights. The first width outside the narrow cutoff, 129, is additionally checked in BF16/FP16 under the unchanged wide-path accuracy contract. The shape, FP64 width-one and determinism cases now run inside the merged parent's `DistributedTest` CUDA class, with Triton imports and availability checks deferred to execution. Earlier validation passed the six BF16/FP16 sub-16 and width-one cases on one NVIDIA H100 80GB HBM3 (torch 2.10/CUDA 13.0, Triton 3.6). Pre-cap rebased validation on two NVIDIA H100 80GB HBM3 GPUs, torch 2.10/CUDA 13.0, Triton 3.6.0 and transformers 5.12.0: **90 passed, 0 skipped** under `pytest --forked`. This includes the narrow numerical/masked-width/FP64/determinism tests and the merged parent's guards, copying, checkpoint loading/reinstallation and two-GPU DataParallel training. A separate public-operator comparison with the merged parent at widths **129, 130, 256 and 2048** passes for BF16 and FP16: all sampled outputs, input gradients and weight gradients have identical bit patterns in these 8 comparisons. The wide kernel implementations and parent dispatch/checkpoint logic are also unchanged by AST comparison. Short-row follow-up in the same pinned two-H100 environment: **56 focused tests passed, 0 skipped**, including 21 explicit scheduling cases and BF16/FP16 regressions that record both real Triton launches while checking forward/dx/dgamma under the unchanged ULP limits. These cover empty, single-row, non-power-of-two, tile-boundary and filled inputs, the FP64 width-one oracle, and affected copying, checkpoint round-trip and two-GPU DataParallel paths. The two new real-launch regressions fail against the previous runtime: a single width-one row launches `BLOCK_M=256` in both passes instead of 1. Separate public-operator controls at `(8192, 128)`, `(257, 16)`, `(7, 129)` and `(7, 2048)` pass in BF16/FP16, with identical output/dx/dgamma bit patterns in all 8 before/after comparisons. Focused GPU selector, from `tests/`: ```bash PYTHONPATH=.. pytest --forked unit/v1/ops/triton_ops/test_fused_rms_norm.py \ -k 'narrow_tile or matches_hf_forward_and_backward or width_one or copy_uses_its_own_epsilon or model_round_trip or data_parallel' ``` The pre-numerical-review file on CPU (macOS, torch 2.10.0, transformers 5.12.0) with `DS_ACCELERATOR=cpu` and `pytest --forked`: **56 passed, 57 CUDA cases skipped**. Existing support limits and opt-in/fallback behavior are unchanged; ZeRO/offload/TP/PP and other GPU architectures were not newly exercised by this bounded follow-up. No full E2E or new performance matrix was rerun, and no measured latency claim for short decoding batches is made. **RTX numerical-review follow-up:** the reviewer reported one of 2,457,600 BF16 `dx` values at 119 ULP in `(2, 300, 32, 128)` on RTX PRO 4500 and suggested an absolute alternative. Only the narrow common `dx` assertion now has the bounded alternative above; kernels, scheduling, forward/dgamma checks and the separate FP64 width-one oracle are unchanged. Five CPU assertion regressions cover mixed per-element ULP/absolute acceptance, strict default behavior, errors beyond both limits, too many small outliers, and NaN/Inf rejection. A synthetic BF16 119-ULP example has absolute error 1.39e-8; it is not the reviewer's tensor. The current whole file passes on CPU under `pytest --forked`: **61 passed, 57 CUDA cases skipped**. The focused H100 run passes **9 tests, 0 skipped** under `--forked`: the five assertion cases, BF16/FP16 at the reported shape and both short-tile correctness/launch regressions. The allocation has 2x H100 80GB HBM3, driver 580.159.04, torch 2.10.0.2/CUDA 13.0, Triton 3.6.0 and transformers 5.12.0. On that H100 environment the reported shape has max `dx` distance 1 ULP in BF16 and 2 ULP in FP16; neither reproduces the RTX outlier. **RTX PRO 4500 has not been rerun by the author for this follow-up**, so confirmation of the reported case on that hardware remains with review. Changed-files pre-commit passed. Focused reproduction, from `tests/`: ```bash PYTHONPATH=.. pytest --forked unit/v1/ops/triton_ops/test_fused_rms_norm.py \ -k 'dx_tolerance or (matches_hf_forward_and_backward and shape6) or narrow_tiles' ``` ## Performance **Hardware-dependent tradeoff:** the reviewer measured narrow forward **11-16% slower on RTX PRO 4500** than the parent row kernels. This is a reviewer-reported forward-only observation, not an author-reproduced timing or a measured E2E regression. The H100 training benefits below do not imply a universal speedup; the path remains opt-in. No new performance tuning or benchmark matrix was added in this review follow-up. **Incremental local effect of this PR:** measured on one H100, Qwen3-30B-A3B attention layer under reentrant checkpointing (forward, recompute and backward), with fused RoPE already enabled in both arms. | Sequence | Q/K norm time | Attention-layer time | |---|---|---| | 8192 | 0.73 → 0.33 ms | 7.80 → 7.52 ms | | 16384 | 1.40 → 0.45 ms | 23.66 → 22.89 ms | **Combined E2E effect:** Qwen3-30B-A3B training with AutoEP, 16x H100, expert parallel 16, 48 layers, one sequence per GPU, activation checkpointing. The "after" arm adds **both this narrow-row RMSNorm change and fused RoPE (deepspeedai/DeepSpeed#8663)** on an otherwise identical stack, in the same allocation. The total change cannot be attributed to this PR alone: | Sequence | Before | After | Change | |---|---|---|---| | 8192 | 1714 ms | 1654 ms | −3.5% | | 16384 | 3632 ms | 3511 ms | −3.3% | - The 8192 result separates cleanly: both "after" runs are below both "before" runs. - The 16384 run ranges overlap, so its −3.3% result is **indicative**, not a confirmed independent speedup. - Peak allocated memory did not increase at either length (50.434 GB at 8192; 57.161 GB at 16384). Peak reserved memory decreased slightly. - Final-step losses differed by at most 0.0023 in these paired runs (within-arm run-to-run spread: 0.0016–0.0068). This is a **final-step observation, not a bound across all steps**. The narrow kernel arithmetic and filled-tile scheduling used by these measured workloads are unchanged; only underfilled short inputs use the smaller row tile. Rebase/conflict resolution preserves the merged parent rather than reinstating the former instance-bound forward or serialized opt-in behavior. The performance numbers above are reused historical measurements, not a retiming of the rebased stack or the short-row follow-up. ## CI `ci/tests_fetcher.py` against current master selects exactly `tests/unit/v1/ops/triton_ops/test_fused_rms_norm.py`. The file retains the parent's fork-safe worker execution and CPU coverage; the DataParallel case additionally requires two CUDA devices. Full GPU CI follows the repository's merge-queue workflow. --------- Signed-off-by: yh0903 <helloyu0903@gmail.com> Co-authored-by: pengdurice <pengduhit@gmail.com> Co-authored-by: Masahiro Tanaka <81312776+tohtana@users.noreply.github.com>
Author
Parents
Loading