Fused RMSNorm: multi-row programs and a one-pass backward for narrow norms (#8664)
## Summary
Builds on the opt-in fused RMSNorm implementation merged in
deepspeedai/DeepSpeed#8645. This branch has been rebased onto current
upstream `master`: the PR diff contains only narrow-row
scheduling/backward changes, boundary tests and documentation, not the
former parent commits.
The merged parent's per-instance opt-in, checkpoint opt-out,
copy/DataParallel behavior, eager fallback and collection-safe CUDA test
framework are preserved.
The parent fused RMSNorm kernels run one row per program. For the
128-wide query and key norms of models like Qwen3, that leaves most of
each program idle. The backward also reads `hidden` and the incoming
gradient twice, once in the dx kernel and once in the weight-gradient
partial kernel.
For norms at most 128 wide, this change:
- runs up to 4096 elements per program in the forward (32 rows at width
128), with smaller power-of-two row tiles for short inputs;
- uses one backward kernel that reads each row once and writes both the
input gradient and the per-block weight-gradient partials, which are
then summed as before.
Wider norms, including the hidden-size norms, keep the existing row
kernels unchanged.
The short-input cap addresses the Codex review in this existing PR,
rather than opening a duplicate optimization PR. Both passes use
`min(max_rows_per_tile, next_power_of_2(n_rows))`; a single width-128 or
width-one row uses `BLOCK_M=1`, not 32 or 256 padded rows. Filled tiles
keep their original scheduling. AI assistance was used for this
follow-up, with author review before submission.
## Numerics
The expression and its rounding points are the same as the parent row
kernels'. Multi-row scheduling can change the FP32 reduction order for
variance, `grad_rstd`, and weight-gradient partials. **Bitwise equality
with eager is not claimed**. Forward, gamma-gradient and wide-path
numerical thresholds are unchanged. The narrow-shape input-gradient
comparison accepts each element at **at most 8 ULP or absolute error at
most 1e-7**, while still requiring **at least 95% within one ULP**. This
bounds near-zero cancellation errors without increasing the ULP limit
for other elements. Identical fused runs give repeatable input and
weight gradients at widths 128 and 2048, which is a separate determinism
property.
At width one and `eps > 0`, the pre-gamma input derivative is `eps /
(x**2 + eps)**(3/2)`, not strictly zero. A tiny epsilon can make that
value hard to distinguish from low-precision cancellation. The dedicated
width-one test uses `eps=0.25` and an FP64 reference so the nonzero
gradient is measurable, without relaxing the existing ULP thresholds.
## Tests
The existing `test_fused_rms_norm.py` forward/backward test gains narrow
shapes:
- many rows with a partial last block: `(4099, 128)`;
- the head-major query-norm shape: `(2, 300, 32, 128)`;
- width 96 with masked columns, and widths 64 and 16 at the scheduling
boundary;
- widths 7 and 15 with masked columns and partial last blocks, under the
same forward/dx/dgamma ULP thresholds.
A width-one test checks the nonzero input derivative against FP64 and
forward/gradient parity with eager. The repeated-run test checks fused
input and weight gradients at widths 128 and 2048. Existing tests cover
head-major non-contiguous inputs, transposed incoming gradients, and
non-contiguous weights.
The first width outside the narrow cutoff, 129, is additionally checked
in BF16/FP16 under the unchanged wide-path accuracy contract. The shape,
FP64 width-one and determinism cases now run inside the merged parent's
`DistributedTest` CUDA class, with Triton imports and availability
checks deferred to execution.
Earlier validation passed the six BF16/FP16 sub-16 and width-one cases
on one NVIDIA H100 80GB HBM3 (torch 2.10/CUDA 13.0, Triton 3.6).
Pre-cap rebased validation on two NVIDIA H100 80GB HBM3 GPUs, torch
2.10/CUDA 13.0, Triton 3.6.0 and transformers 5.12.0: **90 passed, 0
skipped** under `pytest --forked`. This includes the narrow
numerical/masked-width/FP64/determinism tests and the merged parent's
guards, copying, checkpoint loading/reinstallation and two-GPU
DataParallel training.
A separate public-operator comparison with the merged parent at widths
**129, 130, 256 and 2048** passes for BF16 and FP16: all sampled
outputs, input gradients and weight gradients have identical bit
patterns in these 8 comparisons. The wide kernel implementations and
parent dispatch/checkpoint logic are also unchanged by AST comparison.
Short-row follow-up in the same pinned two-H100 environment: **56
focused tests passed, 0 skipped**, including 21 explicit scheduling
cases and BF16/FP16 regressions that record both real Triton launches
while checking forward/dx/dgamma under the unchanged ULP limits. These
cover empty, single-row, non-power-of-two, tile-boundary and filled
inputs, the FP64 width-one oracle, and affected copying, checkpoint
round-trip and two-GPU DataParallel paths.
The two new real-launch regressions fail against the previous runtime: a
single width-one row launches `BLOCK_M=256` in both passes instead of 1.
Separate public-operator controls at `(8192, 128)`, `(257, 16)`, `(7,
129)` and `(7, 2048)` pass in BF16/FP16, with identical output/dx/dgamma
bit patterns in all 8 before/after comparisons.
Focused GPU selector, from `tests/`:
```bash
PYTHONPATH=.. pytest --forked unit/v1/ops/triton_ops/test_fused_rms_norm.py \
-k 'narrow_tile or matches_hf_forward_and_backward or width_one or copy_uses_its_own_epsilon or model_round_trip or data_parallel'
```
The pre-numerical-review file on CPU (macOS, torch 2.10.0, transformers
5.12.0) with `DS_ACCELERATOR=cpu` and `pytest --forked`: **56 passed, 57
CUDA cases skipped**. Existing support limits and opt-in/fallback
behavior are unchanged; ZeRO/offload/TP/PP and other GPU architectures
were not newly exercised by this bounded follow-up. No full E2E or new
performance matrix was rerun, and no measured latency claim for short
decoding batches is made.
**RTX numerical-review follow-up:** the reviewer reported one of
2,457,600 BF16 `dx` values at 119 ULP in `(2, 300, 32, 128)` on RTX PRO
4500 and suggested an absolute alternative. Only the narrow common `dx`
assertion now has the bounded alternative above; kernels, scheduling,
forward/dgamma checks and the separate FP64 width-one oracle are
unchanged. Five CPU assertion regressions cover mixed per-element
ULP/absolute acceptance, strict default behavior, errors beyond both
limits, too many small outliers, and NaN/Inf rejection. A synthetic BF16
119-ULP example has absolute error 1.39e-8; it is not the reviewer's
tensor.
The current whole file passes on CPU under `pytest --forked`: **61
passed, 57 CUDA cases skipped**. The focused H100 run passes **9 tests,
0 skipped** under `--forked`: the five assertion cases, BF16/FP16 at the
reported shape and both short-tile correctness/launch regressions. The
allocation has 2x H100 80GB HBM3, driver 580.159.04, torch 2.10.0.2/CUDA
13.0, Triton 3.6.0 and transformers 5.12.0. On that H100 environment the
reported shape has max `dx` distance 1 ULP in BF16 and 2 ULP in FP16;
neither reproduces the RTX outlier. **RTX PRO 4500 has not been rerun by
the author for this follow-up**, so confirmation of the reported case on
that hardware remains with review. Changed-files pre-commit passed.
Focused reproduction, from `tests/`:
```bash
PYTHONPATH=.. pytest --forked unit/v1/ops/triton_ops/test_fused_rms_norm.py \
-k 'dx_tolerance or (matches_hf_forward_and_backward and shape6) or narrow_tiles'
```
## Performance
**Hardware-dependent tradeoff:** the reviewer measured narrow forward
**11-16% slower on RTX PRO 4500** than the parent row kernels. This is a
reviewer-reported forward-only observation, not an author-reproduced
timing or a measured E2E regression. The H100 training benefits below do
not imply a universal speedup; the path remains opt-in. No new
performance tuning or benchmark matrix was added in this review
follow-up.
**Incremental local effect of this PR:** measured on one H100,
Qwen3-30B-A3B attention layer under reentrant checkpointing (forward,
recompute and backward), with fused RoPE already enabled in both arms.
| Sequence | Q/K norm time | Attention-layer time |
|---|---|---|
| 8192 | 0.73 → 0.33 ms | 7.80 → 7.52 ms |
| 16384 | 1.40 → 0.45 ms | 23.66 → 22.89 ms |
**Combined E2E effect:** Qwen3-30B-A3B training with AutoEP, 16x H100,
expert parallel 16, 48 layers, one sequence per GPU, activation
checkpointing. The "after" arm adds **both this narrow-row RMSNorm
change and fused RoPE (deepspeedai/DeepSpeed#8663)** on an otherwise
identical stack, in the same allocation. The total change cannot be
attributed to this PR alone:
| Sequence | Before | After | Change |
|---|---|---|---|
| 8192 | 1714 ms | 1654 ms | −3.5% |
| 16384 | 3632 ms | 3511 ms | −3.3% |
- The 8192 result separates cleanly: both "after" runs are below both
"before" runs.
- The 16384 run ranges overlap, so its −3.3% result is **indicative**,
not a confirmed independent speedup.
- Peak allocated memory did not increase at either length (50.434 GB at
8192; 57.161 GB at 16384). Peak reserved memory decreased slightly.
- Final-step losses differed by at most 0.0023 in these paired runs
(within-arm run-to-run spread: 0.0016–0.0068). This is a **final-step
observation, not a bound across all steps**.
The narrow kernel arithmetic and filled-tile scheduling used by these
measured workloads are unchanged; only underfilled short inputs use the
smaller row tile. Rebase/conflict resolution preserves the merged parent
rather than reinstating the former instance-bound forward or serialized
opt-in behavior. The performance numbers above are reused historical
measurements, not a retiming of the rebased stack or the short-row
follow-up.
## CI
`ci/tests_fetcher.py` against current master selects exactly
`tests/unit/v1/ops/triton_ops/test_fused_rms_norm.py`. The file retains
the parent's fork-safe worker execution and CPU coverage; the
DataParallel case additionally requires two CUDA devices. Full GPU CI
follows the repository's merge-queue workflow.
---------
Signed-off-by: yh0903 <helloyu0903@gmail.com>
Co-authored-by: pengdurice <pengduhit@gmail.com>
Co-authored-by: Masahiro Tanaka <81312776+tohtana@users.noreply.github.com>