DeepSpeed
Fused RMSNorm: multi-row programs and a one-pass backward for narrow norms
#8664
Merged

Fused RMSNorm: multi-row programs and a one-pass backward for narrow norms #8664

yh0903
yh0903 yh0903 marked this pull request as ready for review 53 days ago
yh0903 yh0903 requested a review from tohtana tohtana 53 days ago
yh0903 yh0903 requested a review from tjruwase tjruwase 53 days ago
yh0903 yh0903 requested a review from loadams loadams 53 days ago
yh0903 yh0903 marked this pull request as draft 53 days ago
yh0903 Give narrow fused RMSNorm rows multi-row programs and a one-pass back…
f6a10d11
yh0903 Test the narrowest RMSNorm block at width 16
8e7b0fd5
yh0903 Test sub-16 RMSNorm widths against meaningful gradient references
db06f991
yh0903 Check the first width outside the narrow RMSNorm cutoff
45afc059
yh0903 yh0903 force pushed from 3827f935 to 45afc059 36 days ago
yh0903 yh0903 changed the title Fused RMSNorm: multi-row programs and a one-pass backward for narrow norms (stacked on #8645) Fused RMSNorm: multi-row programs and a one-pass backward for narrow norms 36 days ago
yh0903 yh0903 marked this pull request as ready for review 36 days ago
chatgpt-codex-connector
chatgpt-codex-connector
chatgpt-codex-connector commented on 2026-10-12
yh0903 Cap narrow RMSNorm tiles to the input row count
bf6b1bf9
hwchen2017 hwchen2017 requested a review from copilot-pull-request-reviewer copilot-pull-request-reviewer 36 days ago
copilot-pull-request-reviewer
copilot-pull-request-reviewer commented on 2026-10-13
tohtana
tohtana requested changes on 2026-10-14
yh0903 Bound near-zero narrow RMSNorm gradient comparisons
5b540919
yh0903
pengdurice
yh0903
tohtana tohtana added needs-gpu-test
pengdurice
pengdurice Merge branch 'master' into yh0903/fused-rmsnorm-narrow-rows
6191a373
tohtana tohtana removed needs-gpu-test
tohtana
tohtana approved these changes on 2026-11-14
tohtana Merge branch 'master' into yh0903/fused-rmsnorm-narrow-rows
1e1f77c1
tohtana tohtana merged 7caa7a22 into master 3 days ago

Login to write a write a comment.

Login via GitHub

Assignees
No one assigned
Labels
Milestone