transformers
Tensor parallelism: let a split model be one replica among several
#48568
Open
Open
Tensor parallelism: let a split model be one replica among several #48568
qgallouedec
wants to merge
10 commits into
main
from
tp-replicas
Let a tensor-parallel model be one replica among several
3a91de80
Sum replicated gradients that are held as DTensors
298c2b85
Compute the gradient norm of a tensor parallel model
b44eb764
Do not keep the gathered weights on ranks that write nothing
7be367b5
Drop the training turn: pause() (#48462) is how a thread takes the mo…
f12534e8
Trim the mesh docstring
c5d7bd42
Merge branch 'main' into tp-replicas
8f7bb4ba
SunMarc
commented
on 2026-09-09
Merge remote-tracking branch 'upstream/main' into tp-replicas
075f05cf
Drop the import the removed grad-norm helper needed
9aed6a22
Merge branch 'main' into tp-replicas
1a124497
Login to write a write a comment.
Login via GitHub