transformers
2dd426ee - Test the Trainer under token dispatch against the Trainer under DDP

Commit
67 days ago
Test the Trainer under token dispatch against the Trainer under DDP The dispatch-specific Trainer code (per-rank sampler, token count and loss scale, per-mesh gradient norm and optimizer groups) had no test: the existing comparison drives the model by hand. Every step consumes the whole dataset, so the token-averaged loss, the gradient norm and the updated weights have to match DDP step by step whichever rank gets which sample. accelerate's `_prepare_tp` (1.12 up to at least 1.14) cannot prepare an all-DTensor model: it imports `ReplicateParallel`, gone since #42809, and remaps optimizer parameters by plain-tensor address. The test skips it, since on an FSDP2-managed model it is a no-op.
Author
Parents
Loading