transformers
f4582493 - Trainer: average the gradients of parameters FSDP2 does not shard

Commit
64 days ago
Trainer: average the gradients of parameters FSDP2 does not shard PEFT adapters are attached after fully_shard, so they are plain replicated tensors that FSDP2 never reduce-scatters, and the native-FSDP path skips the DDP wrap. Each rank kept the gradient of its own batch: in fp32 the lora_B gradients were off by 2x against a single-GPU reference while the loss matched exactly. A post-accumulate-grad hook averages them, which is what DDP did for the same parameters. (cherry picked from commit 213f2ce71cdd2eca576d6f26e134fdbdef05c03c)
Author
Committer
Parents
Loading