transformers
95e96cb3 - [distributed] Refuse context parallelism in Trainer

Commit

6 hours ago

[distributed] Refuse context parallelism in Trainer `Trainer` would give a different batch to the ranks of a cp group, which expect the same one. Use a plain training loop until its dataloader shards the sequences instead.

References

#50496 - [`distributed`] `DistributedConfig(cp_size=N)`

#50499 - [`distributed`] Run the context parallel tests on every causal LM

#50511 - [`distributed`] Context parallelism with tensor parallelism

#50514 - [`distributed`] Context parallelism with expert parallelism

#50515 - [`distributed`] Context parallelism: pad the sequences to a multiple of `cp_size`

#50516 - [`distributed`] Context parallelism: padding masks, sliding windows and chunked attention with sdpa

#50521 - [`distributed`] Train a model with context parallelism in Trainer

#50522 - [`docs`] Context parallelism with `DistributedConfig`

Author

3outeille

Committer

3outeille

Parents

8051af49

transformers 95e96cb3 - [distributed] Refuse context parallelism in Trainer

transformers
95e96cb3 - [distributed] Refuse context parallelism in Trainer