accelerate
9888c7ed - feat: use datasets.IterableDataset shard if possible (#3635)

Commit
251 days ago
feat: use datasets.IterableDataset shard if possible (#3635) * feat: use datasets.IterableDataset shard if possible. When `accelerator.prepare` is called on a `datasets.IterableDataset`, use the `shard` method to split the dataset across the available processes. This allows for more efficient data loading and processing. Without load and slice overhead of `IterableDatasetShard` * dataset * remove unused import * style --------- Co-authored-by: wuwenxu.01 <wuwenxu.01@bytedance.com>
Author
Parents
Loading