PR #2433 split_between_processes for Dataset

split_between_processes for Dataset #2433

muellerzr merged 11 commits into huggingface:main from split-dataset_between_processes

split_between_processes for Dataset

d7140732

muellerzr requested changes on 2024-02-12

Update state.py

97e4c9fc

remove param datasets.Dataset from split_between_processes, add note …

b7387478

geronimi73 requested a review from

muellerzr 2 years ago

is_datasets_available is a function not a var

46f86949

reformat to make ruff happy

ed68b881

shagunsodhani commented on 2024-02-13

isinstance(inputs, Dataset) only if is_datasets_available()

69bb8869

muellerzr approved these changes on 2024-02-13

add test_split_between_processes_dataset

d25a7876

muellerzr commented on 2024-02-13

split_between_processes for Dataset: pad if apply_padding

6fe38b00

removed trailing whitespace

7ca380e8

complete test_split_between_processes_dataset

ee372212

fix test_split_between_processes_dataset for single GPU

ceaa52f6

muellerzr approved these changes on 2024-03-14

muellerzr merged 910c1b6a into main 2 years ago

Reviewers

muellerzr

shagunsodhani

Assignees

No one assigned

Labels

None yet

Milestone

No milestone