Megatron-DeepSpeed
cb7648b0
- added dirty hack to have a maximum size
Go
Login via GitHub
Home
Pricing
FAQ
Install
Login
via GitHub
Commit
View On
GitHub
Commit
3 years ago
added dirty hack to have a maximum size
References
preprocess_from_HF_dataset
#264 - Preprocessing from arrow file to load an HF dataset
Author
TevenLeScao
Parents
ea9051e0
Loading