Update tools/preprocess_data.py

Commit

3 years ago

Update tools/preprocess_data.py Co-authored-by: Thomas Wang <24695242+thomasw21@users.noreply.github.com>

References

Author

Muennighoff

Parents

Megatron-DeepSpeed 63daa46f - Update tools/preprocess_data.py