Add streams support (#946)
* add convert
* fix
* fix convert
* add jsonl
* revert setup
* test precommit
* pre-commit
* test pre-commit
* v0
* review comments
* temporarily trigger test
* test
* add convert
* fix
* v0
* fix
* fix MDS write
* streams support
* fake commit
* fix setup
* format
* add back arxiv
* trigger test
* review comments
* temporarily trigger test
* test
* add convert
* fix
* fix
* fix MDS write
* format
* trigger test
* fix
* format
* resolve conflicts
* add back jsonl
* fix yaml
* comments
* format
* comments
* comments
* add unit test
* comments
* comments
* merge
* format
* typo
* Update llmfoundry/data/finetuning/dataloader.py
Co-authored-by: Daniel King <43149077+dakinggg@users.noreply.github.com>
---------
Co-authored-by: Daniel King <43149077+dakinggg@users.noreply.github.com>
