llm-foundry
f2b7bef4 - Revert "Default to using tokenizer eos and bos in convert_text_to_mds.py (#823)" (#825)

Commit
2 years ago
Revert "Default to using tokenizer eos and bos in convert_text_to_mds.py (#823)" (#825) This reverts commit 5eb96e92e0675d98a12b7bf1147b0c4f54dcc8e6.
Author
Parents
Loading