llm-foundry
f2b7bef4
- Revert "Default to using tokenizer eos and bos in convert_text_to_mds.py (#823)" (#825)
Go
Login via GitHub
Home
Pricing
FAQ
Install
Login
via GitHub
Commit
View On
GitHub
Commit
2 years ago
Revert "Default to using tokenizer eos and bos in convert_text_to_mds.py (#823)" (#825) This reverts commit 5eb96e92e0675d98a12b7bf1147b0c4f54dcc8e6.
References
#825 - Revert "Default to using tokenizer eos and bos in convert_text_to_mds.py"
Author
irenedea
Parents
5eb96e92
Loading