llm-foundry
Default to using tokenizer eos and bos in convert_text_to_mds.py
#823
Merged
Merged
Default to using tokenizer eos and bos in convert_text_to_mds.py #823
irenedea
merged
2 commits into
mosaicml:main
from
irenedea:tok
default to using tokenizer eos and bos
754fe7fc
pyright fixes
50588ac0
Skylion007
approved these changes
on 2023-12-28
Login to write a write a comment.
Login via GitHub