llm-foundry
5eb96e92 - Default to using tokenizer eos and bos in convert_text_to_mds.py (#823)

Commit

2 years ago

Default to using tokenizer eos and bos in convert_text_to_mds.py (#823) * default to using tokenizer eos and bos * pyright fixes

References

Author

irenedea

Parents