llm-foundry

f65b07e8 - Tiktoken wrapper add_eos_token option (#681)

Tiktoken wrapper add_eos_token option (#681) * add add_eos_token arg to tiktoken wrapper * add add_eos_token arg to tiktoken wrapper * yapf * encode_plus tests --------- Co-authored-by: Daniel King <43149077+dakinggg@users.noreply.github.com>