text-generation-inference
9be1db31 - feat: allow null eos and bos tokens in config (#1791)

Commit
1 year ago
feat: allow null eos and bos tokens in config (#1791) This PR resolves an issue loading in tokenizer_configs where the eos or bos token is null as in: [Qwen/Qwen1.5-72B-Chat](https://huggingface.co/Qwen/Qwen1.5-72B-Chat/blob/main/tokenizer_config.json) resolves: https://github.com/huggingface/text-generation-inference/issues/1545 and related to https://github.com/QwenLM/Qwen1.5/issues/162
Author
Parents
Loading