transformers
efb2ba66 - Better handling missing SYS in llama conversation tokenizer (#24997)

Commit
2 years ago
Better handling missing SYS in llama conversation tokenizer (#24997) * Better handling missing SYS in llama conversation tokenizer The existing code failed to add SYS if the conversation has history without SYS, but did modify the passed conversation as it did. Rearrange the code so modification to the conversation object are taken into account for token id generation. * Fix formatting with black * Avoid one-liners * Also fix fast tokenizer * Drop List decl
Author
Parents
Loading