transformers
d0deac8f - Add chat input sanitization (#47386)

Commit
37 days ago
Add chat input sanitization (#47386) * Cleanup and huge simplification to an alternate approach * Cover all control tokens, not just special tokens * Comment cleanup * Comment cleanup * Refactor everything out into chat_template_utils, couple of cleanups * Add FIXME --------- Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
Author
Parents
Loading