transformers
0b3c2732 - Add token-level chat input sanitization

Commit
3 days ago
Add token-level chat input sanitization Decouple template structure from untrusted content generation instead of escaping or stripping special tokens at the text level. Jinja dictates the formatting topology, the tokenizer enforces the safety boundary. `apply_chat_template(..., sanitize_special_tokens=True)` wraps message content in `UntrustedInput`, which encodes it with `split_special_tokens=True`, so the tokenizer treats it strictly as literal string primitives and an injected `<|im_end|>` becomes ordinary text tokens. Templates can call `{{ message.content.tokenize_securely() }}` explicitly, but plain `{{ message.content }}` is protected too. The tokenizer also decides whether content needs isolating at all: content that encodes identically with and without special-token matching holds no control tokens and renders as plain text, so an ordinary chat encodes exactly as before.
Author
Parents
Loading