transformers
c4cbed80 - Processor: prevent duplicated tokens

Commit
1 year ago
Processor: prevent duplicated tokens When using text-only LLMs, the chat template is expected to take care of adding the required special tokens, such as bos. Hence, tokenization must not include special tokens. The same contract should be honored for multimodal processors.
Author
Parents
Loading