transformers
aaa73254 - Patch Mistral Common Tokenizer 2 (#41962)

Commit
13 days ago
Patch Mistral Common Tokenizer 2 (#41962) * Patch Mistral Common Tokenizer 2: fix add_special_tokens * Fix typos * Fix typing issue * Make _get_validation_mode static * wip * Add int support to decode * Add edge test cases * fix * Support batch for decode * fix * Refactor to private logic of batch_decode * Add special properties
Author
Parents
Loading