transformers
315f464b - [tokenizers] Several small improvements and bug fixes (#5287)

Commit
5 years ago
[tokenizers] Several small improvements and bug fixes (#5287) * avoid recursion in id checks for fast tokenizers * better typings and fix #5232 * align slow and fast tokenizers behaviors for Roberta and GPT2 * style and quality * fix tests - improve typings
Author
Parents
Loading