transformers
68bb33d7 - Fixing OPT fast tokenizer option. (#18753)

Commit
3 years ago
Fixing OPT fast tokenizer option. (#18753) * Fixing OPT fast tokenizer option. * Remove dependency on `pt`. * Move it to GPT2 tokenization tests. * Added a few tests.
Author
Parents
Loading