transformers
03ae1f06 - change the way sentinel tokens can retrived (#20373)

Commit
3 years ago
change the way sentinel tokens can retrived (#20373) * change the way sentinel tokens can retrived * Fix line length for doc string * Fix line length for doc string * Add more stronger test for t5 tokenization * Format file changes * Make a stronger test for filtering sentinel tokens * fix file format issues
Author
Parents
Loading