transformers
c6379858 - bump tokenizers, fix added tokens fast (#32535)

Commit
1 year ago
bump tokenizers, fix added tokens fast (#32535) * update based on tokenizers release * update * nits * update * revert re addition * don't break that yet * fmt * revert unwanted * update tokenizers version * update dep table * update * update in conversion script as well * some fix * revert * fully revert * fix training * remove set trace * fixup * update * update
Author
Parents
Loading