Bump tokenizers dependency to 0.5.1 - SemanticDiff

Commit

6 years ago

Bump tokenizers dependency to 0.5.1 Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>

References

#2933 - Fix for fast tokenizers save_pretrained compatibility with Python.

Author

mfuntowicz

mfuntowicz

Committer

mfuntowicz

mfuntowicz

Parents

Loading