transformers
e0d888d3 - TransfoXLTokenizerFast uses a json vocabulary file + warning about incompatibility between Python and Rust

Commit

6 years ago

TransfoXLTokenizerFast uses a json vocabulary file + warning about incompatibility between Python and Rust Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>

References

#2933 - Fix for fast tokenizers save_pretrained compatibility with Python.

Author

mfuntowicz

Committer

mfuntowicz

Parents

11269fb6

transformers e0d888d3 - TransfoXLTokenizerFast uses a json vocabulary file + warning about incompatibility between Python and Rust

transformers
e0d888d3 - TransfoXLTokenizerFast uses a json vocabulary file + warning about incompatibility between Python and Rust