transformers
d30cbaf5 - Merge branch 'master' into iterative_split_on_token

Commit

6 years ago

Merge branch 'master' into iterative_split_on_token

References

#1027 - Re-implemented tokenize() iteratively in PreTrainedTokenizer.

Author

thomwolf

thomwolf

Parents

Loading