transformers
af673222 - Improve the speed of adding tokens from added_tokens.json (#10780)

Commit

4 years ago

Improve the speed of adding tokens from added_tokens.json (#10780) * use bisect to add one token to unique_no_split_tokens * fix style

References

Author

cchen-dialpad

Parents