transformers
65a89a89 - Fix BasicTokenizer to respect `never_split` parameters (#2557)

Commit

5 years ago

Fix BasicTokenizer to respect `never_split` parameters (#2557) * add failing test * fix call to _run_split_on_punc * format with black

References

Author

DeNeutoy

Committer

julien-c

Parents