Update tokenization_openai.py - SemanticDiff

Commit

6 years ago

Update tokenization_openai.py

References

#1059 - Better use of spacy tokenizer in open ai and xlm tokenizers

Author

GuillemGSubies

GuillemGSubies

Parents

Loading