transformers
31565ff0 - Add sentencepiece to BertJapaneseTokenizer (#19769)
Commit
View On GitHub
Commit
3 years ago
Add sentencepiece to BertJapaneseTokenizer (#19769) * support sentencepiece for bertjapanesetokenizer * add test vocab file for sentencepiece, bertjapanesetokenizer * make BasicTokenizer be identical to transformers.models.bert.tokenization_bert.BasicTokenizer * fix missing of \n in comment * fix init argument missing in tests * make spm_file be optional, exclude spiece.model from tests/fixtures, and add description comments * make comment length less than 119 * apply doc style check
Author
conan1024hao conan1024hao
Parents
2ebf4e6a
FAQ Terms Privacy Refunds Impressum
Loading