Add sudachi and jumanpp tokenizers for bert_japanese #19043
sgugger
merged
7 commits into
huggingface:main
from
r-terada:use_another_japanese_tokenizer
add sudachipy and jumanpp tokenizers for bert_japanese
532d6b39
use ImportError instead of ModuleNotFoundError in SudachiTokenizer an…
c3d73f11
put test cases of test_tokenization_bert_japanese in one line
779acdd9
add require_sudachi and require_jumanpp decorator for testing
e555e1f0
add sudachi and pyknp(jumanpp) to dependencies
c8899694
sgugger
commented
on 2022-10-05
remove sudachi_dict_small and sudachi_dict_full from dependencies
b8bf0b0d
empty commit for ci
58a0eb64
sgugger
approved these changes
on 2022-10-05
Login to write a write a comment.
Login via GitHub