Add sudachi_projection option to BertJapaneseTokenizer #28503
ArthurZucker
merged
23 commits into
huggingface:main
from
bert_japanese_sudachi_projection
add sudachi_projection option
006023e2
Upgrade sudachipy>=0.6.8
37c8ee36
add a test case for sudachi_projection
47c43d43
Compatible with older versions of SudachiPy
d5e73018
make fixup
84893e1a
make style
90efa1b1
error message for unidic download
475af996
hiroshi-matsuda-rit
changed the title
revert jumanpp test cases
5093754b
hiroshi-matsuda-rit
changed the title
format options for sudachi_projection
b2974306
format options for sudachi_split_mode and sudachi_dict_type
35cc83cd
comment
c6144973
ArthurZucker
commented
on 2024-01-16
add tests for full_tokenizer kwargs
599116dc
pass projection arg directly
8318f3ce
require_sudachi_projection
d64e2d6d
make style
e27284ac
ArthurZucker
commented
on 2024-01-23
revert upgrade sudachipy
ece01711
check is_sudachi_projection_available()
eda9e0d9
revert dependency_version_table and bugfix
34762a9e
style format
d617ebdf
Merge branch 'main' into bert_japanese_sudachi_projection
b4792c69
ArthurZucker
commented
on 2024-01-26
simply raise ImportError
6c5443f1
simply raise ImportError
c6172769
Merge remote-tracking branch 'origin/main' into bert_japanese_sudachi…
705e7041
ArthurZucker
approved these changes
on 2024-02-13
Login to write a write a comment.
Login via GitHub