Add split special tokens #30772
itazap
merged
21 commits into
main
from
add_split_special_tokens
itazap
commented
on 2024-05-14
ArthurZucker
commented
on 2024-05-14
seems like `split_special_tokens` is used here
56fd608d
split special token
fbb144cb
add new line at end of file
abeaeb4e
moving split special token test to common tests
99580a9d
added assertions
a7ebe7da
test
224aae47
fixup
80adc63c
add co-author
6dbe8789
passing rest of args to gptsan_japanese, fixing tests
4000f635
removing direct comparison of fast and slow models
aec5f710
adding test support for UDOP and LayoutXLM
88b12aa6
ruff fix
fd5d1fef
readd check if slow tokenizer
bb64b8d4
modify test to handle bos tokens
24473ace
removing commented function
fb321f6b
trigger build
fd357bd6
applying review feedback - updated docstrings, var names, and simplif…
34cdba68
ruff fixes
29af7206
Update tests/test_tokenization_common.py
80b4e77b
applying feedback, comments
f5bf1093
shutil temp directory fix
2ce75694
Login to write a write a comment.
Login via GitHub