Add `SplinterTokenizer` unit test #32652
ArthurZucker
merged
5 commits into
huggingface:main
from
ariepratama:splinter_tokenizer_test
add unit tests for splinter_tokenizer
781cbaf2
add unit test for splinter tokenizer, pass in the question_token to b…
d715a44a
remove unused import
4230de46
remove vocab_splinter.txt, add Copied from, use fmt:on and fmt:off to…
d721b3b3
ArthurZucker
commented
on 2024-09-27
remove all the spaces
ccb1ffd8
Login to write a write a comment.
Login via GitHub