huggingface/transformers

Pull Requests Commits
add test tokenizer without sentencepiece for huggyllama
itazap committed 1 year ago
1de64574
fixing test
itazap committed 1 year ago
78bccc63
fix test and ruff
itazap committed 1 year ago
423bce29
updating feedback 2
itazap committed 1 year ago
80ba03ed
remove commont test and ruff
itazap committed 1 year ago
632a21cb
applying feedback to simplify pretokenizer and normalizer updates, + update decoder
itazap committed 1 year ago
f48f1b46
update tests to only test legacy=False
itazap committed 1 year ago
d50bbc8a
reverting sentencepiece test
itazap committed 1 year ago
d2990afe
Trigger CI attempt
itazap committed 1 year ago
83af87b5
Trigger CI attempt
itazap committed 1 year ago
20dc2bb5
removing unused imports
itazap committed 1 year ago
61fd1023
modify tokenization test to test without sentencepiece
itazap committed 1 year ago
ebf781ec
added llama and t5 tests for legacy prefix space in fast tokenizer
itazap committed 1 year ago
2e054a7a
fix case for pretokenizer
itazap committed 1 year ago
3e645791
set pretokenizer if unset
itazap committed 1 year ago
610c319b
remove t5 copy from semaless m4
itazap committed 1 year ago
fafe5b79
ruff
itazap committed 1 year ago
c36e6168
ruff
itazap committed 1 year ago
f8f8f388
remove test:(
itazap committed 1 year ago
406ccde2
add sentencepiece req
itazap committed 1 year ago
584f899d
fix sentencepiece case
itazap committed 1 year ago
b0d4456a
update_test
itazap committed 1 year ago
b9fd8825
ruff llama
itazap committed 1 year ago
0f1c5433
adding no sentencepiece test
itazap committed 1 year ago
9b8257ed
update cases for pretokenizer updating
itazap committed 1 year ago
328c12b6
consider bytelevel
itazap committed 1 year ago
19ca2af5
remove force_froms_slow
itazap committed 1 year ago
2e2db921
fix cases
itazap committed 1 year ago
aeca3f1a
case fix
itazap committed 1 year ago
014d3961
fix merge
itazap committed 1 year ago
50ec1225
Terms Of Use Privacy Policy Refund Policy Impressum