VLM: special multimodal Tokenizer #34461
zucchini-nlp
merged
24 commits into
huggingface:main
from
zucchini-nlp:vlm-tokenizer
kinda works
d4778c91
update
0ba73ec2
add tests
116a49ec
update
661b5df2
use special tokens in processors
5ed93798
typo
a284f318
Merge remote-tracking branch 'upstream/main' into vlm-tokenizer
9ad53626
fix copies
1def73c9
fix
8dfa536d
fix moshi after rebase
200879ba
Merge branch 'main' into vlm-tokenizer
e0bf53b1
ArthurZucker
commented
on 2024-10-29
update
7e5c4bad
Merge branch 'main' into vlm-tokenizer
b77f3be3
fix tests
8b619693
update
276c55e9
Update docs/source/en/main_classes/tokenizer.md
3177519f
update docs
46a30e53
test for load time adding tokens
d7c4eb51
ArthurZucker
approved these changes
on 2024-10-30
Merge branch 'main' into vlm-tokenizer
f3b102e3
Merge branch 'main' into vlm-tokenizer
5240129f
fix some more tests which are now fetched better
58bf9e74
one more fix
76b39aae
Merge branch 'main' into vlm-tokenizer
9994aede
Merge branch 'main' into vlm-tokenizer
c21997d4
Login to write a write a comment.
Login via GitHub