transformers
Add Siglip2Tokenizer to enforce training-time text preprocessing defaults
#43101

Merged

Add Siglip2Tokenizer to enforce training-time text preprocessing defaults #43101

ArthurZucker merged 21 commits into huggingface:main from vaibhav-research:siglip2_text_embedding

fancyerii approved these changes on 2026-01-05

zucchini-nlp commented on 2026-01-05

vaibhav-research changed the title ~~43054: siglip2 text embedding test and document update~~ 43054: siglip2 text embedding updated to default to lowercase 213 days ago

zucchini-nlp commented on 2026-01-08

vaibhav-research changed the title ~~43054: siglip2 text embedding updated to default to lowercase~~ 43054: siglip2 text embedding default to lowercase 212 days ago

vaibhav-research changed the title ~~43054: siglip2 text embedding default to lowercase~~ 43054: Add Siglip2Tokenizer to enforce training-time text preprocessing defaults 212 days ago

ArthurZucker commented on 2026-01-09

zucchini-nlp commented on 2026-01-09

vaibhav-research force pushed from 368b6a4b to e25cc6c4 205 days ago

vaibhav-research force pushed from e25cc6c4 to e1aafd13 205 days ago

zucchini-nlp commented on 2026-01-16

vaibhav-research force pushed from 9993a749 to b1674823 202 days ago

vaibhav-research force pushed from 0c8f75d7 to 64e9c82b 199 days ago

zucchini-nlp commented on 2026-01-22

vaibhav-research force pushed from 0379e11c to d1500569 198 days ago

zucchini-nlp approved these changes on 2026-01-23

ArthurZucker commented on 2026-01-23

Add Siglip2Tokenizer with lowercase normalizer and tests

29f1c43d

updated correct gemmatokenizer reference

f4419725

Updated lowercase logic to keep it simple

4ff5b4bc

added inegration test and removed _unk_id from tokenization

a5b330d5

ruff fixes

e648a65e

vaibhav-research force pushed from 4131a538 to e648a65e 196 days ago

updated the doc

4ed6d375

zucchini-nlp approved these changes on 2026-01-26

Updated doc and conversion script

c913782e