PR #41962 Patch Mistral Common Tokenizer 2

Patch Mistral Common Tokenizer 2 #41962

ArthurZucker merged 12 commits into huggingface:main from juliendenize:patch_mistral_tokenizer_2

juliendenize force pushed from ef7d5fb1 to 0b817f93 94 days ago

juliendenize changed the title ~~Patch Mistral Common Tokenizer 2: fix add_special_tokens~~ Patch Mistral Common Tokenizer 2 94 days ago

juliendenize force pushed from 0b817f93 to c308ff43 73 days ago

juliendenize force pushed from 3e79d411 to 94e07059 71 days ago

Patch Mistral Common Tokenizer 2: fix add_special_tokens

48ee877d

Fix typos

ea9e4adb

Fix typing issue

cece2d42

Make _get_validation_mode static

da40b30a

wip

2a2db548

Add int support to decode

43a84a3f

Add edge test cases

802eb4a4

fix

9e5a1473

juliendenize force pushed from 94e07059 to 9e5a1473 70 days ago

Support batch for decode

732b6a49

fix

e6ab577c

Refactor to private logic of batch_decode

c9605318

Add special properties

a720d6fc

ArthurZucker approved these changes on 2025-12-01

ArthurZucker merged aaa73254 into main 68 days ago

Reviewers

ArthurZucker

Assignees

No one assigned

Labels

None yet

Milestone

No milestone