transformers
Patch MistralCommonTokenizer
#41439
Merged
Overview
Commits
10
Changes
View On GitHub

Patch MistralCommonTokenizer #41439

ArthurZucker merged 10 commits into huggingface:main from juliendenize:patch_mistral_tokenizer
juliendenize
juliendenize
juliendenize commented on 2025-10-08
patrickvonplaten
patrickvonplaten commented on 2025-10-08
patrickvonplaten
patrickvonplaten commented on 2025-10-08
patrickvonplaten
patrickvonplaten approved these changes on 2025-10-08
juliendenize juliendenize changed the title Fix token_to_id and add add_generation_prompt Patch MistralCommonTokenizer 153 days ago
github-actions
ArthurZucker
ArthurZucker approved these changes on 2025-10-09
HuggingFaceDocBuilderDev
juliendenize Fix token_to_id and add add_generation_prompt
3a6d15ea
juliendenize Fix spm download
bb988cfd
juliendenize Refactor spm
2854a324
juliendenize Try another possibly non-gated spm
8c35dfba
juliendenize Improve get_vocab
eba2e7d5
juliendenize lint
fb5f2727
juliendenize Improve get_vocab
b477ba49
juliendenize Add warn to piece_to_id
e0ffb72d
juliendenize Improve from_pretrained raise and revert model spm
904ecbb8
juliendenize Revert fast
8938039f
juliendenize juliendenize force pushed from bbe0c0f2 to 8938039f 151 days ago
ArthurZucker ArthurZucker enabled auto-merge (squash) 147 days ago
ArthurZucker ArthurZucker merged 0566b6f5 into main 147 days ago

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum