vllm
[Bugfix]: Fix `TokenizerLike` interface
#30009
Merged
Overview
Commits
8
Changes
View On GitHub

[Bugfix]: Fix `TokenizerLike` interface #30009

vllm-bot merged 8 commits into vllm-project:main from ROCm:fix_mixtral_tokenizer
Rohan138
Rohan138 add num_special_tokens_to_add
59ba19e6
gemini-code-assist
gemini-code-assist commented on 2025-12-04
Rohan138 add num_special_tokens_to_add
a3014182
Rohan138 add num_special_tokens_to_add
fe65eab5
DarkLight1337
DarkLight1337 commented on 2025-12-04
Rohan138 switch all type hints to use TokenizerLike
491ae446
Rohan138 Rohan138 requested a review from WoosukKwon WoosukKwon 75 days ago
Rohan138 Rohan138 requested a review from youkaichao youkaichao 75 days ago
Rohan138 Rohan138 requested a review from robertgshaw2-redhat robertgshaw2-redhat 75 days ago
Rohan138 Rohan138 requested a review from mgoin mgoin 75 days ago
Rohan138 Rohan138 requested a review from tlrmchlsmth tlrmchlsmth 75 days ago
Rohan138 Rohan138 requested a review from houseroad houseroad 75 days ago
Rohan138 Rohan138 requested a review from hmellor hmellor 75 days ago
Rohan138 Rohan138 requested a review from yewentao256 yewentao256 75 days ago
Rohan138 Rohan138 requested a review from ProExpertProg ProExpertProg 75 days ago
mergify mergify added performance
Rohan138
Rohan138 commented on 2025-12-04
Rohan138
Rohan138 commented on 2025-12-04
Rohan138
Rohan138 commented on 2025-12-04
Rohan138 revert get_requests
29f266ac
Rohan138 revert get_requests
e4b64d04
Rohan138
Rohan138 commented on 2025-12-04
Rohan138
Rohan138 commented on 2025-12-04
Rohan138
Rohan138 commented on 2025-12-04
Rohan138 fix tokenizer-mode argparse
4e4c7c26
Rohan138 Rohan138 requested a review from DarkLight1337 DarkLight1337 75 days ago
Rohan138 Rohan138 changed the title [Bugfix]: Add `num_special_tokens_to_add` to MistralTokenizer [Bugfix]: Add `num_special_tokens_to_add` to MistralTokenizer, update PretrainedTokenizerBase -> TokenizerLike 75 days ago
Rohan138 Rohan138 changed the title [Bugfix]: Add `num_special_tokens_to_add` to MistralTokenizer, update PretrainedTokenizerBase -> TokenizerLike [Bugfix]: Fix `TokenizerLike` interface 75 days ago
DarkLight1337
mergify
Rohan138 cleanup; drop choices; fix lint; signoff
6f67bc51
Rohan138 Rohan138 force pushed to 6f67bc51 74 days ago
DarkLight1337
DarkLight1337 approved these changes on 2025-12-06
DarkLight1337 DarkLight1337 enabled auto-merge (squash) 74 days ago
github-actions github-actions added ready
vllm-bot vllm-bot merged 40a046cd into main 74 days ago
gshtras gshtras deleted the fix_mixtral_tokenizer branch 70 days ago

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
performance ready
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum