llama.cpp
vocab: add gemma4 tokenizer tests, fix edge case
#21534

Merged

vocab: add gemma4 tokenizer tests, fix edge case #21534

pwilkin merged 3 commits into ggml-org:master from pwilkin:gemma4-yatf

YATF (Yet Another Tokenizer Fix) for Gemma 4. With tests!

320c29c2

pwilkin requested a review from

ggerganov 6 days ago

pwilkin requested a review from

CISC 6 days ago

pwilkin requested a review from

aldehir 6 days ago

pwilkin requested a review from

ngxson 6 days ago

github-actions added testing

github-actions added python

Remove unnecessary hash from update script.

0e98596d

ggerganov approved these changes on 2026-04-09

pwilkin changed the title ~~YATF (Yet Another Tokenizer Fix) for Gemma 4. With tests!~~ vocab: add gemma4 tokenizer tests, fix edge case 4 days ago

taronaeo approved these changes on 2026-04-09

ggerganov commented on 2026-04-09

minor: move constant

d60a08f1

ServeurpersoCom approved these changes on 2026-04-09

pwilkin merged 0ec191e1 into master 4 days ago

Reviewers

ServeurpersoCom

ggerganov

taronaeo

CISC

aldehir

ngxson

Assignees

No one assigned

Labels

testing python

Milestone

No milestone

llama.cpp vocab: add gemma4 tokenizer tests, fix edge case #21534 Merged

vocab: add gemma4 tokenizer tests, fix edge case #21534

llama.cpp
vocab: add gemma4 tokenizer tests, fix edge case
#21534

Merged