PR #21343 fix: Gemma4 tokenizer

fix: Gemma4 tokenizer #21343

ngxson merged 3 commits into ggml-org:master from pwilkin:gemma4-tokenizer-fix

seems to work

e1b7abf2

pwilkin requested a review from

CISC 81 days ago

pwilkin requested a review from

ggerganov 81 days ago

ngxson approved these changes on 2026-04-03

fix case with new line

5ec574fd

gemma 4: fix pre tok regex

5959c504

github-actions added python

danielhanchen approved these changes on 2026-04-03

ggerganov approved these changes on 2026-04-03

ngxson merged b069b10a into master 81 days ago

Reviewers

ggerganov

ngxson

danielhanchen

CISC

Assignees

No one assigned

Labels

python

Milestone

No milestone