llama.cpp
vocab : add \p{M} to lookaheads in Tekken and GPT-4o pre-tokenizer regexes
#30845
Merged
Merged
vocab : add \p{M} to lookaheads in Tekken and GPT-4o pre-tokenizer regexes #30845
CISC
merged
1 commit into
ggml-org:master
from
tkran2:vocab-gpt4o-tekken-marks
vocab : add \p{M} to lookaheads in Tekken and GPT-4o pre-tokenizer re…
2f94020e
tkran2
changed the title 
CISC
approved these changes
on 2026-10-20
Login to write a write a comment.
Login via GitHub