llama.cpp
d8d92973
- vocab : add \p{M} to lookaheads in Tekken and GPT-4o pre-tokenizer regexes (#30845)
Go
Login via GitHub
Home
Pricing
FAQ
Install
Login
via GitHub
Commit
View On
GitHub
Commit
25 days ago
vocab : add \p{M} to lookaheads in Tekken and GPT-4o pre-tokenizer regexes (#30845)
References
#30845 - vocab : add \p{M} to lookaheads in Tekken and GPT-4o pre-tokenizer regexes
Author
tkran2
Parents
7879491c
Loading