llama.cpp
unicode : add opt-in NFC normalization for Unigram tokenizers
#30390

Merged

unicode : add opt-in NFC normalization for Unigram tokenizers #30390

ggerganov merged 3 commits into ggml-org:master from sandrohanea:sandroh/unigram-nfc

unicode : add opt-in NFC normalization for Unigram tokenizers

2e326a2f

sandrohanea requested a review from

ggerganov 17 days ago

sandrohanea requested a review from

CISC 17 days ago

github-actions added testing

github-actions added conversion

Fixing initial bot comments + linter error (using existing out instea…

1fa23096

CISC commented on 2026-10-12

removed test_nfc_tokenizer

c8ff8f20

CISC approved these changes on 2026-10-12

CISC added merge ready

ggerganov merged 10da8c46 into master 15 days ago

sandrohanea deleted the sandroh/unigram-nfc branch 13 days ago

Reviewers

CISC

ggerganov

Assignees

No one assigned

Labels

testing merge ready conversion

Milestone

No milestone