llama.cpp
Ignore invalid UTF-8 input in the BPE tokenizer
#11729

Merged

Ignore invalid UTF-8 input in the BPE tokenizer #11729

ggerganov merged 1 commit into ggml-org:master from cfillion:bpe-tokenize-mojibake-noexcept

ggerganov approved these changes on 2025-02-07

slaren commented on 2025-02-07

ignore invalid UTF-8 input in the BPE tokenizer

cff1c3bc

cfillion force pushed to cff1c3bc 1 year ago

slaren approved these changes on 2025-02-07

ggerganov merged 2d219b38 into master 1 year ago

Reviewers

slaren

ggerganov

Assignees

No one assigned

Labels

None yet

Milestone

No milestone