llama.cpp
Ignore invalid UTF-8 input in the BPE tokenizer
#11729
Merged

Commits
  • ignore invalid UTF-8 input in the BPE tokenizer
    cfillion committed 1 year ago
Loading