llama.cpp
c4e6dd59 - llama : allow raw byte in SPM vocabs; don't crash on nl 404 (#5478)

Commit
1 year ago
llama : allow raw byte in SPM vocabs; don't crash on nl 404 (#5478) * common : don't crash if newline token is not found * common : llama_byte_to_token: allow falling back to finding just the token byte in SPM vocabs
Author
akx akx
Parents
Loading