llama.cpp
c4e6dd59
- llama : allow raw byte in SPM vocabs; don't crash on nl 404 (#5478)
Go
Login via GitHub
Home
Pricing
FAQ
Install
Login
via GitHub
Commit
View On
GitHub
Commit
1 year ago
llama : allow raw byte in SPM vocabs; don't crash on nl 404 (#5478) * common : don't crash if newline token is not found * common : llama_byte_to_token: allow falling back to finding just the token byte in SPM vocabs
References
#5478 - common : allow raw byte in SPM vocabs; don't crash if newline token is not found
Author
akx
Parents
037259be
Loading