llama.cpp
d7e852c1 - Tokenizer SPM fixes for phi-3 and llama-spm (bugfix) (#7425)

Commit

1 year ago

Tokenizer SPM fixes for phi-3 and llama-spm (bugfix) (#7425) * Update brute force test: add_special * Update brute force test: default values for add_bos_token and add_eos_token * Enable rtrim when pre-inserting BOS Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Revert "server : fix test regexes"

References

#7425 - Tokenizer SPM fixes for phi-3 and llama-spm (bugfix)

Author

jaime-m-p

Parents

917dc8cf

Files5

convert-hf-to-gguf.py
examples/server/tests/features
- server.feature
- slotsave.feature
llama.cpp
tests
- test-tokenizer-random.py

llama.cpp d7e852c1 - Tokenizer SPM fixes for phi-3 and llama-spm (bugfix) (#7425)

llama.cpp
d7e852c1 - Tokenizer SPM fixes for phi-3 and llama-spm (bugfix) (#7425)