llama.cpp
98edea60 - llama : add UNKNOWN tokens in the special tokens cache

Commit

2 years ago

llama : add UNKNOWN tokens in the special tokens cache

References

#8228 - llama : fix pre-tokenization of non-special added tokens

Author

compilade

compilade

Parents

Loading