llama.cpp
9a55ffe6 - tokenize : add --no-parse-special option (#8423)

Commit

1 year ago

tokenize : add --no-parse-special option (#8423) This should allow more easily explaining how parse_special affects tokenization.

References

Author

compilade

Parents