llama.cpp
ba06b2de - tokenize : add --no-parse-special option

Commit

1 year ago

tokenize : add --no-parse-special option This should allow more easily explaining how parse_special affects tokenization.

References

Author

compilade

Committer

compilade

Parents