llama : make llm_tokenizer more private - SemanticDiff

Commit

1 year ago

llama : make llm_tokenizer more private ggml-ci

References

gg/tokenizer-cleanup

#9449 - vocab: refactor tokenizer to reduce the overhead of creating multi times tokenizer

Author

ggerganov

ggerganov

Parents

Loading