PR #390 fix perplexity after c-api refactor

Commits

preallocate a buffer of fitting size for tokenization (utils.cpp)

Green-Sky committed 2 years ago
don't create a new std::string (especially here, where it's usually large)

Green-Sky committed 2 years ago

llama.cpp fix perplexity after c-api refactor #390 Merged