PR #390 fix perplexity after c-api refactor

fix perplexity after c-api refactor #390

ggerganov merged 2 commits into ggml-org:master from Green-Sky:fix_for_perplexity_tok

preallocate a buffer of fitting size for tokenization (utils.cpp)

7b1b575f

don't create a new std::string (especially here, where it's usually l…

57fee166

Green-Sky added bug

Green-Sky requested a review from

ggerganov 2 years ago

ggerganov approved these changes on 2023-03-22

ggerganov merged 56e659a0 into master 2 years ago

Green-Sky deleted the fix_for_perplexity_tok branch 2 years ago

Reviewers

ggerganov

Assignees

No one assigned

Labels

bug

Milestone

No milestone