llama.cpp
83c2d012 - spec : save the dynamic ngram cache file (#22055)

Commit
30 days ago
spec : save the dynamic ngram cache file (#22055) * spec : save the dynamic ngram cache file * save the cache at the common_speculative_impl_ngram_cache destruction * remove save_static/save_dynamic flags as well as the relevant TODO * if the dynamic cache file does not exist, start with an empty cache that is written on exit. if it exists but cannot be read, abort instead of overwriting it. * spec : fix draft_one not updating the ngram cache history correctly * common_ngram_cache_update received only the new tokens with none of the tokens before them causing ngrams ending at the new token to not being formed, leading the context cache to only ever hold the prompt * fix: pass the full history to common_ngram_cache_update with nnew set to the number of new tokens the way llama-lookup does and also reset cache_size to 0 on each new request --------- Co-authored-by: Petros Sideris <petros.sideris@nokia.com>
Author
Parents
Loading