llama.cpp
f43070aa - kv-cache: clear the cache once when restoring a whole context

Commit
6 days ago
kv-cache: clear the cache once when restoring a whole context state_read walks the streams of the cache in turn, and for a whole-context restore each stream went through state_read_meta, which starts by calling clear(). clear() resets every stream at once, so each stream after the first threw away the streams already restored, and the K/V buffers with them. A non-unified cache holds one stream per sequence, so a context saved with N sequences in it came back with only the sequence in the last stream that carried any cells - the highest sequence id. A unified cache has one stream and never showed it. The cache is now emptied once, before the loop, which is what a whole-context restore means. A blob whose streams are all empty now empties the cache as well, where before it left the old contents in place.
Author
Committer
Parents
Loading