llama.cpp
4c4a0f0a - llama : reserve graph nodes for the MoE cache (#32636)

Commit
2 hours ago
llama : reserve graph nodes for the MoE cache (#32636) The MoE cache adds a slot lookup to every MoE layer, and with --moe-cache-cpu-share a second expert path, but graph_max_nodes() did not account for it. Models on the default 8*n_tensors budget with few tensors per layer (cohere2moe: 443 tensors, 49 layers) overflow the graph context when the cache is combined with a quantized KV cache (K/V rotation), aborting in ggml_new_tensor with GGML_ASSERT(obj_new). Fixes #32584 Assisted-by: Claude Opus 5.5
Author
Parents
Loading