llama.cpp
llama_moe_cache: run missed experts on CPU, add tuning script
#30966
Merged
Merged
llama_moe_cache: run missed experts on CPU, add tuning script #30966
am17an
merged
14 commits into
master
from
aman/moe-cache-cpu-misses
pwilkin
approved these changes
on 2026-10-22
eapache
commented
on 2026-10-23
sasa7812
commented
on 2026-10-27
ggerganov
commented
on 2026-11-02
moe-cache : compute part of the misses on the CPU
b0d27534
scripts : add moe-cache-tune.py to pick the MoE cache arguments
85ed5474
add am17an as codeowner for MoE cache
9a2e1677
moe-cache : upload all the misses until the cache is full
fab91ac8
add backend sync for async transfer queue
a571975b
moe-cache : let the scheduler copy the results of the CPU part
f75a5af2
Fix MSVC compilation
b3a17ff5
remove sync change
740fdfc1
llama-graph : extract the CPU share condition into can_cpu_share
0bd7328e
llama-graph : make llm_graph_params::moe_cache const
fed73adc
scripts : replay the ClassLRU policy in moe-cache-tune.py
0290c8da
moe-cache : pick CPU misses by CRF instead of random (#31853)
c6717826
moe-cache : remove the backend logic of the CPU part from the graph
1f17b9c9
llama-graph : return the MoE cache slots and CPU ids in a struct
c0b02cd6
ggerganov
approved these changes
on 2026-11-07
Login to write a write a comment.
Login via GitHub