llama.cpp
server: add router device memory margin parameter for dynamic unloading
#21231
Open
Open
server: add router device memory margin parameter for dynamic unloading #21231
0cc4m
wants to merge
18 commits into
master
from
0cc4m/server-memory-limit
ngxson
commented
on 2026-03-31
ngxson
commented
on 2026-03-31
0cc4m
changed the title
ggerganov
commented
on 2026-04-02
ggerganov
commented
on 2026-04-16
ggerganov
commented
on 2026-05-04
ggerganov
commented
on 2026-05-04
ggerganov
commented
on 2026-05-04
ngxson
commented
on 2026-05-13
danbev
commented
on 2026-05-21
ggerganov
commented
on 2026-06-16
server: add --models-memory-max parameter to allow dynamically unload…
7369ec06
estimate with to-be-loaded model size included
b6fa94e3
use no_alloc to get memory requirements for model load
1019aadc
only set model memory_mb if not previously calculated
4f2efcbb
use memory margin instead of total size limit, apply to each device s…
2c733776
add server memory debug logging
6ecddda7
move llama_context_device_memory function to llama-ext.h
626fd177
fix model count exceeded check
5b2ab4ed
improve memory_per_device map naming
9c6e4218
improve variable naming, fix style
719ab427
also strip models memory margin from child processes
f24b9d53
cont : clean-up
0ade9b8c
replace device memory map with buft memory map. Use llama_get_memory_…
c5847772
extract duplicated check into helper function
bd39f024
move model memory estimation to subprocess
c03917ad
precompute name->buft map, map GPU host types to CPU buft
1201397c
cleanup unused variable
db5d5b3b
remove duplicated init calls
d1b5a682
Login to write a write a comment.
Login via GitHub