ggml-org/llama.cpp

Pull Requests Commits

server: keep router model refcount to avoid unloading models that have running requests

0cc4m committed 43 days ago

a5355a02

load directly from downloaded state

0cc4m committed 44 days ago

cf0ebc4e

handle models that need to be downloaded before estimation

0cc4m committed 44 days ago

b1623a61

cont : clean-up

ggerganov committed 44 days ago

1a8aec0a

also strip models memory margin from child processes

0cc4m committed 44 days ago

eb2cf73f

improve variable naming, fix style

0cc4m committed 44 days ago

69e30861

improve memory_per_device map naming

0cc4m committed 44 days ago

173da43c

fix model count exceeded check

0cc4m committed 44 days ago

75000630

move llama_context_device_memory function to llama-ext.h

0cc4m committed 44 days ago

ba2521c6

add server memory debug logging

0cc4m committed 44 days ago

51538c1f

use memory margin instead of total size limit, apply to each device separately

0cc4m committed 44 days ago

56122b35

only set model memory_mb if not previously calculated

0cc4m committed 44 days ago

9b5af58a

use no_alloc to get memory requirements for model load

0cc4m committed 44 days ago

2603b4c5

estimate with to-be-loaded model size included

0cc4m committed 44 days ago

777395f6

server: add --models-memory-max parameter to allow dynamically unloading models when they exceed a memory size threshold

0cc4m committed 44 days ago

8e8e2007

vulkan: Support F16 OP_FILL (#22177)

jeffbolznv committed 44 days ago

Verified 82209efb

mtmd: correct mtmd_decode_use_mrope() (#22188)

ngxson committed 44 days ago

Verified 9998d88b

llama-ext : fix exports (#22202)

ggerganov committed 44 days ago

Verified cd03ec76

ggerganov committed 44 days ago

4889afba

ggml : bump version to 0.10.0 (ggml/1463)

ggerganov committed 44 days ago

041fe83d

fit-params : refactor + add option to output estimated memory per device (#22171)

ggerganov committed 44 days ago

Verified cfe9838d

server : fix hardcoded proxy connection timeout in router mode (#18760) (#22003)

xris99 committed 44 days ago

Verified ff6b1062

ggml-cuda: flush legacy pool on OOM and retry (#22155)

leonardHONG committed 45 days ago

Verified 97895129

mtmd: correct get_n_pos / get_decoder_pos (#22175)

ngxson committed 45 days ago

Verified 86f8daac

server : remove /api endpoints (#22165)

ggerganov committed 45 days ago

Verified cf8b0dbd

Tensor-parallel: Fix delayed AllReduce on Gemma-4 MoE (#22129)

gaugarg-nv committed 45 days ago

Verified fd6ae4ca

TP: fix 0-sized tensor slices, AllReduce fallback (#21808)

JohannesGaessler committed 45 days ago

Verified fb19f94c

ggml-cpu: Optimized x86 and generic cpu q1_0 dot (follow up) (#21636)

pl752 committed 45 days ago

Verified 7f251fdb

ggml-webgpu: updated matrix-vector multiplication (#21738)

neha-ha committed 45 days ago

Verified a6cc43c2

mtmd: refactor mtmd_decode_use_mrope (#22161)

ngxson committed 45 days ago

Verified a6789166

Older