llama.cpp

95da4ba8 - llama: opt-in random-access mmap advice for host-resident gather tables

llama: opt-in random-access mmap advice for host-resident gather tables qwen4exp keeps per_layer_token_embd on the host: 26.8 GiB at IQ4_NL, read by ggml_get_rows as 16 gathers of ~90-170 bytes per token, spread across 16 head regions ~20M rows apart. Measured over 4.75M gathers, no two consecutive gathers land on the same 4 KiB page, so the readahead the loader asks for buys nothing here and the whole table ends up cached to serve about 4% of itself. llama_mmap applies POSIX_FADV_SEQUENTIAL, MAP_POPULATE and a whole-file POSIX_MADV_WILLNEED unconditionally. Those are right for streaming the file once into buffers and wrong for whatever stays mapped afterwards. Under LLAMA_MMAP_RANDOM the eager pull-in is skipped and the mapping is advised random once every tensor has been read, so the load itself keeps its sequential readahead. That alone drops the table to 4.4% resident but serializes one NVMe latency per gather. The second half is what pays for it: the PLE input already computes every row index for the ubatch before the graph runs, so the pages those rows fall on are handed to the kernel in one batch and the reads overlap. POSIX_MADV_WILLNEED on POSIX, PrefetchVirtualMemory on Windows, which takes the discontiguous ranges in a single call. Off by default and off for every other model: the batched prefetch keys off "this mapping was advised random", which nothing sets unless the user opts in. -c 512 --chunks 60, cold, IQ1_S, mean of 3: default 35.3 s 26.82 GiB resident (100%) advice only 104.5 s 1.19 GiB resident (4.4%) advice + prefetch 34.2 s 1.19 GiB resident (4.4%) PPL 4.2346 +/- 0.07862 in all three. IQ1_S KLD is unchanged in every field, including Mean KLD 0.396070 +/- 0.001931 and Same top p 77.325%.