llama.cpp
4663133f - ggml: allow input async copies

Commit
7 days ago
ggml: allow input async copies Copy the host inputs of a split with set_tensor_async followed by a single synchronize, instead of one blocking copy per input. This is opt-in per device through the new async_host_input cap, which only CUDA sets. Other backends keep the current path. The async path is used without pipeline parallelism only. With pipeline parallelism, the input backend also waits on the slot event before an async copy, since CUDA peer copies run on the source stream. This replaces the synchronize on graph reuse in process_ubatch.
Author
Parents
Loading