transformers
5d9c61e3 - tp loading: per-batch pinned-memory reads + pipeline overlap

Commit
103 days ago
tp loading: per-batch pinned-memory reads + pipeline overlap - Thread pool reads safetensors → CPU pin_memory() per batch (not all upfront). Next batch's disk reads overlap with current batch's scatter. - Sync DMA (pin_memory + .to(device)) — async DMA on dedicated stream caused silent data corruption; reverted to synchronous for correctness. The pipeline overlap still works at the CPU level (thread pool disk reads run concurrently with GPU scatter). - Removed dead _redistribute_async function and unused dma_stream. - Correctness: synthetic OK, Qwen2.5-14B generates correctly.
Author
Parents
Loading