diffusers
780e112c - [modular] auto offload: tunable memory reserve, OOM retry, non-disruptive add/remove

Commit
100 days ago
[modular] auto offload: tunable memory reserve, OOM retry, non-disruptive add/remove - each offloading decision checks the memory actually available on the device (`mem_get_info` free plus the allocator's reusable cache) and keeps a tunable `memory_reserve` of it free; `memory_reserve_margin` is renamed to `memory_reserve` - if a forward still runs out of device memory, offload the smallest model on the device and retry, escalating one model at a time until it fits (`retry_on_oom=True` by default); when nothing else is resident, point at group offloading - adding or removing a component attaches/detaches a single hook instead of re-running `enable_auto_cpu_offload` and offloading the resident working set - add a `simulate_accelerator_memory` test util that makes a large accelerator behave like a smaller card, so offloading can be exercised with real models Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Author
Parents
Loading