diffusers
319e1e01 - exclude only the SigLIP image encoder from leaf-level group offloading

Commit
4 days ago
exclude only the SigLIP image encoder from leaf-level group offloading `test_pipeline_level_group_offloading_inference` was skipped outright for HunyuanVideoFramepack because `image_encoder` is a `SiglipVisionModel`, whose attention pooling head wraps a `torch.nn.MultiheadAttention`. That hands `self.out_proj.weight` to `torch.nn.functional.multi_head_attention_forward` instead of calling `self.out_proj`, so the leaf-level onload hook on `out_proj` never fires and its weights stay on the offload device. Add a `group_offloading_leaf_level_exclude_modules` knob to the old-style `PipelineTesterMixin` and the new-style `BasePipelineTesterConfig` (empty by default, so no behavior change elsewhere), pass it through to `enable_group_offload(exclude_modules=...)` in both implementations of the test, and set it to `["image_encoder"]` for framepack instead of skipping. Block-level offloading is unaffected — the whole head is onloaded as one unmatched module — hence the level in the name. The test now passes and covers leaf-level offloading of the transformer, VAE and both text encoders. The VAE is coverage nothing else provided: `test_group_offloading_inference` deliberately excludes `vae` and `image_encoder`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Author
Committer
Parents
Loading