transformers
dd623c8c - Fix flaky DeepseekV32 / Whisper / HiggsAudioV2 CUDA graph capture failures (GC inside the capture, torch 2.15) (#50899)

Commit
19 hours ago
Fix flaky DeepseekV32 / Whisper / HiggsAudioV2 CUDA graph capture failures (GC inside the capture, torch 2.15) (#50899) * Fix DeepseekV32 CUDA graph capture failures caused by GC inside the capture `DeepseekV32ModelTest::test_static_cache_no_recompile_with_smaller_length` fails on torch 2.15 CI with "operation failed due to a previous error during capture". Cause: Triton kernels left over from earlier compilations in the same process are unreachable objects in reference cycles. When Python's automatic garbage collection runs while inductor records a CUDA graph, it finalizes them: Triton's `CompiledKernel.__del__` calls `cuModuleUnload`, which is not allowed during a capture and invalidates it. torch 2.15 only runs `gc.collect()` before a capture when `torch.compiler.config.force_cudagraph_gc` is set (default: False). Enable `force_cudagraph_gc` for the tests of `DeepseekV32ModelTest` (restored after each test), so leftover kernels are collected right before `capture_begin`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fix Whisper CUDA graph capture failures caused by GC inside the capture `WhisperModelTest::test_onnx_export_generate_dynamic_static_cache` fails on torch 2.15 CI with the same capture invalidation as DeepseekV32: ONNX Runtime CUDA resources left over from the earlier ONNX export tests in the same process are finalized by an automatic GC pass inside the CUDA graph capture (cuMemFree / cuMemFreeHost / cuCtxSynchronize on the capturing thread). Same fix as for DeepseekV32ModelTest: enable `force_cudagraph_gc` for the tests of WhisperModelTest (restored after each test). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Make the force_cudagraph_gc guard safe on older torch `hasattr(torch.compiler.config, ...)` raises AttributeError if the `torch.compiler.config` submodule is not imported (torch 2.6 `torch.compiler` does not import it), which would break `setUp` of both test classes on older torch. Look it up with `getattr(torch.compiler, "config", None)` instead; the patch stays a no-op where the flag does not exist (it was added in torch 2.9). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Revert "Make the force_cudagraph_gc guard safe on older torch" This reverts commit 2e0620f716: torch 2.6 does not need special handling here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fix HiggsAudioV2 CUDA graph capture failures caused by GC inside the capture `HiggsAudioV2ModelTest::test_static_cache_no_recompile_with_smaller_length` fails intermittently with the same capture invalidation as DeepseekV32 (Triton `CompiledKernel.__del__` -> `cuModuleUnload` run by a GC pass inside the capture). Same fix: enable `force_cudagraph_gc` for the tests of HiggsAudioV2ModelTest (restored after each test). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
Author
Parents
Loading