Fix flaky DeepseekV32 / Whisper / HiggsAudioV2 CUDA graph capture failures (GC inside the capture, torch 2.15) (#50899)
* Fix DeepseekV32 CUDA graph capture failures caused by GC inside the capture
`DeepseekV32ModelTest::test_static_cache_no_recompile_with_smaller_length` fails on
torch 2.15 CI with "operation failed due to a previous error during capture".
Cause: Triton kernels left over from earlier compilations in the same process are
unreachable objects in reference cycles. When Python's automatic garbage collection
runs while inductor records a CUDA graph, it finalizes them: Triton's
`CompiledKernel.__del__` calls `cuModuleUnload`, which is not allowed during a
capture and invalidates it. torch 2.15 only runs `gc.collect()` before a capture
when `torch.compiler.config.force_cudagraph_gc` is set (default: False).
Enable `force_cudagraph_gc` for the tests of `DeepseekV32ModelTest` (restored after
each test), so leftover kernels are collected right before `capture_begin`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Fix Whisper CUDA graph capture failures caused by GC inside the capture
`WhisperModelTest::test_onnx_export_generate_dynamic_static_cache` fails on torch
2.15 CI with the same capture invalidation as DeepseekV32: ONNX Runtime CUDA
resources left over from the earlier ONNX export tests in the same process are
finalized by an automatic GC pass inside the CUDA graph capture (cuMemFree /
cuMemFreeHost / cuCtxSynchronize on the capturing thread).
Same fix as for DeepseekV32ModelTest: enable `force_cudagraph_gc` for the tests of
WhisperModelTest (restored after each test).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Make the force_cudagraph_gc guard safe on older torch
`hasattr(torch.compiler.config, ...)` raises AttributeError if the
`torch.compiler.config` submodule is not imported (torch 2.6 `torch.compiler`
does not import it), which would break `setUp` of both test classes on older torch.
Look it up with `getattr(torch.compiler, "config", None)` instead; the patch stays a
no-op where the flag does not exist (it was added in torch 2.9).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Revert "Make the force_cudagraph_gc guard safe on older torch"
This reverts commit 2e0620f716: torch 2.6 does not need special handling here.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Fix HiggsAudioV2 CUDA graph capture failures caused by GC inside the capture
`HiggsAudioV2ModelTest::test_static_cache_no_recompile_with_smaller_length` fails
intermittently with the same capture invalidation as DeepseekV32 (Triton
`CompiledKernel.__del__` -> `cuModuleUnload` run by a GC pass inside the capture).
Same fix: enable `force_cudagraph_gc` for the tests of HiggsAudioV2ModelTest
(restored after each test).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>