onnxruntime
c64c9fd7 - Fix 1DS telemetry deadlock during Windows process termination (#33291)

Commit
15 days ago
Fix 1DS telemetry deadlock during Windows process termination (#33291) ### Description Native inference can finish successfully, write its profiling output, and then hang indefinitely while Windows terminates the process. Skip 1DS uploader teardown in `OneDsTelemetry::~OneDsTelemetry` when Windows DLL shutdown is in progress, before acquiring telemetry locks. The Windows helper resolves `RtlDllShutdownInProgress` from the already loaded `ntdll.dll`. This process-wide check also works for telemetry compiled into an EP plugin DLL. Normal object destruction and explicit `FreeLibrary` retain the existing cleanup path; non-Windows behavior is unchanged. Pending telemetry is not flushed during process termination. ### Motivation and Context Two captured native benchmark hangs had only the terminating thread remaining, with this shutdown stack: ```text ExitProcess -> LdrShutdownProcess -> WindowsEnv static destructor -> OneDsTelemetry::~OneDsTelemetry -> LogManagerImpl::FlushAndTeardown -> HttpClientManager::cancelAllRequests / cancelAllRequestsAsync -> HttpClient_WinHttp::CancelAllRequests -> condition-variable wait ``` Windows terminates the other threads before calling DLL process-detach destructors. The uploader can therefore wait forever for WinHTTP callbacks that can no longer run; a telemetry lock may also be held by a terminated thread. Checking shutdown state before taking locks avoids both waits. Resolved ORT/SDK frames were verified against identical machine code in a build with symbols. ### Regression Coverage Three committed Windows CTest cases load a fixture DLL linked to the production `onnxruntime_common` telemetry implementation. A test peer injects a mock log manager with network telemetry disabled; no model, GPU, or symbol-address patching is required. - `onnxruntime_telemetry_process_exit_test`: calls `ExitProcess` with a pending mock callback and verifies that the real DLL-owned telemetry destructor skips uploader cleanup. - `onnxruntime_telemetry_normal_destruction_test`: destroys the object before DLL unload and verifies one `Flush`, one `FlushAndTeardown`, and cleared shared state. - `onnxruntime_telemetry_explicit_unload_test`: verifies the same cleanup during `FreeLibrary`. The parent requires both a successful child exit and an inherited event signaled only after teardown assertions pass. Each child has a 10-second timeout, with a 15-second CTest limit. The mock drain itself is bounded to two seconds so a regressed shutdown cannot strand a child inside DLL teardown. Run from the CMake build directory after building `onnxruntime_telemetry_shutdown_test`: ```text ctest -C Release -R "^onnxruntime_telemetry_(process_exit|normal_destruction|explicit_unload)_test$" --output-on-failure ``` ### Validation - [x] Windows x64/MSVC Release: configured a separate build and compiled the committed test target and its production dependencies; all three lifecycle tests passed. - [x] Local mutation checks: removing the shutdown guard failed only the process-exit case; returning unconditionally from the destructor failed both normal destruction and explicit unload. Restoring the fix passed all three cases again. - [x] Windows/MSVC runtime validation: recompiled the changed translation units and relinked isolated built-in WebGPU ORT, plugin ORT host, and WebGPU plugin binaries using compatible cached build objects. - [x] Original local lifecycle harness with network telemetry suppressed and a mock pending uploader injected into the actual ORT DLL: baseline entered SDK draining and timed out; fixed runtime exited normally in **20/20** trials. Explicit DLL unload with an inactive uploader passed **3/3** trials for both baseline and fixed runtimes. - [x] Native GenAI benchmark: **built-in/plugin WebGPU × R32 FP16 LoRA on/off**, 1,024 prompt tokens and 32 output tokens, batch 1, one warmup and two measured runs. All four configurations exited with code 0; generated text matched across providers for each adapter state. An additional built-in/LoRA-on run matched the original runtime's output and exited normally. - [x] Changed-file `lintrunner -a` and `git diff --check` passed. The committed lifecycle tests use a mock uploader; they do not cover real pending-upload unload stress. The benchmark executable, GenAI DLL, model graph, configuration, and adapter were unchanged during runtime validation.
Author
Parents
Loading