onnxruntime
6572dd0b - [NvTensorRTRTX EP] Skip GPU JIT during compile-only sessions (#28503)

Commit
60 days ago
[NvTensorRTRTX EP] Skip GPU JIT during compile-only sessions (#28503) ### Description Add an internal session config entry, `"session.compile_only"`, set by `CompileModel()` before session initialization. The NvTensorRTRTX EP reads it in `NvExecutionProviderInfo::FromProviderOptions()` and, when set, skips `deserializeCudaEngine()` / `createExecutionContext()` in `CreateNodeComputeInfoFromGraph()`. The EP context node is still saved — that path uses the serialized engine buffer directly and does not depend on the deserialized engine. A stub compute function is registered to satisfy the framework; it returns `NOT_IMPLEMENTED` if called, which cannot happen in practice because compile-only sessions are destroyed without inference. ### Motivation and Context `OrtCompileAPI::CompileModel()` creates an `InferenceSession` solely to drive `EP::Compile()` and write out the EPContext model, then destroys it without running inference. During that session, the NvTensorRTRTX EP was performing a full `deserializeCudaEngine()` and `createExecutionContext()` — uploading engine weights to the GPU and JIT-ing the engine, only to free everything when the session was destroyed. When the user then loads the EPContext model in a real session, the same JIT and upload happen again. Net effect on the typical "compile, then load and run" flow: ``` ONNX model → CompileModel() [JIT + GPU upload #1 — discarded] → EP context model saved to disk → Session from EP context model [JIT + GPU upload #2 — necessary] → Inference ``` JIT and GPU upload run twice.
Author
Parents
Loading