onnxruntime
2aa23ccd - [OpenVINO EP] Support dot-separated KV cache tensor names in stateful (CausalLM) transform (#29895)

Commit
6 days ago
[OpenVINO EP] Support dot-separated KV cache tensor names in stateful (CausalLM) transform (#29895) ### Description The OpenVINO EP's stateful/CausalLM transform (enable_causallm=True) fails to compile ORT-GenAI–built decoder models with: `ov_stateful_patch_utils.cc PatchStatefulDecoder: No key_value_input_names or key_value_output_names found` ### Root cause `ExtractKVPatternsFromOutputs()` recognizes KV-cache output tensors only when they use the underscore naming convention present_<...> (as produced by HF/optimum-intel exports). Models produced by the ONNX Runtime GenAI model builder name these tensors with dots instead — `present.<layer>.key` / `present.<layer>.value` (and correspondingly `past_key_values.<layer>.key`). Because `name.find("present_") == 0` never matches the dot form, `key_value_output_names` comes back empty and `PatchStatefulDecoder()` throws before applying MakeStateful. The input side is unaffected — its fallback matcher keys on the "key_values" substring, which the dot-named inputs contain — so this is purely an output-name matching gap. This blocks NPU inference in particular: the EP forces static shapes on NPU unless enable_causallm is enabled (openvino_provider_factory.cc), so a stateful decoder is the only viable NPU path for these models; without this fix it can't compile. ### Changes ov_stateful_patch_utils.cc — `ExtractKVPatternsFromOutputs()` now also accepts the dot-separated prefix present.. Matched outputs are collected while unique_patterns is left empty, which routes `ExtractInputKVTensors()` to its existing substring fallback so the `past_key_values.*` inputs are paired in the same layer order. Underscore-named models are unchanged. ov_interface.cc — add a catch (const std::exception&) clause to OvExceptionBoundary. Previously, non-ov::Exception errors (e.g. ORT_THROW, which raises OnnxRuntimeException) were caught by catch (...) and their message discarded, yielding an opaque "Exception while Loading Network" with no cause. The new clause preserves e.what(). This is what made the root cause above diagnosable. ### Testing Phi-4-mini-instruct (native onnx), enable_causallm=True: Before: fails at compile on CPU and NPU (No key_value_input_names...). After: compiles and generates correctly on CPU, GPU, and NPU (OpenVINO 2026.3). Underscore-named models continue to work (matching logic for that path is untouched).
Author
Parents
Loading