onnxruntime
e0786fe7 - ORT perf test support for plugin EP (#25374)

Commit
189 days ago
ORT perf test support for plugin EP (#25374) ### Description Add support for onnxruntime_perf_test to register plugin EP dll and run plugin EP. As support for plugin execution providers (EPs) requires additional options and most single-character options have already been used, multi-character options are now necessary to ensure clarity and readability. Therefore, support for `Abseil flags` is added, which enables multi-character options and provides cross-platform compatibility. **New options:** - `--plugin_ep_libs [registration names and libraries]` Specifies a list of plugin execution provider (EP) registration names and their corresponding shared libraries to register. [Usage]: `--plugin_ep_libs "plugin_ep_name_1|plugin_ep_1.dll plugin_ep_name_2|plugin_ep_2.dll ... "` - `--plugin_eps [Plugin EPs]` Specifies a semicolon-separated list of plugin execution providers (EPs) to use. [Usage]: `--plugin_eps "plugin_ep_1;plugin_ep_2;... "` - `--plugin_ep_options [EP options]` Specifies provider options for each EP listed in --plugin_eps. Options (key-value pairs) for each EP are separated by space and EPs are separated by semicolons. [Usage]: `--plugin_ep_options "ep_1_option_1_key|ep_1_option_1_value ...;ep_2_option_1_key|ep_2_option_1_value ...;..."` or `--plugin_ep_options ";ep_2_option_1_key|ep_2_option_1_value ...;..."` or `--plugin_ep_options "ep_1_option_1_key|ep_1_option_1_value ...;;ep_3_option_1_key|ep_3_option_1_value ...;..."` - `--list_ep_devices` Prints all available device indices and their properties (including metadata). This option makes the program exit early without performing inference. - ` --select_ep_devices [list of device indices]` A semicolon-separated list of device indices to add to the session and run with. **Usage:** 1. Use `--plugin_ep_libs` and `--list_ep_devices` to list all the devices. ````sh --list_ep_devices --plugin_ep_libs "TensorRTEp|C:\TensorRTEp.dll example_ep|C:\example_plugin_ep.dll" ```` It will print the devices info ```` ===== EP device id 0 ====== name: CPUExecutionProvider vendor: Microsoft metadata: version: 1.23.0 ===== EP device id 1 ====== name: example_ep vendor: Contoso metadata: supported_devices: CrackGriffin 7+ version: 0.1.0 ===== EP device id 2 ====== name: TensorRTEp vendor: Nvidia metadata: gpu_type: data center version: 0.1.0 ```` 2. Use `--select_ep_devices` to select the device by index. And add `--plugin_eps` to specify the EP name. The EP name should match the name when ep library passes in to create the ep factory. ````sh --plugin_ep_libs "TensorRTEp|C:\TensorRTEp.dll" --select_ep_devices 2 --plugin_eps TensorRTEp -r 1 C:\mul_op\mul_1.onnx ```` 3. Or simply use `-e` to specify the EP name. ORT perf test will add all the devices created by the plugin EP. The EP name should match the name when ep library passes in to create the ep factory. ````sh --plugin_ep_libs "TensorRTEp|C:\TensorRTEp.dll" --plugin_eps TensorRTEp -r 1 C:\mul_op\mul_1.onnx ````
Author
Parents
Loading