Inference Device Support¶
The OpenVINO™ runtime enables you to use a selection of devices to run your deep learning models: CPU, GPU, NPU.
Beside running inference with a specific device, OpenVINO offers the option of running automated inference with the following inference modes:
Automatic Device Selection - automatically selects the best device available for the given task. It offers many additional options and optimizations, including inference on multiple devices at the same time.
Heterogeneous Inference - enables splitting inference among several devices automatically, for example, if one device doesn’t support certain operations.
Multi-device Inference - executes inference on multiple devices. Currently, this mode is considered a legacy solution. Using Automatic Device Selection is advised.
Automatic Batching - automatically groups inference requests to improve device utilization.
Feature Support and API Coverage¶
Supported Feature |
CPU |
GPU |
NPU |
|---|---|---|---|
Yes |
Yes |
No |
|
Yes |
Yes |
Partial |
|
No |
Yes |
No |
|
Yes |
Yes |
No |
|
Yes |
Partial |
Yes |
|
Yes |
Partial |
No |
|
Yes |
No |
Yes |
|
Yes |
Yes |
No |
|
Yes |
No |
Yes |
|
Yes |
Yes |
No |
API Coverage: |
plugin |
infer_request |
compiled_model |
|---|---|---|---|
CPU |
80.0 % |
100.0 % |
89.74 % |
CPU_ARM |
80.0 % |
100.0 % |
89.74 % |
GPU |
84.0 % |
100.0 % |
100.0 % |
dGPU |
82.0 % |
100.0 % |
100.0 % |
NPU |
16.0 % |
0.0 % |
10.26 % |
AUTO |
40.0 % |
100.0 % |
97.44 % |
BATCH |
26.0 % |
100.0 % |
58.97 % |
MULTI |
30.0 % |
100.0 % |
58.97 % |
HETERO |
30.0 % |
99.23 % |
58.97 % |
Percentage of API supported by the device,
as of OpenVINO 2023.3, 08 Jan, 2024.
|
|||
Devices similar to the ones used for benchmarking can be accessed using Intel® DevCloud for the Edge, a remote development environment with access to Intel® hardware and the latest versions of the Intel® Distribution of OpenVINO™ Toolkit. Learn more or Register here.
For setting up a relevant configuration, refer to the Integrate with Customer Application topic (step 3 “Configure input and output”).
Note
With OpenVINO 2024.0 release, support for GNA has been discontinued. To keep using it in your solutions, revert to the 2023.3 (LTS) version.
With OpenVINO™ 2023.0 release, support has been cancelled for: - Intel® Neural Compute Stick 2 powered by the Intel® Movidius™ Myriad™ X - Intel® Vision Accelerator Design with Intel® Movidius™
To keep using the MYRIAD and HDDL plugins with your hardware, revert to the OpenVINO 2022.3 (LTS) version.