Supported Devices

The OpenVINO™ runtime enables you to use the following devices to run your deep learning models: CPU, GPU, NPU.

For their usage guides, see Devices and Modes.

For a detailed list of devices, see System Requirements.

Beside running inference with a specific device, OpenVINO offers the option of running automated inference with the following inference modes:

Automatic Device Selection:

automatically selects the best device available for the given task. It offers many additional options and optimizations, including inference on multiple devices at the same time.

Heterogeneous Inference:

enables splitting inference among several devices automatically, for example, if one device doesn’t support certain operations.

Automatic Batching:

automatically groups inference requests to improve device utilization.

(LEGACY) Multi-device Inference:

executes inference on multiple devices. Currently, this mode is considered a legacy solution. Using Automatic Device Selection instead is advised.

Feature Support and API Coverage

Supported Feature	CPU	GPU	NPU
Automatic Device Selection	Yes	Yes	Partial
Heterogeneous execution	Yes	Yes	No
Automatic batching	No	Yes	No
Multi-stream execution	Yes	Yes	No
Model caching	Yes	Partial	Yes
Dynamic shapes	Yes	Partial	No
Import/Export	Yes	Yes	Yes
Preprocessing acceleration	Yes	Yes	No
Stateful models	Yes	Yes	Yes
Extensibility	Yes	Yes	No
(LEGACY) Multi-device execution	Yes	Yes	Partial

API Coverage:	plugin	infer_request	compiled_model
CPU	98.31 %	100.0 %	90.7 %
CPU_ARM	80.0 %	100.0 %	89.74 %
GPU	91.53 %	100.0 %	100.0 %
dGPU	89.83 %	100.0 %	100.0 %
NPU	18.64 %	0.0 %	9.3 %
AUTO	93.88 %	100.0 %	100.0 %
BATCH	86.05 %	100.0 %	86.05 %
HETERO	61.22 %	99.24 %	86.05 %
	Percentage of API supported by the device, as of OpenVINO 2023.3, 08 Jan, 2024.

For setting up a relevant configuration, refer to the Integrate with Customer Application topic (step 3 “Configure input and output”).

Note

With the OpenVINO 2024.0 release, support for GNA has been discontinued. To keep using it in your solutions, revert to the 2023.3 (LTS) version.

With the OpenVINO™ 2023.0 release, support has been cancelled for: - Intel® Neural Compute Stick 2 powered by the Intel® Movidius™ Myriad™ X - Intel® Vision Accelerator Design with Intel® Movidius™

To keep using the MYRIAD and HDDL plugins with your hardware, revert to the OpenVINO 2022.3 (LTS) version.