Supported Devices#

The OpenVINO™ runtime enables you to use the following devices to run your deep learning models: CPU, GPU, NPU.

For their usage guides, see Devices and Modes.
For a detailed list of devices, see System Requirements.

Beside running inference with a specific device, OpenVINO offers the option of running automated inference with the following inference modes:

Automatic Device Selection:
automatically selects the best device available for the given task. It offers many additional options and optimizations, including inference on multiple devices at the same time.
Heterogeneous Inference:
enables splitting inference among several devices automatically, for example, if one device doesn’t support certain operations.
Automatic Batching:
automatically groups inference requests to improve device utilization.
(LEGACY) Multi-device Inference:
executes inference on multiple devices. Currently, this mode is considered a legacy solution. Using Automatic Device Selection instead is advised.

Feature Support and API Coverage#

Supported Feature

CPU

GPU

NPU

Automatic Device Selection

Yes

Yes

Partial

Heterogeneous execution

Yes

Yes

No

Automatic batching

No

Yes

No

Multi-stream execution

Yes

Yes

No

Model caching

Yes

Partial

Yes

Dynamic shapes

Yes

Partial

No

Import/Export

Yes

Yes

Yes

Preprocessing acceleration

Yes

Yes

No

Stateful models

Yes

Yes

Yes

Extensibility

Yes

Yes

No

(LEGACY) Multi-device execution

Yes

Yes

Partial

API Coverage:

plugin

infer_request

compiled_model

CPU

80.0 %

100.0 %

89.74 %

CPU_ARM

80.0 %

100.0 %

89.74 %

GPU

84.0 %

100.0 %

100.0 %

dGPU

82.0 %

100.0 %

100.0 %

NPU

16.0 %

0.0 %

10.26 %

AUTO

40.0 %

100.0 %

97.44 %

BATCH

26.0 %

100.0 %

58.97 %

HETERO

30.0 %

99.23 %

58.97 %
Percentage of API supported by the device,
as of OpenVINO 2023.3, 08 Jan, 2024.

For setting up a relevant configuration, refer to the Integrate with Customer Application topic (step 3 “Configure input and output”).

Note

With the OpenVINO 2024.0 release, support for GNA has been discontinued. To keep using it in your solutions, revert to the 2023.3 (LTS) version.

With the OpenVINO™ 2023.0 release, support has been cancelled for: - Intel® Neural Compute Stick 2 powered by the Intel® Movidius™ Myriad™ X - Intel® Vision Accelerator Design with Intel® Movidius™

To keep using the MYRIAD and HDDL plugins with your hardware, revert to the OpenVINO 2022.3 (LTS) version.