Inference Device Support#

The OpenVINO™ runtime enables you to use a selection of devices to run your deep learning models: CPU, GPU, NPU.

For their usage guides, see Devices and Modes.
For a detailed list of devices, see System Requirements.

Beside running inference with a specific device, OpenVINO offers the option of running automated inference with the following inference modes:

  • Automatic Device Selection - automatically selects the best device available for the given task. It offers many additional options and optimizations, including inference on multiple devices at the same time.

  • Heterogeneous Inference - enables splitting inference among several devices automatically, for example, if one device doesn’t support certain operations.

  • Multi-device Inference - executes inference on multiple devices. Currently, this mode is considered a legacy solution. Using Automatic Device Selection is advised.

  • Automatic Batching - automatically groups inference requests to improve device utilization.

Feature Support and API Coverage#

Supported Feature

CPU

GPU

NPU

Heterogeneous execution

Yes

Yes

No

Multi-device execution

Yes

Yes

Partial

Automatic batching

No

Yes

No

Multi-stream execution

Yes

Yes

No

Models caching

Yes

Partial

Yes

Dynamic shapes

Yes

Partial

No

Import/Export

Yes

Yes

Yes

Preprocessing acceleration

Yes

Yes

No

Stateful models

Yes

Yes

Yes

Extensibility

Yes

Yes

No

API Coverage:

plugin

infer_request

compiled_model

CPU

80.0 %

100.0 %

89.74 %

CPU_ARM

80.0 %

100.0 %

89.74 %

GPU

84.0 %

100.0 %

100.0 %

dGPU

82.0 %

100.0 %

100.0 %

NPU

16.0 %

0.0 %

10.26 %

AUTO

40.0 %

100.0 %

97.44 %

BATCH

26.0 %

100.0 %

58.97 %

MULTI

30.0 %

100.0 %

58.97 %

HETERO

30.0 %

99.23 %

58.97 %

Percentage of API supported by the device,
as of OpenVINO 2023.3, 08 Jan, 2024.

Devices similar to the ones used for benchmarking can be accessed using Intel® DevCloud for the Edge, a remote development environment with access to Intel® hardware and the latest versions of the Intel® Distribution of OpenVINO™ Toolkit. Learn more or Register here.

For setting up a relevant configuration, refer to the Integrate with Customer Application topic (step 3 “Configure input and output”).

Note

With OpenVINO 2024.0 release, support for GNA has been discontinued. To keep using it in your solutions, revert to the 2023.3 (LTS) version.

With OpenVINO™ 2023.0 release, support has been cancelled for: - Intel® Neural Compute Stick 2 powered by the Intel® Movidius™ Myriad™ X - Intel® Vision Accelerator Design with Intel® Movidius™

To keep using the MYRIAD and HDDL plugins with your hardware, revert to the OpenVINO 2022.3 (LTS) version.