Working with devices¶

The OpenVINO Runtime provides capabilities to infer deep learning models on the following device types with corresponding plugins:

Plugin	Device types
CPU	Intel® Xeon®, Intel® Core™ and Intel® Atom® processors with Intel® Streaming SIMD Extensions (Intel® SSE4.2), Intel® Advanced Vector Extensions 2 (Intel® AVX2), Intel® Advanced Vector Extensions 512 (Intel® AVX-512), Intel® Vector Neural Network Instructions (Intel® AVX512-VNNI) and bfloat16 extension for AVX-512 (Intel® AVX-512_BF16 Extension)
GPU	Intel® Graphics, including Intel® HD Graphics, Intel® UHD Graphics, Intel® Iris® Graphics, Intel® Xe Graphics, Intel® Xe MAX Graphics
VPUs	Intel® Neural Compute Stick 2 powered by the Intel® Movidius™ Myriad™ X, Intel® Vision Accelerator Design with Intel® Movidius™ VPUs
GNA	Intel® Speech Enabling Developer Kit ; Amazon Alexa* Premium Far-Field Developer Kit ; Intel® Pentium® Silver Processors N5xxx, J5xxx and Intel® Celeron® Processors N4xxx, J4xxx (formerly codenamed Gemini Lake) : Intel® Pentium® Silver J5005 Processor , Intel® Pentium® Silver N5000 Processor , Intel® Celeron® J4005 Processor , Intel® Celeron® J4105 Processor , Intel® Celeron® J4125 Processor , Intel® Celeron® Processor N4100 , Intel® Celeron® Processor N4000 ; Intel® Pentium® Processors N6xxx, J6xxx, Intel® Celeron® Processors N6xxx, J6xxx and Intel Atom® x6xxxxx (formerly codenamed Elkhart Lake) ; Intel® Core™ Processors (formerly codenamed Cannon Lake) ; 10th Generation Intel® Core™ Processors (formerly codenamed Ice Lake) : Intel® Core™ i7-1065G7 Processor , Intel® Core™ i7-1060G7 Processor , Intel® Core™ i5-1035G4 Processor , Intel® Core™ i5-1035G7 Processor , Intel® Core™ i5-1035G1 Processor , Intel® Core™ i5-1030G7 Processor , Intel® Core™ i5-1030G4 Processor , Intel® Core™ i3-1005G1 Processor , Intel® Core™ i3-1000G1 Processor , Intel® Core™ i3-1000G4 Processor ; 11th Generation Intel® Core™ Processors (formerly codenamed Tiger Lake) ; 12th Generation Intel® Core™ Processors (formerly codenamed Alder Lake)
Arm® CPU	Raspberry Pi™ 4 Model B, Apple® Mac mini with M1 chip, NVIDIA® Jetson Nano™, Android™ devices

OpenVINO runtime also has several execution capabilities which work on top of other devices:

Capability	Description
Multi-Device execution	Multi-Device enables simultaneous inference of the same model on several devices in parallel
Auto-Device selection	Auto-Device selection enables selecting Intel device for inference automatically
Heterogeneous execution	Heterogeneous execution enables automatic inference splitting between several devices (for example if a device doesn’t support certain operation )
Automatic Batching	Auto-Batching plugin enables the batching (on top of the specified device) that is completely transparent to the application

Devices similar to the ones we have used for benchmarking can be accessed using Intel® DevCloud for the Edge, a remote development environment with access to Intel® hardware and the latest versions of the Intel® Distribution of the OpenVINO™ Toolkit. Learn more or Register here.

Features Support Matrix¶

The table below demonstrates support of key features by OpenVINO device plugins.

Capability	CPU	GPU	GNA	Arm® CPU
Heterogeneous execution	Yes	Yes	No	Yes
Multi-device execution	Yes	Yes	Partial	Yes
Automatic batching	No	Yes	No	No
Multi-stream execution	Yes	Yes	No	Yes
Models caching	Yes	Partial	Yes	No
Dynamic shapes	Yes	Partial	No	No
Import/Export	Yes	No	Yes	No
Preprocessing acceleration	Yes	Yes	No	Partial
Stateful models	Yes	No	Yes	No
Extensibility	Yes	Yes	No	No

For more details on plugin specific feature limitation, see corresponding plugin pages.

Prev Next