Supported Devices

The Inference Engine can infer models in different formats with various input and output formats. This section provides supported and optimal configurations per device.

NOTE: With OpenVINO™ 2020.4 release, Intel® Movidius™ Neural Compute Stick is no longer supported.

The Inference Engine provides unique capabilities to infer deep learning models on the following device types with corresponding plugins:

Plugin Device types
GPU plugin Intel® Processor Graphics, including Intel® HD Graphics and Intel® Iris® Graphics
CPU plugin Intel® Xeon® with Intel® Advanced Vector Extensions 2 (Intel® AVX2), Intel® Advanced Vector Extensions 512 (Intel® AVX-512), and AVX512_BF16, Intel® Core™ Processors with Intel® AVX2, Intel® Atom® Processors with Intel® Streaming SIMD Extensions (Intel® SSE)
FPGA plugin (available in the Intel® Distribution of OpenVINO™ toolkit) Intel® Vision Accelerator Design with an Intel® Arria 10 FPGA (Speed Grade 2), Intel® Programmable Acceleration Card with Intel® Arria® 10 GX FPGA
VPU plugins (available in the Intel® Distribution of OpenVINO™ toolkit) Intel® Neural Compute Stick 2 powered by the Intel® Movidius™ Myriad™ X, Intel® Vision Accelerator Design with Intel® Movidius™ VPUs
GNA plugin (available in the Intel® Distribution of OpenVINO™ toolkit) Intel® Speech Enabling Developer Kit, Amazon Alexa* Premium Far-Field Developer Kit, Intel® Pentium® Silver J5005 Processor, Intel® Pentium® Silver N5000 Processor, Intel® Celeron® J4005 Processor, Intel® Celeron® J4105 Processor, Intel® Celeron® Processor N4100, Intel® Celeron® Processor N4000, Intel® Core™ i3-8121U Processor, Intel® Core™ i7-1065G7 Processor, Intel® Core™ i7-1060G7 Processor, Intel® Core™ i5-1035G4 Processor, Intel® Core™ i5-1035G7 Processor, Intel® Core™ i5-1035G1 Processor, Intel® Core™ i5-1030G7 Processor, Intel® Core™ i5-1030G4 Processor, Intel® Core™ i3-1005G1 Processor, Intel® Core™ i3-1000G1 Processor, Intel® Core™ i3-1000G4 Processor
Multi-Device plugin Multi-Device plugin enables simultaneous inference of the same network on several Intel® devices in parallel
Heterogeneous plugin Heterogeneous plugin enables automatic inference splitting between several Intel® devices (for example if a device doesn't support certain layers).

Supported Configurations

The Inference Engine can inference models in different formats with various input and output formats. This chapter provides supported and optimal configurations for each plugin.

Terminology

Acronym/Term Description
DL Deep Learning
FP32 format Single-precision floating-point format
BF16 format Brain floating-point format
FP16 format Half-precision floating-point format
I16 format 2-byte signed integer format
I8 format 1-byte signed integer format
U16 format 2-byte unsigned integer format
U8 format 1-byte unsigned integer format

NHWC, NCHW - Image data layout. Refers to the representation of batches of images. NCDHW - Images sequence data layout.

CHW, NC, C - Tensor memory layout. For example, the CHW value at index (c,h,w) is physically located at index (c*H+h)*W+w, for others by analogy

Supported Model Formats

Plugin FP32 FP16 I8
CPU plugin Supported and preferred Supported Supported
GPU plugin Supported Supported and preferred Supported*
FPGA plugin Supported Supported Not supported
VPU plugins Not supported Supported Not supported
GNA plugin Supported Supported Not supported


* - currently, only limited set of topologies might benefit from enabling I8 model on GPU
For Multi-Device and Heterogeneous execution the supported models formats depends on the actual underlying devices. Generally, FP16 is preferable as it is most ubiquitous and performant.

Supported Input Precision

Plugin FP32 FP16 U8 U16 I8 I16
CPU plugin Supported Not supported Supported Supported Not supported Supported
GPU plugin Supported Supported* Supported* Supported* Not supported Supported*
FPGA plugin Supported Supported* Supported Supported Not supported Supported
VPU plugins Supported Supported Supported Not supported Not supported Not supported
GNA plugin Supported Not supported Supported Not supported Supported Supported


* - Supported via SetBlob only, GetBlob returns FP32
For Multi-Device and Heterogeneous execution the supported input precision depends on the actual underlying devices. Generally, U8 is preferable as it is most ubiquitous.

Supported Output Precision

Plugin FP32 FP16
CPU plugin Supported Not supported
GPU plugin Supported Supported
FPGA plugin Supported Supported
VPU plugins Supported Supported
GNA plugin Supported Not supported

For Multi-Device and Heterogeneous execution the supported output precision depends on the actual underlying devices. Generally, FP32 is preferable as it is most ubiquitous.

Supported Input Layout

Plugin NCDHW NCHW NHWC NC
CPU plugin Supported Supported Supported Supported
GPU plugin Supported Supported Supported Supported
FPGA plugin Not supported Supported Supported Not supported
VPU plugins Not supported Supported Supported Supported
GNA plugin Not supported Not supported Not supported Supported

Supported Output Layout

Number of dimensions5 4 3 2 1
Layout NCDHWNCHW CHW NC C

For setting relevant configuration, refer to the Integrate with Customer Application New Request API topic (step 3 "Configure input and output").

Supported Layers

The following layers are supported by the plugins and by Shape Inference feature:

Layers GPU CPU VPU GNA FPGA ShapeInfer
Abs Supported Supported** Supported Not Supported Not Supported Supported
Acos Supported Supported** Not Supported Not Supported Not Supported Supported
Acosh Supported Supported** Not Supported Not Supported Not Supported Supported
Activation-Clamp Supported Supported***Supported Supported Supported Supported
Activation-ELU Supported Supported***Supported Not Supported Supported Supported
Activation-Exp Supported Supported***Not Supported Supported Not Supported Supported
Activation-Leaky ReLU Supported Supported***Supported Supported Supported Supported
Activation-Not Supported Supported***Not Supported Not Supported Not Supported Supported
Activation-PReLU Supported Supported***Supported Not Supported Supported Supported
Activation-ReLU Supported Supported***Supported Supported Supported Supported
Activation-ReLU6 Supported Supported***Supported Not Supported Not Supported Supported
Activation-Sigmoid/Logistic Supported Supported***Supported Supported Not Supported Supported
Activation-TanH Supported Supported***Supported Supported Not Supported Supported
ArgMax Supported Supported** Supported Not Supported Not Supported Supported
Asin Supported Supported** Not Supported Not Supported Not Supported Supported
Asinh Supported Supported** Not Supported Not Supported Not Supported Supported
Atan Supported Supported** Not Supported Not Supported Not Supported Supported
Atanh Supported Supported** Not Supported Not Supported Not Supported Supported
BatchNormalization Supported Supported Supported Not Supported Supported* Supported
BinaryConvolution Supported Supported Not Supported Not Supported Not Supported Supported
Broadcast Supported Supported** Supported Not Supported Not Supported Supported
Ceil Supported Supported** Not Supported Not Supported Not Supported Supported
Concat Supported Supported***Supported Supported Supported Supported
Const Supported Supported Supported Supported Not Supported Not Supported
Convolution-Dilated Supported Supported Supported Not Supported Supported Supported
Convolution-Dilated 3D Supported Supported Not Supported Not Supported Not Supported Not Supported
Convolution-Grouped Supported Supported Supported Not Supported Supported Supported
Convolution-Grouped 3D Supported Supported Not Supported Not Supported Not Supported Not Supported
Convolution-Ordinary Supported Supported Supported Supported* Supported Supported
Convolution-Ordinary 3D Supported Supported Not Supported Not Supported Not Supported Not Supported
Cos Supported Supported** Not Supported Not Supported Not Supported Supported
Cosh Supported Supported** Not Supported Not Supported Not Supported Supported
Crop Supported Supported Supported Supported Not Supported Supported
CTCGreedyDecoder Supported** Supported** Supported* Not Supported Not Supported Supported
Deconvolution Supported Supported Supported Not Supported Supported* Supported
Deconvolution 3D Supported Supported Not Supported Not Supported Not Supported Not Supported
DeformableConvolution Supported Supported Not Supported Not Supported Not Supported Supported
DepthToSpace Supported Supported** Not Supported Not Supported Not Supported Supported
DetectionOutput Supported Supported** Supported* Not Supported Not Supported Supported
Eltwise-And Supported Supported***Not Supported Not Supported Not Supported Supported
Eltwise-Add Supported Supported***Not Supported Not Supported Supported Supported
Eltwise-Div Supported Supported***Supported Not Supported Not Supported Supported
Eltwise-Equal Supported Supported***Supported Not Supported Not Supported Supported
Eltwise-FloorMod Supported Supported***Supported Not Supported Not Supported Supported
Eltwise-Greater Supported Supported***Supported Not Supported Not Supported Supported
Eltwise-GreaterEqual Supported Supported***Supported Not Supported Not Supported Supported
Eltwise-Less Supported Supported***Supported Not Supported Not Supported Supported
Eltwise-LessEqual Supported Supported***Supported Not Supported Not Supported Supported
Eltwise-LogicalAnd Supported Supported***Supported Not Supported Not Supported Supported
Eltwise-LogicalOr Supported Supported***Supported Not Supported Not Supported Supported
Eltwise-LogicalXor Supported Supported***Supported Not Supported Not Supported Supported
Eltwise-Max Supported Supported***Supported Not Supported Not Supported Supported
Eltwise-Min Supported Supported***Supported Not Supported Not Supported Supported
Eltwise-Mul Supported Supported***Supported Supported Not Supported Supported
Eltwise-NotEqual Supported Supported***Supported Not Supported Not Supported Supported
Eltwise-Pow Supported Supported***Supported Not Supported Not Supported Supported
Eltwise-Prod Supported Supported***Supported Supported Not Supported Supported
Eltwise-SquaredDiff Supported Supported***Supported Not Supported Not Supported Supported
Eltwise-Sub Supported Supported***Supported Supported Supported Supported
Eltwise-Sum Supported Supported***Supported Supported Supported Supported
Erf Supported Supported** Not Supported Not Supported Not Supported Supported
Exp Supported Supported Not Supported Supported Not Supported Supported
FakeQuantize Not Supported Supported Not Supported Not Supported Not Supported Supported
Fill Not Supported Supported** Not Supported Not Supported Not Supported Supported
Flatten Supported Supported Supported Not Supported Not Supported Supported
Floor Supported Supported** Not Supported Not Supported Not Supported Supported
FullyConnected (Inner Product) Supported Supported***Supported Supported Supported Supported
Gather Supported Supported** Supported Not Supported Not Supported Supported
GatherTree Not Supported Supported** Not Supported Not Supported Not Supported Supported
Gemm Supported Supported Supported Not Supported Not Supported Supported
GRN Supported** Supported** Supported Not Supported Not Supported Supported
HardSigmoid Supported Supported** Not Supported Not Supported Not Supported Supported
Interp Supported** Supported** Supported Not Supported Not Supported Supported*
Log Supported Supported** Supported Supported Not Supported Supported
LRN (Norm) Supported Supported Supported Not Supported Supported Supported
LSTMCell Supported Supported Supported Supported Not Supported Not Supported
GRUCell Supported Supported Not Supported Not Supported Not Supported Not Supported
RNNCell Supported Supported Not Supported Not Supported Not Supported Not Supported
LSTMSequence Supported Supported Supported Not Supported Not Supported Not Supported
GRUSequence Supported Supported Not Supported Not Supported Not Supported Not Supported
RNNSequence Supported Supported Not Supported Not Supported Not Supported Not Supported
LogSoftmax Supported Supported** Not Supported Not Supported Not Supported Not Supported
Memory Not Supported Supported Not Supported Supported Not Supported Supported
MVN Supported Supported** Supported* Not Supported Not Supported Supported
Neg Supported Supported** Not Supported Not Supported Not Supported Supported
NonMaxSuppression Not Supported Supported** Not Supported Not Supported Not Supported Supported
Normalize Supported Supported** Supported* Not Supported Not Supported Supported
OneHot Supported Supported** Not Supported Not Supported Not Supported Supported
Pad Supported Supported** Supported* Not Supported Not Supported Supported
Permute Supported Supported Supported Supported* Not Supported Supported
Pooling(AVG,MAX) Supported Supported Supported Supported Supported Supported
Pooling(AVG,MAX) 3D Supported Supported Not Supported Not Supported Not Supported Not Supported
Power Supported Supported** Supported Supported* Supported* Supported
PowerFile Not Supported Supported** Not Supported Not Supported Not Supported Not Supported
PriorBox Supported Supported** Supported Not Supported Not Supported Supported
PriorBoxClustered Supported** Supported** Supported Not Supported Not Supported Supported
Proposal Supported Supported** Supported Not Supported Not Supported Supported
PSROIPooling Supported Supported** Supported Not Supported Not Supported Supported
Range Not Supported Supported** Not Supported Not Supported Not Supported Supported
Reciprocal Supported Supported** Not Supported Not Supported Not Supported Supported
ReduceAnd Supported Supported** Not Supported Not Supported Not Supported Supported
ReduceL1 Supported Supported** Not Supported Not Supported Not Supported Supported
ReduceL2 Supported Supported** Not Supported Not Supported Not Supported Supported
ReduceLogSum Supported Supported** Not Supported Not Supported Not Supported Supported
ReduceLogSumExp Supported Supported** Not Supported Not Supported Not Supported Supported
ReduceMax Supported Supported** Not Supported Not Supported Not Supported Supported
ReduceMean Supported Supported** Not Supported Not Supported Not Supported Supported
ReduceMin Supported Supported** Not Supported Not Supported Not Supported Supported
ReduceOr Supported Supported** Not Supported Not Supported Not Supported Supported
ReduceProd Supported Supported** Not Supported Not Supported Not Supported Supported
ReduceSum Supported Supported** Not Supported Not Supported Not Supported Supported
ReduceSumSquare Supported Supported** Not Supported Not Supported Not Supported Supported
RegionYolo Supported Supported** Supported Not Supported Not Supported Supported
ReorgYolo Supported Supported** Supported Not Supported Not Supported Supported
Resample Supported Supported** Supported Not Supported Supported* Supported
Reshape Supported Supported***Supported Supported Not Supported Supported*
ReverseSequence Supported Supported** Supported Not Supported Not Supported Supported
RNN Not Supported Supported Supported Not Supported Not Supported Not Supported
ROIPooling Supported* Supported Supported Not Supported Not Supported Supported
ScaleShift Supported Supported***Supported* Supported Supported Supported
ScatterUpdate Not Supported Supported** Not Supported Not Supported Not Supported Supported
Select Supported Supported Supported Not Supported Not Supported Supported
Selu Supported Supported** Not Supported Not Supported Not Supported Supported
ShuffleChannels Supported Supported** Not Supported Not Supported Not Supported Supported
Sign Supported Supported** Supported Not Supported Not Supported Supported
Sin Supported Supported** Not Supported Not Supported Not Supported Supported
Sinh Supported Supported** Not Supported Not Supported Not Supported Supported
SimplerNMS Supported Supported** Not Supported Not Supported Not Supported Supported
Slice Supported Supported***Supported Supported Supported* Supported
SoftMax Supported Supported***Supported Not Supported Not Supported Supported
Softplus Supported Supported** Not Supported Not Supported Not Supported Supported
Softsign Supported Supported** Not Supported Supported Not Supported Supported
SpaceToDepth Not Supported Supported** Not Supported Not Supported Not Supported Supported
SpatialTransformer Not Supported Supported** Not Supported Not Supported Not Supported Supported
Split Supported Supported***Supported Supported Supported* Supported
Squeeze Supported Supported** Supported Supported Not Supported Supported
StridedSlice Supported Supported** Not Supported Not Supported Not Supported Supported
Tan Supported Supported** Not Supported Not Supported Not Supported Supported
TensorIterator Not Supported Supported Supported Supported Not Supported Not Supported
Tile Supported** Supported***Supported Not Supported Not Supported Supported
TopK Supported Supported** Not Supported Not Supported Not Supported Supported
Unpooling Supported Not Supported Not Supported Not Supported Not Supported Not Supported
Unsqueeze Supported Supported** Supported Supported Not Supported Supported
Upsampling Supported Not Supported Not Supported Not Supported Not Supported Not Supported

*- support is limited to the specific parameters. Refer to "Known Layers Limitation" section for the device from the list of supported.

**- support is implemented via Extensibility mechanism.

***- supports NCDHW layout.