The Inference Engine can infer models in different formats with various input and output formats. This section provides supported and optimal configurations per device.
NOTE: With OpenVINO™ 2020.4 release, Intel® Movidius™ Neural Compute Stick is no longer supported.
The Inference Engine provides unique capabilities to infer deep learning models on the following device types with corresponding plugins:
Plugin | Device types |
---|---|
GPU plugin | Intel® Processor Graphics, including Intel® HD Graphics and Intel® Iris® Graphics |
CPU plugin | Intel® Xeon® with Intel® Advanced Vector Extensions 2 (Intel® AVX2), Intel® Advanced Vector Extensions 512 (Intel® AVX-512), and AVX512_BF16, Intel® Core™ Processors with Intel® AVX2, Intel® Atom® Processors with Intel® Streaming SIMD Extensions (Intel® SSE) |
VPU plugins (available in the Intel® Distribution of OpenVINO™ toolkit) | Intel® Neural Compute Stick 2 powered by the Intel® Movidius™ Myriad™ X, Intel® Vision Accelerator Design with Intel® Movidius™ VPUs |
GNA plugin (available in the Intel® Distribution of OpenVINO™ toolkit) | Intel® Speech Enabling Developer Kit, Amazon Alexa* Premium Far-Field Developer Kit, Intel® Pentium® Silver J5005 Processor, Intel® Pentium® Silver N5000 Processor, Intel® Celeron® J4005 Processor, Intel® Celeron® J4105 Processor, Intel® Celeron® Processor N4100, Intel® Celeron® Processor N4000, Intel® Core™ i3-8121U Processor, Intel® Core™ i7-1065G7 Processor, Intel® Core™ i7-1060G7 Processor, Intel® Core™ i5-1035G4 Processor, Intel® Core™ i5-1035G7 Processor, Intel® Core™ i5-1035G1 Processor, Intel® Core™ i5-1030G7 Processor, Intel® Core™ i5-1030G4 Processor, Intel® Core™ i3-1005G1 Processor, Intel® Core™ i3-1000G1 Processor, Intel® Core™ i3-1000G4 Processor |
Multi-Device plugin | Multi-Device plugin enables simultaneous inference of the same network on several Intel® devices in parallel |
Heterogeneous plugin | Heterogeneous plugin enables automatic inference splitting between several Intel® devices (for example if a device doesn't support certain layers). |
The Inference Engine can inference models in different formats with various input and output formats. This chapter provides supported and optimal configurations for each plugin.
Acronym/Term | Description |
---|---|
DL | Deep Learning |
FP32 format | Single-precision floating-point format |
BF16 format | Brain floating-point format |
FP16 format | Half-precision floating-point format |
I16 format | 2-byte signed integer format |
I8 format | 1-byte signed integer format |
U16 format | 2-byte unsigned integer format |
U8 format | 1-byte unsigned integer format |
NHWC, NCHW - Image data layout. Refers to the representation of batches of images. NCDHW - Images sequence data layout.
CHW, NC, C - Tensor memory layout. For example, the CHW value at index (c,h,w) is physically located at index (c*H+h)*W+w, for others by analogy
Plugin | FP32 | FP16 | I8 |
---|---|---|---|
CPU plugin | Supported and preferred | Supported | Supported |
GPU plugin | Supported | Supported and preferred | Supported* |
VPU plugins | Not supported | Supported | Not supported |
GNA plugin | Supported | Supported | Not supported |
* - currently, only limited set of topologies might benefit from enabling I8 model on GPU
For Multi-Device and Heterogeneous execution the supported models formats depends on the actual underlying devices. Generally, FP16 is preferable as it is most ubiquitous and performant.
Plugin | FP32 | FP16 | U8 | U16 | I8 | I16 |
---|---|---|---|---|---|---|
CPU plugin | Supported | Not supported | Supported | Supported | Not supported | Supported |
GPU plugin | Supported | Supported* | Supported* | Supported* | Not supported | Supported* |
VPU plugins | Supported | Supported | Supported | Not supported | Not supported | Not supported |
GNA plugin | Supported | Not supported | Supported | Not supported | Supported | Supported |
* - Supported via SetBlob
only, GetBlob
returns FP32
For Multi-Device and Heterogeneous execution the supported input precision depends on the actual underlying devices. Generally, U8 is preferable as it is most ubiquitous.
Plugin | FP32 | FP16 |
---|---|---|
CPU plugin | Supported | Not supported |
GPU plugin | Supported | Supported |
VPU plugins | Supported | Supported |
GNA plugin | Supported | Not supported |
For Multi-Device and Heterogeneous execution the supported output precision depends on the actual underlying devices. Generally, FP32 is preferable as it is most ubiquitous.
Plugin | NCDHW | NCHW | NHWC | NC |
---|---|---|---|---|
CPU plugin | Supported | Supported | Supported | Supported |
GPU plugin | Supported | Supported | Supported | Supported |
VPU plugins | Not supported | Supported | Supported | Supported |
GNA plugin | Not supported | Supported | Supported | Supported |
Number of dimensions | 5 | 4 | 3 | 2 | 1 |
---|---|---|---|---|---|
Layout | NCDHW | NCHW | CHW | NC | C |
For setting relevant configuration, refer to the Integrate with Customer Application New Request API topic (step 3 "Configure input and output").
The following layers are supported by the plugins and by Shape Inference feature:
Layers | GPU | CPU | VPU | GNA | ShapeInfer |
---|---|---|---|---|---|
Abs | Supported | Supported** | Supported | Not Supported | Supported |
Acos | Supported | Supported** | Not Supported | Not Supported | Supported |
Acosh | Supported | Supported** | Not Supported | Not Supported | Supported |
Activation-Clamp | Supported | Supported*** | Supported | Supported | Supported |
Activation-ELU | Supported | Supported*** | Supported | Not Supported | Supported |
Activation-Exp | Supported | Supported*** | Not Supported | Supported | Supported |
Activation-Leaky ReLU | Supported | Supported*** | Supported | Supported | Supported |
Activation-Not | Supported | Supported*** | Not Supported | Not Supported | Supported |
Activation-PReLU | Supported | Supported*** | Supported | Not Supported | Supported |
Activation-ReLU | Supported | Supported*** | Supported | Supported | Supported |
Activation-ReLU6 | Supported | Supported*** | Supported | Not Supported | Supported |
Activation-Sigmoid/Logistic | Supported | Supported*** | Supported | Supported | Supported |
Activation-TanH | Supported | Supported*** | Supported | Supported | Supported |
ArgMax | Supported | Supported** | Supported | Not Supported | Supported |
Asin | Supported | Supported** | Not Supported | Not Supported | Supported |
Asinh | Supported | Supported** | Not Supported | Not Supported | Supported |
Atan | Supported | Supported** | Not Supported | Not Supported | Supported |
Atanh | Supported | Supported** | Not Supported | Not Supported | Supported |
BatchNormalization | Supported | Supported | Supported | Not Supported | Supported |
BinaryConvolution | Supported | Supported | Not Supported | Not Supported | Supported |
Broadcast | Supported | Supported** | Supported | Not Supported | Supported |
Ceil | Supported | Supported** | Not Supported | Not Supported | Supported |
Concat | Supported | Supported*** | Supported | Supported | Supported |
Const | Supported | Supported | Supported | Supported | Not Supported |
Convolution-Dilated | Supported | Supported | Supported | Not Supported | Supported |
Convolution-Dilated 3D | Supported | Supported | Not Supported | Not Supported | Not Supported |
Convolution-Grouped | Supported | Supported | Supported | Not Supported | Supported |
Convolution-Grouped 3D | Supported | Supported | Not Supported | Not Supported | Not Supported |
Convolution-Ordinary | Supported | Supported | Supported | Supported* | Supported |
Convolution-Ordinary 3D | Supported | Supported | Not Supported | Not Supported | Not Supported |
Cos | Supported | Supported** | Not Supported | Not Supported | Supported |
Cosh | Supported | Supported** | Not Supported | Not Supported | Supported |
Crop | Supported | Supported | Supported | Supported | Supported |
CTCGreedyDecoder | Supported** | Supported** | Supported* | Not Supported | Supported |
Deconvolution | Supported | Supported | Supported | Not Supported | Supported |
Deconvolution 3D | Supported | Supported | Not Supported | Not Supported | Not Supported |
DeformableConvolution | Supported | Supported | Not Supported | Not Supported | Supported |
DepthToSpace | Supported | Supported** | Not Supported | Not Supported | Supported |
DetectionOutput | Supported | Supported** | Supported* | Not Supported | Supported |
Eltwise-And | Supported | Supported*** | Not Supported | Not Supported | Supported |
Eltwise-Add | Supported | Supported*** | Not Supported | Not Supported | Supported |
Eltwise-Div | Supported | Supported*** | Supported | Not Supported | Supported |
Eltwise-Equal | Supported | Supported*** | Supported | Not Supported | Supported |
Eltwise-FloorMod | Supported | Supported*** | Supported | Not Supported | Supported |
Eltwise-Greater | Supported | Supported*** | Supported | Not Supported | Supported |
Eltwise-GreaterEqual | Supported | Supported*** | Supported | Not Supported | Supported |
Eltwise-Less | Supported | Supported*** | Supported | Not Supported | Supported |
Eltwise-LessEqual | Supported | Supported*** | Supported | Not Supported | Supported |
Eltwise-LogicalAnd | Supported | Supported*** | Supported | Not Supported | Supported |
Eltwise-LogicalOr | Supported | Supported*** | Supported | Not Supported | Supported |
Eltwise-LogicalXor | Supported | Supported*** | Supported | Not Supported | Supported |
Eltwise-Max | Supported | Supported*** | Supported | Not Supported | Supported |
Eltwise-Min | Supported | Supported*** | Supported | Not Supported | Supported |
Eltwise-Mul | Supported | Supported*** | Supported | Supported | Supported |
Eltwise-NotEqual | Supported | Supported*** | Supported | Not Supported | Supported |
Eltwise-Pow | Supported | Supported*** | Supported | Not Supported | Supported |
Eltwise-Prod | Supported | Supported*** | Supported | Supported | Supported |
Eltwise-SquaredDiff | Supported | Supported*** | Supported | Not Supported | Supported |
Eltwise-Sub | Supported | Supported*** | Supported | Supported | Supported |
Eltwise-Sum | Supported | Supported*** | Supported | Supported | Supported |
Erf | Supported | Supported** | Not Supported | Not Supported | Supported |
Exp | Supported | Supported | Not Supported | Supported | Supported |
FakeQuantize | Not Supported | Supported | Not Supported | Not Supported | Supported |
Fill | Not Supported | Supported** | Not Supported | Not Supported | Supported |
Flatten | Supported | Supported | Supported | Not Supported | Supported |
Floor | Supported | Supported** | Not Supported | Not Supported | Supported |
FullyConnected (Inner Product) | Supported | Supported*** | Supported | Supported | Supported |
Gather | Supported | Supported** | Supported | Not Supported | Supported |
GatherTree | Not Supported | Supported** | Not Supported | Not Supported | Supported |
Gemm | Supported | Supported | Supported | Not Supported | Supported |
GRN | Supported** | Supported** | Supported | Not Supported | Supported |
HardSigmoid | Supported | Supported** | Not Supported | Not Supported | Supported |
Interp | Supported** | Supported** | Supported | Not Supported | Supported* |
Log | Supported | Supported** | Supported | Supported | Supported |
LRN (Norm) | Supported | Supported | Supported | Not Supported | Supported |
LSTMCell | Supported | Supported | Supported | Supported | Not Supported |
GRUCell | Supported | Supported | Not Supported | Not Supported | Not Supported |
RNNCell | Supported | Supported | Not Supported | Not Supported | Not Supported |
LSTMSequence | Supported | Supported | Supported | Not Supported | Not Supported |
GRUSequence | Supported | Supported | Not Supported | Not Supported | Not Supported |
RNNSequence | Supported | Supported | Not Supported | Not Supported | Not Supported |
LogSoftmax | Supported | Supported** | Not Supported | Not Supported | Not Supported |
Memory | Not Supported | Supported | Not Supported | Supported | Supported |
MVN | Supported | Supported** | Supported* | Not Supported | Supported |
Neg | Supported | Supported** | Not Supported | Not Supported | Supported |
NonMaxSuppression | Not Supported | Supported** | Not Supported | Not Supported | Supported |
Normalize | Supported | Supported** | Supported* | Not Supported | Supported |
OneHot | Supported | Supported** | Not Supported | Not Supported | Supported |
Pad | Supported | Supported** | Supported* | Not Supported | Supported |
Permute | Supported | Supported | Supported | Supported* | Supported |
Pooling(AVG,MAX) | Supported | Supported | Supported | Supported | Supported |
Pooling(AVG,MAX) 3D | Supported | Supported | Not Supported | Not Supported | Not Supported |
Power | Supported | Supported** | Supported | Supported* | Supported |
PowerFile | Not Supported | Supported** | Not Supported | Not Supported | Not Supported |
PriorBox | Supported | Supported** | Supported | Not Supported | Supported |
PriorBoxClustered | Supported** | Supported** | Supported | Not Supported | Supported |
Proposal | Supported | Supported** | Supported | Not Supported | Supported |
PSROIPooling | Supported | Supported** | Supported | Not Supported | Supported |
Range | Not Supported | Supported** | Not Supported | Not Supported | Supported |
Reciprocal | Supported | Supported** | Not Supported | Not Supported | Supported |
ReduceAnd | Supported | Supported** | Not Supported | Not Supported | Supported |
ReduceL1 | Supported | Supported** | Not Supported | Not Supported | Supported |
ReduceL2 | Supported | Supported** | Not Supported | Not Supported | Supported |
ReduceLogSum | Supported | Supported** | Not Supported | Not Supported | Supported |
ReduceLogSumExp | Supported | Supported** | Not Supported | Not Supported | Supported |
ReduceMax | Supported | Supported** | Not Supported | Not Supported | Supported |
ReduceMean | Supported | Supported** | Not Supported | Not Supported | Supported |
ReduceMin | Supported | Supported** | Not Supported | Not Supported | Supported |
ReduceOr | Supported | Supported** | Not Supported | Not Supported | Supported |
ReduceProd | Supported | Supported** | Not Supported | Not Supported | Supported |
ReduceSum | Supported | Supported** | Not Supported | Not Supported | Supported |
ReduceSumSquare | Supported | Supported** | Not Supported | Not Supported | Supported |
RegionYolo | Supported | Supported** | Supported | Not Supported | Supported |
ReorgYolo | Supported | Supported** | Supported | Not Supported | Supported |
Resample | Supported | Supported** | Supported | Not Supported | Supported |
Reshape | Supported | Supported*** | Supported | Supported | Supported* |
ReverseSequence | Supported | Supported** | Supported | Not Supported | Supported |
RNN | Not Supported | Supported | Supported | Not Supported | Not Supported |
ROIPooling | Supported* | Supported | Supported | Not Supported | Supported |
ScaleShift | Supported | Supported*** | Supported* | Supported | Supported |
ScatterUpdate | Not Supported | Supported** | Not Supported | Not Supported | Supported |
Select | Supported | Supported | Supported | Not Supported | Supported |
Selu | Supported | Supported** | Not Supported | Not Supported | Supported |
ShuffleChannels | Supported | Supported** | Not Supported | Not Supported | Supported |
Sign | Supported | Supported** | Supported | Not Supported | Supported |
Sin | Supported | Supported** | Not Supported | Not Supported | Supported |
Sinh | Supported | Supported** | Not Supported | Not Supported | Supported |
SimplerNMS | Supported | Supported** | Not Supported | Not Supported | Supported |
Slice | Supported | Supported*** | Supported | Supported | Supported |
SoftMax | Supported | Supported*** | Supported | Not Supported | Supported |
Softplus | Supported | Supported** | Not Supported | Not Supported | Supported |
Softsign | Supported | Supported** | Not Supported | Supported | Supported |
SpaceToDepth | Not Supported | Supported** | Not Supported | Not Supported | Supported |
SpatialTransformer | Not Supported | Supported** | Not Supported | Not Supported | Supported |
Split | Supported | Supported*** | Supported | Supported | Supported |
Squeeze | Supported | Supported** | Supported | Supported | Supported |
StridedSlice | Supported | Supported** | Not Supported | Not Supported | Supported |
Tan | Supported | Supported** | Not Supported | Not Supported | Supported |
TensorIterator | Not Supported | Supported | Supported | Supported | Not Supported |
Tile | Supported** | Supported*** | Supported | Not Supported | Supported |
TopK | Supported | Supported** | Not Supported | Not Supported | Supported |
Unpooling | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
Unsqueeze | Supported | Supported** | Supported | Supported | Supported |
Upsampling | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
*- support is limited to the specific parameters. Refer to "Known Layers Limitation" section for the device from the list of supported.
**- support is implemented via Extensibility mechanism.
***- supports NCDHW layout.