NOTE: Intel® System Studio (click "Intel® System Studio Users" tab) is an all-in-one, cross-platform tool suite, purpose-built to simplify system bring-up and improve system and IoT device application performance on Intel® platforms. If you are using the Intel® Distribution of OpenVINO™ with Intel® System Studio, go to Get Started with Intel® System Studio.
This Guide provides an overview of the Inference Engine describing the typical workflow for performing inference of a pre-trained and optimized deep learning model and a set of sample applications.
NOTE: Before you perform inference with the Inference Engine, your models should be converted to the Inference Engine format using the Model Optimizer or built directly in runtime using nGraph API. To learn about how to use Model Optimizer, refer to the Model Optimizer Developer Guide. To learn about the pre-trained and optimized models delivered with the OpenVINO™ toolkit, refer to Pre-Trained Models.
After you have used the Model Optimizer to create an Intermediate Representation (IR), use the Inference Engine to infer the result for a given input data.
Inference Engine is a set of C++ libraries providing a common API to deliver inference solutions on the platform of your choice: CPU, GPU, or VPU. Use the Inference Engine API to read the Intermediate Representation, set the input and output formats, and execute the model on devices. While the C++ libraries is the primary implementation, C libraries and Python bindings are also available.
For Intel® Distribution of OpenVINO™ toolkit, Inference Engine binaries are delivered within release packages.
To learn about how to use the Inference Engine API for your application, see the Integrating Inference Engine in Your Application documentation.
For complete API Reference, see the Inference Engine API References section.
Inference Engine uses a plugin architecture. Inference Engine plugin is a software component that contains complete implementation for inference on a certain Intel® hardware device: CPU, GPU, VPU, etc. Each plugin implements the unified API and provides additional hardware-specific APIs.
Your application must link to the core Inference Engine libraries:
libinference_engine.so, which depends on
inference_engine.dll, which depends on
libinference_engine.dylib, which depends on
The required C++ header files are located in the
This library contains the classes to:
Starting from 2020.4 release, Inference Engine introduced a concept of
CNNNetwork reader plugins. Such plugins can be automatically dynamically loaded by Inference Engine in runtime depending on file format:
libinference_engine_ir_reader.soto read a network from IR
libinference_engine_onnx_reader.soto read a network from ONNX model format
inference_engine_ir_reader.dllto read a network from IR
inference_engine_onnx_reader.dllto read a network from ONNX model format
For each supported target device, Inference Engine provides a plugin — a DLL/shared library that contains complete implementation for inference on this particular device. The following plugins are available:
|CPU||Intel® Xeon® with Intel® AVX2 and AVX512, Intel® Core™ Processors with Intel® AVX2, Intel® Atom® Processors with Intel® SSE|
|GPU||Intel® Processor Graphics, including Intel® HD Graphics and Intel® Iris® Graphics|
|MYRIAD||Intel® Neural Compute Stick 2 powered by the Intel® Movidius™ Myriad™ X|
|GNA||Intel® Speech Enabling Developer Kit, Amazon Alexa* Premium Far-Field Developer Kit, Intel® Pentium® Silver J5005 Processor, Intel® Pentium® Silver N5000 Processor, Intel® Celeron® J4005 Processor, Intel® Celeron® J4105 Processor, Intel® Celeron® Processor N4100, Intel® Celeron® Processor N4000, Intel® Core™ i3-8121U Processor, Intel® Core™ i7-1065G7 Processor, Intel® Core™ i7-1060G7 Processor, Intel® Core™ i5-1035G4 Processor, Intel® Core™ i5-1035G7 Processor, Intel® Core™ i5-1035G1 Processor, Intel® Core™ i5-1030G7 Processor, Intel® Core™ i5-1030G4 Processor, Intel® Core™ i3-1005G1 Processor, Intel® Core™ i3-1000G1 Processor, Intel® Core™ i3-1000G4 Processor|
|HETERO||Automatic splitting of a network inference between several devices (for example if a device doesn't support certain layers|
|MULTI||Simultaneous inference of the same network on several devices in parallel|
The table below shows the plugin libraries and additional dependencies for Linux, Windows and macOS platforms.
|Plugin||Library name for Linux||Dependency libraries for Linux||Library name for Windows||Dependency libraries for Windows||Library name for macOS||Dependency libraries for macOS|
|GPU||Is not supported||-|
|HDDL||Is not supported||-|
|GNA||Is not supported||-|
|HETERO||Same as for selected plugins||Same as for selected plugins||Same as for selected plugins|
|MULTI||Same as for selected plugins||Same as for selected plugins||Same as for selected plugins|
NOTE: All plugin libraries also depend on core Inference Engine libraries.
Make sure those libraries are in your computer's path or in the place you pointed to in the plugin loader. Make sure each plugin's related dependencies are in the:
On Linux and macOS, use the script
bin/setupvars.sh to set the environment variables.
On Windows, run the
bin\setupvars.bat batch file to set the environment variables.
To learn more about supported devices and corresponding plugins, see the Supported Devices chapter.
The common workflow contains the following steps:
InferenceEngine::Coreobject to work with different devices, all device plugins are managed internally by the
Coreobject. Register extensions with custom nGraph operations (
InferenceEngine::Coreclass, read an Intermediate Representation file into an object of the
InferenceEngine::CNNNetworkclass. This class represents the network in the host memory.
InferenceEngine::Core::SetConfig), and register extensions to this device (
InferenceEngine::Core::LoadNetwork()method with specific device (e.g.
GPU, etc.) to compile and load the network on the device. Pass in the per-target load configuration for this compilation and load operation.
InferenceEngine::ExecutableNetworkobject. Use this object to create an
InferenceEngine::InferRequestin which you signal the input buffers to use for input and output. Specify a device-allocated memory and copy it into the device memory directly, or tell the device to use your application memory to save a copy.
For more details on the Inference Engine API, refer to the Integrating Inference Engine in Your Application documentation.