Integrate the Inference Engine with Your Application

This section provides a high-level description of the process of integrating the Inference Engine into your application. Refer to the Hello Classification Sample sources for example of using the Inference Engine in applications.

NOTE: For 2019 R2 Release, the new Inference Engine Core API is introduced. This guide is updated to reflect the new API approach. The Inference Engine Plugin API is still supported, but is going to be deprecated in future releases. Please, refer to Migration from Inference Engine Plugin API to Core API guide to update your application.

Use the Inference Engine API in Your Code

The core library implements loading and parsing a model Intermediate Representation (IR), and triggers inference using a specified device. The core library has the following API:

C++ Inference Engine API wraps the capabilities of core library:

Integration Steps

Integration process includes the following steps:


1) Create Inference Engine Core to manage available devices and read network objects:

2) Read a model IR created by the Model Optimizer (.xml is supported format):

auto network = core.ReadNetwork("Model.xml");

Or read the model from ONNX format (.onnx and .prototxt are supported formats)

auto network = core.ReadNetwork("model.onnx");

3) Configure input and output. Request input and output information using InferenceEngine::CNNNetwork::getInputsInfo(), and InferenceEngine::CNNNetwork::getOutputsInfo() methods:

/** Take information about all topology inputs **/
InferenceEngine::InputsDataMap input_info = network.getInputsInfo();
/** Take information about all topology outputs **/
InferenceEngine::OutputsDataMap output_info = network.getOutputsInfo();

Optionally, set the number format (precision) and memory layout for inputs and outputs. Refer to the Supported configurations chapter to choose the relevant configuration.

You can also allow input of any size. To do this, mark each input as resizable by setting a desired resize algorithm (e.g. BILINEAR) inside of the appropriate input info.

Basic color format conversions are supported as well. By default, the Inference Engine assumes that the input color format is BGR and color format conversions are disabled. The Inference Engine supports the following color format conversions:

where X is a channel that will be ignored during inference. To enable the conversions, set a desired color format (for example, RGB) for each input inside of the appropriate input info.

If you want to run inference for multiple images at once, you can use the built-in batch pre-processing functionality.

NOTE: Batch pre-processing is not supported if input color format is set to ColorFormat::NV12.

You can use the following code snippet to configure input and output:

/** Iterate over all input info**/
for (auto &item : input_info) {
auto input_data = item.second;
/** Iterate over all output info**/
for (auto &item : output_info) {
auto output_data = item.second;

NOTE: NV12 input color format pre-processing differs from other color conversions. In case of NV12, Inference Engine expects two separate image planes (Y and UV). You must use a specific InferenceEngine::NV12Blob object instead of default blob object and set this blob to the Inference Engine Infer Request using InferenceEngine::InferRequest::SetBlob(). Refer to Hello NV12 Input Classification C++ Sample for more details.

If you skip this step, the default values are set:

Number of dimensions 5 4 3 2 1

4) Load the model to the device using InferenceEngine::Core::LoadNetwork():

auto executable_network = core.LoadNetwork(network, "CPU");

It creates an executable network from a network object. The executable network is associated with single hardware device. It is possible to create as many networks as needed and to use them simultaneously (up to the limitation of the hardware resources). Third parameter is a configuration for plugin. It is map of pairs: (parameter name, parameter value). Choose device from Supported devices page for more details about supported configuration parameters.

/** Optional config. E.g. this enables profiling of performance counters. **/
std::map<std::string, std::string> config = {{ PluginConfigParams::KEY_PERF_COUNT, PluginConfigParams::YES }};
auto executable_network = core.LoadNetwork(network, "CPU", config);

5) Create an infer request:

auto infer_request = executable_network.CreateInferRequest();

6) Prepare input. You can use one of the following options to prepare input:


  • SetBlob() method compares precision and layout of an input blob with ones defined on step 3 and throws an exception if they do not match. It also compares a size of the input blob with input size of the read network. But if input was configured as resizable, you can set an input blob of any size (for example, any ROI blob). Input resize will be invoked automatically using resize algorithm configured on step 3. Similarly to the resize, color format conversions allow the color format of an input blob to differ from the color format of the read network. Color format conversion will be invoked automatically using color format configured on step 3.
  • GetBlob() logic is the same for pre-processable and not pre-processable input. Even if it is called with input configured as resizable or as having specific color format, a blob allocated by an infer request is returned. Its size and color format are already consistent with the corresponding values of the read network. No pre-processing will happen for this blob. If you call GetBlob() after SetBlob(), you will get the blob you set in SetBlob().

7) Do inference by calling the InferenceEngine::InferRequest::StartAsync and InferenceEngine::InferRequest::Wait methods for asynchronous request:


or by calling the InferenceEngine::InferRequest::Infer method for synchronous request:


StartAsync returns immediately and starts inference without blocking main thread, Infer blocks main thread and returns when inference is completed. Call Wait for waiting result to become available for asynchronous request.

There are three ways to use it:

Both requests are thread-safe: can be called from different threads without fearing corruption and failures.

Multiple requests for single ExecutableNetwork are executed sequentially one by one in FIFO order.

While request is ongoing, all its methods except InferenceEngine::InferRequest::Wait would throw an exception.

8) Go over the output blobs and process the results. Note that casting Blob to TBlob via std::dynamic_pointer_cast is not recommended way, better to access data via buffer() and as() methods as follows:

for (auto &item : output_info) {
auto output_name = item.first;
auto output = infer_request.GetBlob(output_name);
auto const memLocker = output->cbuffer(); // use const memory locker
// output_buffer is valid as long as the lifetime of memLocker
const float *output_buffer =<const float *>();
/** output_buffer[] - accessing output blob data **/

Build Your Application

For details about building your application, refer to the CMake files for the sample applications. All samples source code is located in the <INSTALL_DIR>/openvino/inference_engine/samples directory, where INSTALL_DIR is the OpenVINO™ installation directory.

CMake project creation

  1. Create a structure for the project:
    ├── CMakeLists.txt - CMake file to build
    ├── ... - Additional folders like includes/
    └── src/ - source folder
    └── main.cpp
    build/ - build directory
  2. Include Inference Engine, nGraph and OpenCV libraries in project/CMakeLists.txt OpenCV integration is needed mostly for pre-processing input data and ngraph for more complex applications using ngraph API.
    cmake_minimum_required(VERSION 3.0.0)
    find_package(ngraph REQUIRED)
    find_package(InferenceEngine REQUIRED)
    find_package(OpenCV REQUIRED)
    add_executable(${PROJECT_NAME} src/main.cpp)
    target_link_libraries(${PROJECT_NAME} PRIVATE ${InferenceEngine_LIBRARIES} ${OpenCV_LIBS} ${NGRAPH_LIBRARIES})
  3. To build your project using CMake with the default build tools currently available on your machine, execute the following commands:

    NOTE: Make sure Set the Environment Variables step in OpenVINO Installation document is applied to your terminal, otherwise InferenceEngine_DIR and OpenCV_DIR variables won't be configured properly to pass find_package calls.

    cd build/
    cmake ../project
    cmake --build .
    It's allowed to specify additional build options (e.g. to build CMake project on Windows with a specific build tools). Please refer to the CMake page for details.

Run Your Application

NOTE: Before running, make sure you completed Set the Environment Variables section in OpenVINO Installation document so that the application can find the libraries.

To run compiled applications on Microsoft* Windows* OS, make sure that Microsoft* Visual C++ 2017 Redistributable and Intel® C++ Compiler 2017 Redistributable packages are installed and <INSTALL_DIR>/bin/intel64/Release/*.dll files are placed to the application folder or accessible via PATH% environment variable.