Overview of Inference Engine Python* API

NOTE: It is a preview version of the Inference Engine Python* API for evaluation purpose only. Module structure and API itself may be changed in future releases.

This API provides a simplified interface for Inference Engine functionality that allows to:

• handle the models
• load and configure Inference Engine plugins based on device names
• perform inference in synchronous and asynchronous modes with arbitrary number of infer requests (the number of infer requests may be limited by target device capabilities)

Supported OSes

Currently the Inference Engine Python* API is supported on Ubuntu* 16.04 and 18.04, Windows* 10, macOS* 10.x and CentOS* 7.3 OSes. Supported Python* versions:

Operating System	Supported Python* versions:
Ubuntu* 16.04	2.7, 3.5, 3.6, 3.7
Ubuntu* 18.04	2.7, 3.5, 3.6, 3.7
Windows* 10	3.5, 3.6, 3.7
CentOS* 7.3	3.4, 3.5, 3.6, 3.7
macOS* 10.x	3.5, 3.6, 3.7

Setting Up the Environment

To configure the environment for the Inference Engine Python* API, run:

• On Ubuntu* 16.04 or 18.04, CentOS* 7.4 or macOS* 10.x: source <INSTALL_DIR>/bin/setupvars.sh .
• On Windows* 10: call <INSTALL_DIR>\deployment_tools\inference_engine\python_api\setenv.bat

The script automatically detects latest installed Python* version and configures required environment if the version is supported. If you want to use certain version of Python*, set the environment variable PYTHONPATH=<INSTALL_DIR>/deployment_tools/inference_engine/python_api/<desired_python_version> after running the environment configuration script.

IECore

This class represents an Inference Engine entity and allows you to manipulate with plugins using unified interfaces.

Class Constructor

__init__(xml_config_file: str = "")

• Parameters:
  • xml_config_file - A full path to .xml file containing plugins configuration. If the parameter is not specified, the default configuration is handled automatically.
• Usage examples:
  • Initialize an IECore object with default configuration:

    ie = IECore()

  • Initialize an IECore object with a custom configuration location specified:

    ie = IECore("/localdisk/plugins/my_custom_cfg.xml")

    .xml file has the following structure:

    <ie>
        <plugins>
            <plugin name="" location="" optional="yes/no">
                <extensions>
                    <extension location=""/>
                </extensions>
                <properties>
                    <property key="" value=""/>
                </properties>
            </plugin>
        </plugin>
    </ie>

Class Attributes

• available_devices - A vector of devices. The devices are returned as [CPU, FPGA.0, FPGA.1, MYRIAD]. If there are more than one device of a specific type, they all are listed followed by a dot and a number.

Instance Methods

• get_versions(device_name: str)
  • Description: Returns a namedtuple object with versions of the plugin specified
  • Parameters:
    • device_name - Name of the the registered plugin
  • Return value: Dictionary mapping a plugin name and Versions namedtuple object with the following fields:
    • major - major plugin integer version
    • minor - minor plugin integer version
    • build_number - plugin build number string
    • description - plugin description string
  • Usage example:

    ie = IECore()
    ver = ie.get_versions("CPU")["CPU"]
    print("{descr}: {maj}.{min}.{num}".format(descr=ver.description, maj=ver.major, min=ver.minor, num=ver.build_number))

• load_network(network: IENetwork, device_name: str, config=None, num_requests: int=1)
  • Description: Loads a network that was read from the Intermediate Representation (IR) to the plugin with specified device name and creates an ExecutableNetwork object of the IENetwork class. You can create as many networks as you need and use them simultaneously (up to the limitation of the hardware resources).
  • Parameters:
    • network - A valid IENetwork instance
    • device_name - A device name of a target plugin
    • num_requests - A positive integer value of infer requests to be created. Number of infer requests is limited by device capabilities.
    • config - A dictionary of plugin configuration keys and their values
  • Return value: An ExecutableNetwork object
  • Usage example:

    net = IENetwork(model=path_to_xml_file, weights=path_to_bin_file)
    ie = IECore()
    exec_net = plugin.load_network(network=net, device_name="CPU", num_requsts=2)

• query_network(network: IENetwork, device_name: str, config=None)
  • Description: Queries the plugin with specified device name what network layers are supported in the current configuration. Please note that layers support depends on plugin configuration and loaded extensions.
  • Parameters:
    • network - A valid IENetwork instance
    • device_name - A device name of a target plugin
    • config - A dictionary of plugin configuration keys and their values
  • Return value: A dictionary mapping layers and device names on which they are supported
  • Usage example:

    net = IENetwork(model=path_to_xml_file, weights=path_to_bin_file)
    ie = IECore()
    exec_net = plugin.query_network(network=net, device_name="HETERO:GPU,CPU")

• set_config(config: dict, device_name: str)
  • Description: Sets a configuration for a plugin
  • Parameters:
    • config - a dictionary of configuration parameters as keys and their values
    • device_name - a device name of a target plugin
  • Return value: None
  • Usage examples: See the set_affinity method of the IENetwork class.
• register_plugin(plugin_name: str, device_name: str = "")
  • Description: Registers a new device and a plugin which implement this device inside Inference Engine.
  • Parameters:
    • plugin_name - A name of a plugin. Depending on a platform, plugin_name is wrapped with a shared library suffix and a prefix to identify a full name of the library
    • device_name - A target device name for the plugin. If not specified, the method registers a plugin with the default name.
  • Return value: None
  • Usage examples:

    ie = IECore()
    ie.register_plugin(plugin="MKLDNNPlugin", device_name="MY_NEW_PLUGIN")

• register_plugins(xml_config_file: str)
  • Description: Registers plugins specified in an .xml configuration file
  • Parameters:
    • xml_config_file - A full path to .xml file containing plugins configuration
  • Return value: None
  • Usage examples:

    ie = IECore()
    ie.register_plugins("/localdisk/plugins/my_custom_cfg.xml")

• unregister_plugin(device_name: str = "")
  • Description: Unregisters a plugin with a specified device name
  • Parameters:
    • device_name - A device name of the plugin to unregister
  • Return value: None
  • Usage examples:

    ie = IECore()
    plugin = IEPlugin("GPU")
    ie.register_plugin(plugin=plugin, device_name="MY_NEW_GPU")
    ie.unregister_plugin(device_name="GPU")

• add_extension(extension_path: str, device_name: str)
  • Description: Loads extension library to the plugin with a specified device name
  • Parameters:
    • extension_path - Path to the extensions library file to load to a plugin
    • device_name - A device name of a plugin to load the extensions to
  • Return value: None
  • Usage examples:

    ie = IECore()
    ie.add_extension(extension_path="/some_dir/libcpu_extension_avx2.so", device_name="CPU")

• get_metric(device_name: str, metric_name: str)
  • Description: Gets a general runtime metric for dedicated hardware. Enables to request common device properties, which are ExecutableNetwork agnostic, such as device name, temperature, and other devices-specific values.
  • Parameters:
    • device_name - A name of a device to get a metric value.
    • metric_name - A metric name to request.
  • Return value: A metric value corresponding to a metric key.
  • Usage example

    ie = IECore()
    ie.get_metric(metric_name="SUPPORTED_METRICS", device_name="CPU")

• get_config(device_name: str, metric_name: str)
  • Description: Gets a configuration dedicated to device behavior. The method targets to extract information which can be set via SetConfig method.
  • Parameters:
    • device_name - A name of a device to get a metric value.
    • metric_name - A metric name to request.
  • Return value: A metric value corresponding to a metric key.
  • Usage example

    ie = IECore()
    ie.get_config(metric_name="CPU_BIND_THREAD", device_name="CPU")

IENetLayer

This class stores main information about the layer and allow to modify some layer parameters

Class Attributes

• name - Name of the layer
• type- Layer type
• precision - Layer base operating precision. Provides getter and setter interfaces.
• layout - Returns the layout of shape of the layer
• shape - Return the list of the shape of the layer
• parents - Returns a list, which contains names of layers preceding this layer
• children - Returns a list, which contains names of layers following this layer

• affinity - Layer affinity set by user or a default affinity set by the IEPlugin.set_initial_affinity() method. The affinity attribute provides getter and setter interfaces, so the layer affinity can be modified directly. For example:

  >>> net = IENetwork(model=path_to_xml_file, weights=path_to_bin_file)
  >>> plugin = IEPlugin(device="HETERO:FPGA,CPU")
  >>> plugin.set_config({"TARGET_FALLBACK": "HETERO:FPGA,CPU"})
  >>> plugin.set_initial_affinity(net)
  >>> for l in net.layers.values():
  ...     if l.type == "Convolution":
  ...         l.affinity = "CPU"
  

  To correctly set affinity for the network, you must first initialize and properly configure the HETERO plugin. set_config({"TARGET_FALLBACK": "HETERO:FPGA,GPU"}) function configures the plugin fallback devices and their order. plugin.set_initial_affinity(net) function sets affinity parameter of model layers according to its support on specified devices.

  After default affinity is set by the plugin, override the default values by setting affinity manually how it's described in example above

  To understand how default and non-default affinities are set:

  1. Call net.layers function right after model loading and check that layer affinity parameter is empty.
  2. Call plugin.set_default_affinity(net).
  3. Call net.layers and check layer affinity parameters to see how plugin set a default affinity
  4. Set layer affinity how it's described above
  5. Call net.layers again and check layer affinity parameters to see how it was changed after manual affinity setting
• weights- Dictionary with layer weights, biases or custom blobs if any
• params - Layer specific parameters. Provides getter and setter interfaces to get and modify layer parameters. Please note that some modifications can be ignored and overwriten by target plugin (e.g. modification of convolution kernel size will be reflected in layer parameters but finally the plugin will ignore it and will use initial kernel size)

IENetwork

This class contains the information about the network model read from IR and allows you to manipulate with some model parameters such as layers affinity and output layers.

Class Constructor

__init__(model: [bytes, str], weights: [bytes, str], init_from_buffer: bool=False, ngrpah_compatibility: bool=False)

• Parameters:
  • model - An .xml file of the IR. Depending on init_from_buffer value, can be a string path or bytes with file content.
  • weights - A .bin file of the IR. Depending on init_from_buffer value, can be a string path or bytes with file content.
  • init_from_buffer - Defines the way of how model and weights attributes are interpreted. If True, attributes are interpreted as strings with paths to .xml and .bin files of IR. If False, they are interpreted as Python bytes object with .xml and .bin files content.
• Usage examples:
  • Initializing IENetwork object from IR files:

    net = IENetwork(model=path_to_xml_file, weights=path_to_bin_file)

  • Initializing IENetwork object bytes with content of IR files:

    with open(path_to_bin_file, 'rb') as f:
        bin = f.read()
    with open(path_to_xml_file, 'rb') as f:
        xml = f.read()
    net = IENetwork(model=xml, weights=bin, init_from_buffer=True)

Class Attributes

• name - Name of the loaded network
• inputs - A dictionary that maps input layer names to InputInfo objects. For example, to get a shape of the input layer:

  >>> net = IENetwork(model=path_to_xml_file, weights=path_to_bin_file)
  >>> net.inputs
  {'data': <inference_engine.ie_api.InputInfo object at 0x7efe042dedd8>}
  >>> net.inputs['data'].shape
  [1, 3, 224, 224]

• outputs - A dictionary that maps output layer names to OutputInfo objects For example, to get a shape of the output layer:

  >>> net = IENetwork(model=path_to_xml_file, weights=path_to_bin_file)
  >>> net.inputs
  {'prob': <inference_engine.ie_api.OutputInfo object at 0x7efe03ab95d0>}
  >>> net.outputs['prob'].shape
  [1, 1000]

• batch_size - Batch size of the network. Provides getter and setter interfaces to get and modify the network batch size. For example:

  >>> net = IENetwork(model=path_to_xml_file, weights=path_to_bin_file)
  >>> net.batch_size
  1
  >>> net.batch_size = 4
  >>> net.batch_size
  4
  >>> net.inputs['data'].shape
      [4, 3, 224, 224]

• layers - Return dictionary that maps network layer names to IENetLayer objects containing layer properties in topological order. For example, to list all network layers:

  >>> net = IENetwork(model=path_to_xml_file, weights=path_to_bin_file)
  >>> net.layers
  {'conv0': <inference_engine.ie_api.IENetLayer object at 0x7f3a4c102370>
  ...
  }

  • stats - Returns LayersStatsMap object containing dictionary that maps network layer names to calibration statistics represented by LayerStats objects. LayersStatsMap class inherited from built-in python dict and overrides default update()method to allow to set or modify layers calibration statistics.

    >>> net = IENetwork(model=path_to_xml_file, weights=path_to_bin_file)
    >>> net.stats.update({
            "conv1_2d" : LayserStats(min=(-25, -1, 0), max=(63, 124, 70)),
            "conv2_2d" : LayserStats(min=(-5, -1, 0, 1, -7, 2), max=(63, 124, 70, 174, 99, 106)),
        })

    For more details about low precision inference please refer to "Low-Precision 8-bit Integer Inference" section in Inference Engine Developers Guide documentation.

Class Methods

• from_ir(model: str, weights: str)

  NOTE: The function is deprecated. Please use the IENetwork() class constructor to create valid instance of IENetwork.

  • Description: Reads the model from the .xml and .bin files of the IR.
  • Parameters:
    • model - Path to .xml file of the IR
    • weights - Path to .bin file of the IR
  • Return value: An instance of the IENetwork class
  • Usage example:

    >>> net = IENetwork(model=path_to_xml_file, weights=path_to_bin_file)
    >>> net
    <inference_engine.ie_api.IENetwork object at 0x7fd7dbce54b0>

Instance Methods

• add_outputs(outputs)
  • Description: Marks any intermediate layer as output layer to retrieve the inference results from the specified layers.
  • Parameters:
    • outputs - List of layer to be set as model outputs. The list can contain strings with layer names to be set as outputs or tuples with layer name as first element and output port id as second element. In case of setting one layer as output, string or tuple with one layer can be provided.
  • Return value: None
  • Usage example:

    >>> net = IENetwork(model=path_to_xml_file, weights=path_to_bin_file)
    >>> net.add_outputs(["conv5_1/dwise', conv2_1/expand'])]
    
    >>> net.outputs
    
    ['prob', 'conv5_1/dwise', 'conv2_1/expand']

    NOTE: The last layers (nodes without successors in graph representation of the model) are set as output by default. In the case above, prob layer is a default output and conv5_1/dwise, conv2_1/expand are user-defined outputs.

• reshape(input_shapes: dict)
  • Description: Reshapes the network to change spatial dimensions, batch size, or any dimension.

    NOTE: Before using this method, make sure that the target shape is applicable for the network. Changing the network shape to an arbitrary value may lead to unpredictable behaviour.

  • Parameters:
    • input_shapes - A dictionary that maps input layer names to tuples with the target shape
  • Return value: None
  • Usage example:

    >>> net = IENetwork(model=path_to_xml_file, weights=path_to_bin_file)
    >>> input_layer = next(iter(net.inputs))
    >>> n, c, h, w = net.inputs[input_layer]
    >>> net.reshape({input_layer: (n, c, h*2, w*2)}]

• serialize(path_to_xml, path_to_bin)
  • Description: Serializes the network and stores it in files.
  • Parameters:
    • path_to_xml - Path to a file, where a serialized model will be stored
    • path_to_bin - Path to a file, where serialized weights will be stored
  • Return value: None
  • Usage example:

    >>> net = IENetwork(model=path_to_model, weights=path_to_weights)
    >>> net.serialize(path_to_xml, path_to_bin)

LayerStats

Layer calibration statistic container.

Class Constructor

• __init__(min: tuple = (), max: tuple = ())
  • Parameters:
    • min - Tuple with per-channel minimum layer activation values
    • max - Tuple with per-channel maximum layer activation values

InputInfo

This class contains the information about the network input layers

Class Attributes

• precision - Precision of the input data provided by user. Provides setter and getter interfaces to get and modify input layer precision. List of applicable precisions: FP32 FP16, I32, I16, I8, U32, U16

  NOTE: Support of any calculation precision depends on the target plugin.

• layout - Layout of the input data provided by user. Provides setter and getter interfaces to get and modify input layer layout. List of applicable layouts: NCHW, NHWC, OIHW, C, CHW, HW, NC, CN, BLOCKED
• shape - input layer data shape

OutputInfo

This class contains the information about the network input layers

Class Attributes

• precision - Precision of the output data. Provides setter and getter interfaces to get and modify output layer precision.
• layout - Layout of the output data provided by user
• shape - Input layer data shape

IEPlugin Class

This class is the main plugin interface and serves to initialize and configure the plugin.

Class Constructor

• __init__(device: str, plugin_dirs=None)
  • Parameters:
    • device - Target device name. Supported devices: CPU, GPU, FPGA, MYRIAD, HETERO
    • plugin_dirs - List of paths to plugin directories

Properties

• device - a name of the device that was specified to initialize IEPlugin
• version - a version of the plugin

Instance Methods

• load(network: IENetwork, num_requests: int=1, config=None)
  • Description: Loads a network that was read from the IR to the plugin and creates an executable network from a network object. You can create as many networks as you need and use them simultaneously (up to the limitation of the hardware resources).
  • Parameters:
    • network - A valid IENetwork instance
    • num_requests - A positive integer value of infer requests to be created. Number of infer requests may be limited by device capabilities.
    • config - A dictionary of plugin configuration keys and their values
  • Return value: None
  • Usage example:

    >>> net = IENetwork(model=path_to_xml_file, weights=path_to_bin_file)
    >>> plugin = IEPlugin(device="CPU")
    >>> exec_net = plugin.load(network=net, num_requsts=2)
    >>> exec_net
    <inference_engine.ie_api.ExecutableNetwork object at 0x7f5140bbcd38>

• set_initial_affinity(net: IENetwork)
  • Description: Sets initial affinity for model layers according to the HETERO plugin logic. Applicable only if IEPlugin was initialized for a HETERO device.
  • Parameters:
    • net - A valid instance of IENetwork
  • Return value: None
  • Usage example: See affinity attribute of the IENetLayer class.
• add_cpu_extension(extension_path: str)
  • Description: Loads extensions library to the plugin. Applicable only for a CPU device and a HETERO device with CPU
  • Parameters:
    • extension_path - A full path to CPU extensions library

Return value: None

• Usage example:

  >>> plugin = IEPlugin(device="CPU")
  >>> plugin.add_cpu_extenstions(ext_lib_path)

set_config(config: dict)

• Description: Sets a configuration for the plugin. Refer to SetConfig() in Inference Engine C++ documentation for acceptable keys and values list.
• Parameters:
  • config - A dictionary of keys and values of acceptable configuration parameters
• Return value: None
• Usage examples: See set_affinity method of the IENetwork class.

get_supported_layers(net: IENetwork)

• Description: Returns the set of layers supported by the plugin. Please note that for the CPU plugin, support of a layer may depends on extension loaded by add_cpu_extenstion() method.
• Parameters:
  • net - A valid instance of IENetwork
• Return value: Set of layers supported by the plugin
• Usage example: See affinity attribute of the IENetLayer class.

ExecutableNetwork Class

This class represents a network instance loaded to plugin and ready for inference.

Class Constructor

There is no explicit class constructor. To make a valid instance of ExecutableNetwork, use load() method of the IEPlugin class.

Class Attributes

• requests - A tuple of InferRequest instances
  • Usage example:

    >>> net = IENetwork(model=path_to_xml_file, weights=path_to_bin_file)
    >>> plugin = IEPlugin(device="CPU")
    >>> exec_net = plugin.load(network=net, num_requsts=3)
    >>> exec_net.requests
    (<inference_engine.ie_api.InferRequest object at 0x7f66f56c57e0>,
    <inference_engine.ie_api.InferRequest object at 0x7f66f56c58b8>,
    <inference_engine.ie_api.InferRequest object at 0x7f66f56c5900>)

Instance Methods

• infer(inputs=None)
  • Description: Starts synchronous inference for the first infer request of the executable network and returns output data. Wraps infer() method of the InferRequest class
  • Parameters:
    • inputs - A dictionary that maps input layer names to numpy.ndarray objects of proper shape with input data for the layer
  • Return value: A dictionary that maps output layer names to numpy.ndarray objects with output data of the layer
  • Usage example:

    >>> net = IENetwork(model=path_to_xml_file, weights=path_to_bin_file)
    >>> plugin = IEPlugin(device="CPU")
    >>> exec_net = plugin.load(network=net, num_requsts=2)
    >>> res = exec_net.infer({'data': img})
    >>> res
    {'prob': array([[[[2.83426580e-08]],
                     [[2.40166020e-08]],
                     [[1.29469613e-09]],
                     [[2.95946148e-08]]
                     ......
                  ]])}

    For illustration of input data preparation, please see the samples (for example, classification_sample.py).
• start_async(request_id, inputs=None)
  • Description: Starts asynchronous inference for specified infer request. Wraps async_infer() method of the InferRequest class.
  • Parameters:
    • request_id - Index of infer request to start inference
    • inputs - A dictionary that maps input layer names to numpy.ndarray objects of proper shape with input data for the layer
  • Return value: A handler of specified infer request, which is an instance of the InferRequest class.
  • Usage example:

    >>> infer_request_handle = exec_net.start_async(request_id=0, inputs={input_blob: image})
    >>> infer_status = infer_request_handle.wait()
    >>> res = infer_request_handle.outputs[out_blob]

For more details about infer requests processing, see classification_sample_async.py (simplified case) and object_detection_demo_ssd_async.py (real asynchronous use case) samples.

• get_exec_graph_info()
  • Description: Gets executable graph information from a device
  • Parameters: None
  • Return value: An instance of IENetwork
  • Usage_example:

net = IENetwork(model=path_to_xml_file, weights=path_to_bin_file)
plugin = IEPlugin(device="CPU")
exec_net = plugin.load(network=net, num_requsts=2)
exec_graph = exec_net.get_exec_graph_info()

• get_metric(metric_name: str)
  • Description: - Gets general runtime metric for an executable network. It can be network name, actual device ID on which executable network is running or all other properties which cannot be changed dynamically.
  • Parameters:
    • metric_name - A metric name to request.
  • Return value: A metric value corresponding to a metric key.
  • Usage example

    ie = IECore()
    net = IENetwork(model=path_to_xml_file, weights=path_to_bin_file)
    exec_net = ie.load_network(net, "CPU")
    exec_net.get_metric("NETWORK_NAME")

• get_config(metric_config: str)
  • Description: - Gets configuration for current executable network. The method is responsible to extract information
    • which affects executable network execution
  • Parameters:
    • config_name - A configuration parameter name to request.
  • Return value: A configuration value corresponding to a configuration key.
  • Usage example

    ie = IECore()
    net = IENetwork(model=path_to_xml_file, weights=path_to_bin_file)
    exec_net = ie.load_network(net, "CPU")
    exec_net.get_metric("DEVICE_ID")

InferRequest Class

This class provides an interface to infer requests of ExecutableNetwork and serves to handle infer requests execution and to set and get output data.

Class Constructor

There is no explicit class constructor. To make a valid InferRequest instance, use load() method of the IEPlugin class with specified number of requests to get ExecutableNetwork instance which stores infer requests.

Class Attributes

• inputs - A dictionary that maps input layer names to numpy.ndarray objects of proper shape with input data for the layer

• outputs - A dictionary that maps output layer names to numpy.ndarray objects with output data of the layer

  Usage example:

  >>> exec_net.requests[0].inputs['data'][:] = image
  >>> exec_net.requests[0].infer()
  >>> res = exec_net.requests[0].outputs['prob']
  >>> np.flip(np.sort(np.squeeze(res)),0)
  array([4.85416055e-01, 1.70385033e-01, 1.21873841e-01, 1.18894853e-01,
         5.45198545e-02, 2.44456064e-02, 5.41366823e-03, 3.42589128e-03,
         2.26027006e-03, 2.12283316e-03 ...])

Instance Methods

It is not recommended to run inference directly on InferRequest instance. To run inference, please use simplified methods infer() and start_async() of ExecutableNetwork.

• infer(inputs=None)
  • Description: Starts synchronous inference of the infer request and fill outputs array
    • Parameters:
      • inputs - A dictionary that maps input layer names to numpy.ndarray objects of proper shape with input data for the layer
  • Return value: None
  • Usage example:

    >>> exec_net = plugin.load(network=net, num_requests=2)
    >>> exec_net.requests[0].infer({input_blob: image})
    >>> res = exec_net.requests[0].outputs['prob']
    >>> np.flip(np.sort(np.squeeze(res)),0)
    array([4.85416055e-01, 1.70385033e-01, 1.21873841e-01, 1.18894853e-01,
           5.45198545e-02, 2.44456064e-02, 5.41366823e-03, 3.42589128e-03,
           2.26027006e-03, 2.12283316e-03 ...])

• async_infer(inputs=None)
  • Description: Starts asynchronous inference of the infer request and fill outputs array
    • Parameters:
      • inputs - A dictionary that maps input layer names to numpy.ndarray objects of proper shape with input data for the layer
  • Return value: None
  • Usage example:

    >>> exec_net = plugin.load(network=net, num_requests=2)
    >>> exec_net.requests[0].async_infer({input_blob: image})
    >>> exec_net.requests[0].wait()
    >>> res = exec_net.requests[0].outputs['prob']
    >>> np.flip(np.sort(np.squeeze(res)),0)
    array([4.85416055e-01, 1.70385033e-01, 1.21873841e-01, 1.18894853e-01,
           5.45198545e-02, 2.44456064e-02, 5.41366823e-03, 3.42589128e-03,
           2.26027006e-03, 2.12283316e-03 ...])

• wait(timeout=-1)
  • Description: Waits for the result to become available. Blocks until specified timeout elapses or the result becomes available, whichever comes first.

    NOTE: There are special values of the timeout parameter:

    • 0 - Immediately returns the inference status. It does not block or interrupt execution. To find statuses meaning, please refer to InferenceEngine::StatusCode in Inference Engine C++ documentation
    • -1 - Waits until inference result becomes available (default value)
  • Parameters:
    • timeout - Time to wait in milliseconds or special (0, -1) cases described above. If not specified, timeout value is set to -1 by default.
  • Usage example: See async_infer() method of the the InferRequest class.
• get_perf_counts()
  • Description: Queries performance measures per layer to get feedback of what is the most time consuming layer.

    NOTE: Performance counters data and format depends on the plugin

  • Parameters: None
  • Usage example:

    >>> exec_net = plugin.load(network=net, num_requests=2)
    >>> exec_net.requests[0].infer({input_blob: image})
    >>> exec_net.requests[0].get_perf_counts()
    {'Conv2D': {'exec_type': 'jit_avx2_1x1',
                'real_time': 154,
                'cpu_time': 154,
                'status': 'EXECUTED',
                'layer_type': 'Convolution'},
     'Relu6':  {'exec_type': 'undef',
                'real_time': 0,
                'cpu_time': 0,
                'status': 'NOT_RUN',
                'layer_type': 'Clamp'}
    ...
    }

• set_batch(size)
  • Description: Sets new batch size for certain infer request when dynamic batching is enabled in executable network that created this request.

    NOTE: Support of dynamic batch size depends on the target plugin.

  • Parameters:
    • batch - New batch size to be used by all the following inference calls for this request
  • Usage example:

    >>> plugin.set_config({"DYN_BATCH_ENABLED": "YES"})
    >>> exec_net = plugin.load(network=net)
    >>> exec_net.requests[0].set_batch(inputs_count)

• set_completion_callback(py_callback, py_data = None)
  • Description: Sets a callback function that is called on success or failure of an asynchronous request
  • Parameters:
    • py_callback - Any defined or lambda function
    • py_data - Data that is passed to the callback function
  • Return value: None
  • Usage example:

callback = lambda status, py_data: print("Request with id {} finished with status {}".format(py_data, status))

net = IENetwork("./model.xml", "./model.bin")
ie = IECore()
exec_net = ie.load_network(net, "CPU", num_requests=4)


for id, req in enumerate(exec_net.requests):
    req.set_completion_callback(py_callback=callback, py_data=id)


for req in exec_net.requests:
    req.async_infer({"data": img})