ie_api.InferRequest Class Reference

This class provides an interface to infer requests of ExecutableNetwork and serves to handle infer requests execution and to set and get output data. More...

Public Member Functions
def	__init__ (self)
	There is no explicit class constructor. More...
def	set_completion_callback (self, py_callback, py_data=None)
	Description: Sets a callback function that is called on success or failure of an asynchronous request. More...
def	input_blobs (self)
	Dictionary that maps input layer names to corresponding Blobs.
def	output_blobs (self)
	Dictionary that maps output layer names to corresponding Blobs.
def	preprocess_info (self)
	Dictionary that maps input layer names to corresponding preprocessing information.
def	query_state (self)
	Gets state control interface for given infer request State control essential for recurrent networks. More...
def	set_blob (self, blob_name, blob, preprocess_info)
	Sets user defined Blob for the infer request. More...
def	infer (self, inputs=None)
	Starts synchronous inference of the infer request and fill outputs array. More...
def	async_infer (self, inputs=None)
	Starts asynchronous inference of the infer request and fill outputs array. More...
def	wait (self, timeout=None)
	Waits for the result to become available. More...
def	get_perf_counts (self)
	Queries performance measures per layer to get feedback of what is the most time consuming layer. More...
def	inputs (self)
	A dictionary that maps input layer names to numpy.ndarray objects of proper shape with input data for the layer.
def	outputs (self)
	A dictionary that maps output layer names to numpy.ndarray objects with output data of the layer.
def	latency (self)
	Current infer request inference time in milliseconds.
def	set_batch (self, size)
	Sets new batch size for certain infer request when dynamic batching is enabled in executable network that created this request. More...

Data Fields
	input_blobs
	Dictionary that maps input layer names to corresponding Blobs.
	output_blobs
	Dictionary that maps output layer names to corresponding Blobs.
	preprocess_info
	Dictionary that maps input layer names to corresponding preprocessing information.
	inputs
	A dictionary that maps input layer names to numpy.ndarray objects of proper shape with input data for the layer.
	outputs
	A dictionary that maps output layer names to numpy.ndarray objects with output data of the layer.
	latency
	Current infer request inference time in milliseconds.

Detailed Description

This class provides an interface to infer requests of ExecutableNetwork and serves to handle infer requests execution and to set and get output data.

Constructor & Destructor Documentation

__init__()

def ie_api.InferRequest.__init__

(

self

)

There is no explicit class constructor.

To make a valid InferRequest instance, use load_network() method of the IECore class with specified number of requests to get ExecutableNetwork instance which stores infer requests.

Member Function Documentation

async_infer()

def ie_api.InferRequest.async_infer	(		self,
			inputs = None
	)

Starts asynchronous inference of the infer request and fill outputs array.

Parameters

inputs

A dictionary that maps input layer names to numpy.ndarray objects of proper shape with input data for the layer

Returns

: None

Usage example:

exec_net = ie_core.load_network(network=net, device_name="CPU", num_requests=2)
exec_net.requests[0].async_infer({input_blob: image})
request_status = exec_net.requests[0].wait()
res = exec_net.requests[0].output_blobs['prob']

get_perf_counts()

def ie_api.InferRequest.get_perf_counts

(

self

)

Queries performance measures per layer to get feedback of what is the most time consuming layer.

Note

Performance counters data and format depends on the plugin

Returns

Dictionary containing per-layer execution information.

Usage example:

exec_net = ie_core.load_network(network=net, device_name="CPU", num_requests=2)
exec_net.requests[0].infer({input_blob: image})
exec_net.requests[0].get_perf_counts()
{'Conv2D': {'exec_type': 'jit_avx2_1x1',
            'real_time': 154,
            'cpu_time': 154,
            'status': 'EXECUTED',
            'layer_type': 'Convolution'},
 'Relu6':  {'exec_type': 'undef',
            'real_time': 0,
            'cpu_time': 0,
            'status': 'NOT_RUN',
            'layer_type': 'Clamp'}
 ...
}

infer()

def ie_api.InferRequest.infer	(		self,
			inputs = None
	)

Starts synchronous inference of the infer request and fill outputs array.

Parameters

inputs

A dictionary that maps input layer names to numpy.ndarray objects of proper shape with input data for the layer

Returns

None

Usage example:

exec_net = ie_core.load_network(network=net, device_name="CPU", num_requests=2)
exec_net.requests[0].infer({input_blob: image})
res = exec_net.requests[0].output_blobs['prob']
np.flip(np.sort(np.squeeze(res)),0)
array([4.85416055e-01, 1.70385033e-01, 1.21873841e-01, 1.18894853e-01,
       5.45198545e-02, 2.44456064e-02, 5.41366823e-03, 3.42589128e-03,
       2.26027006e-03, 2.12283316e-03 ...])

query_state()

def ie_api.InferRequest.query_state

(

self

)

Gets state control interface for given infer request State control essential for recurrent networks.

Returns

A vector of Memory State objects

set_batch()

def ie_api.InferRequest.set_batch	(		self,
			size
	)

Sets new batch size for certain infer request when dynamic batching is enabled in executable network that created this request.

Note

Support of dynamic batch size depends on the target plugin.

Parameters

size	New batch size to be used by all the following inference calls for this request

Returns

None

Usage example:

ie = IECore()
net = ie.read_network(model=path_to_xml_file, weights=path_to_bin_file)
# Set max batch size
net.batch = 10
ie.set_config(config={"DYN_BATCH_ENABLED": "YES"}, device_name=device)
exec_net = ie.load_network(network=net, device_name=device)
# Set batch size for certain network.
# NOTE: Input data shape will not be changed, but will be used partially in inference which increases performance
exec_net.requests[0].set_batch(2)

set_blob()

def ie_api.InferRequest.set_blob	(		self,
			blob_name,
			blob,
			preprocess_info
	)

Sets user defined Blob for the infer request.

Parameters

blob_name	A name of input blob
blob	Blob object to set for the infer request
preprocess_info	PreProcessInfo object to set for the infer request.

Returns

None

Usage example:

ie = IECore()
net = IENetwork("./model.xml", "./model.bin")
exec_net = ie.load_network(net, "CPU", num_requests=2)
td = TensorDesc("FP32", (1, 3, 224, 224), "NCHW")
blob_data = np.ones(shape=(1, 3, 224, 224), dtype=np.float32)
blob = Blob(td, blob_data)
exec_net.requests[0].set_blob(blob_name="input_blob_name", blob=blob),

set_completion_callback()

def ie_api.InferRequest.set_completion_callback	(		self,
			py_callback,
			py_data = None
	)

Description: Sets a callback function that is called on success or failure of an asynchronous request.

Parameters

py_callback	- Any defined or lambda function
py_data	- Data that is passed to the callback function

Returns

None

Usage example:

callback = lambda status, py_data: print(f"Request with id {py_data} finished with status {status}")
ie = IECore()
net = ie.read_network(model="./model.xml", weights="./model.bin")
exec_net = ie.load_network(net, "CPU", num_requests=4)
for id, req in enumerate(exec_net.requests):
    req.set_completion_callback(py_callback=callback, py_data=id)
 
for req in exec_net.requests:
    req.async_infer({"data": img})

wait()

def ie_api.InferRequest.wait	(		self,
			timeout = None
	)

Waits for the result to become available.

Blocks until specified timeout elapses or the result becomes available, whichever comes first.

Note

There are special values of the timeout parameter:

• 0 - Immediately returns the inference status. It does not block or interrupt execution. To find statuses meaning, please refer to InferenceEngine::StatusCode in Inference Engine C++ documentation
• -1 - Waits until inference result becomes available (default value)

Parameters

timeout

Time to wait in milliseconds or special (0, -1) cases described above. If not specified, timeout value is set to -1 by default.

Returns

Request status code.

Usage example: See async_infer() method of the the InferRequest class.

---

The documentation for this class was generated from the following file:

• ie_api.pyx