openvino.runtime.AsyncInferQueue

class openvino.runtime.AsyncInferQueue

Bases: AsyncInferQueue

AsyncInferQueue with a pool of asynchronous requests.

AsyncInferQueue represents a helper that creates a pool of asynchronous InferRequests and provides synchronization functions to control flow of a simple pipeline.

__init__(self: openvino._pyopenvino.AsyncInferQueue, model: openvino._pyopenvino.CompiledModel, jobs: int = 0) → None

Creates AsyncInferQueue.

Parameters:

• model (openvino.runtime.CompiledModel) – Model to be used to create InferRequests in a pool.

• jobs – Number of InferRequests objects in a pool. If 0, jobs number

will be set automatically to the optimal number. Default: 0 :type jobs: int :rtype: openvino.runtime.AsyncInferQueue

Methods

__delattr__(name, /)	Implement delattr(self, name).
__dir__()	Default dir() implementation.
__eq__(value, /)	Return self==value.
__format__(format_spec, /)	Default object formatter.
__ge__(value, /)	Return self>=value.
__getattribute__(name, /)	Return getattr(self, name).
__getitem__(i)	Gets InferRequest from the pool with given i id.
__gt__(value, /)	Return self>value.
__hash__()	Return hash(self).
__init__(self, model[, jobs])	Creates AsyncInferQueue.
__init_subclass__	This method is called when a class is subclassed.
__iter__()	Allows to iterate over AsyncInferQueue.
__le__(value, /)	Return self<=value.
__len__(self)	Number of InferRequests in the pool.
__lt__(value, /)	Return self<value.
__ne__(value, /)	Return self!=value.
__new__(**kwargs)
__reduce__()	Helper for pickle.
__reduce_ex__(protocol, /)	Helper for pickle.
__repr__(self)
__setattr__(name, value, /)	Implement setattr(self, name, value).
__sizeof__()	Size of object in memory, in bytes.
__str__()	Return str(self).
__subclasshook__	Abstract classes can override this to customize issubclass().
get_idle_request_id(self)	Returns next free id of InferRequest from queue's pool.
is_ready(self)	One of 'flow control' functions.
set_callback(self, arg0)	Sets unified callback on all InferRequests from queue's pool.
start_async([inputs, userdata, share_inputs])	Run asynchronous inference using the next available InferRequest from the pool.
wait_all(self)	One of 'flow control' functions.

Attributes

userdata

return: List of all passed userdata. List is filled with None if the data wasn't passed yet.

__class__

alias of pybind11_type

__delattr__(name, /)

Implement delattr(self, name).

__dir__()

Default dir() implementation.

__eq__(value, /)

Return self==value.

__format__(format_spec, /)

Default object formatter.

__ge__(value, /)

Return self>=value.

__getattribute__(name, /)

Return getattr(self, name).

__getitem__(i: int) → InferRequest

Gets InferRequest from the pool with given i id.

Resulting object is guaranteed to work with read-only methods like getting tensors. Any mutating methods (e.g. start_async, set_callback) of a request will put the parent AsyncInferQueue object in an invalid state.

Parameters:

i (int) – InferRequest id.

Returns:

InferRequests from the pool with given id.

Return type:

openvino.runtime.InferRequest

__gt__(value, /)

Return self>value.

__hash__()

Return hash(self).

__init__(self: openvino._pyopenvino.AsyncInferQueue, model: openvino._pyopenvino.CompiledModel, jobs: int = 0) → None

Creates AsyncInferQueue.

Parameters:

• model (openvino.runtime.CompiledModel) – Model to be used to create InferRequests in a pool.

• jobs – Number of InferRequests objects in a pool. If 0, jobs number

will be set automatically to the optimal number. Default: 0 :type jobs: int :rtype: openvino.runtime.AsyncInferQueue

__init_subclass__()

This method is called when a class is subclassed.

The default implementation does nothing. It may be overridden to extend subclasses.

__iter__() → Iterable[InferRequest]

Allows to iterate over AsyncInferQueue.

Resulting objects are guaranteed to work with read-only methods like getting tensors. Any mutating methods (e.g. start_async, set_callback) of a single request will put the parent AsyncInferQueue object in an invalid state.

Returns:

a generator that yields InferRequests.

Return type:

Iterable[openvino.runtime.InferRequest]

__le__(value, /)

Return self<=value.

__len__(self: openvino._pyopenvino.AsyncInferQueue) → int

Number of InferRequests in the pool.

Return type:

int

__lt__(value, /)

Return self<value.

__ne__(value, /)

Return self!=value.

__new__(**kwargs)

__reduce__()

Helper for pickle.

__reduce_ex__(protocol, /)

Helper for pickle.

__repr__(self: openvino._pyopenvino.AsyncInferQueue) → str

__setattr__(name, value, /)

Implement setattr(self, name, value).

__sizeof__()

Size of object in memory, in bytes.

__str__()

Return str(self).

__subclasshook__()

Abstract classes can override this to customize issubclass().

This is invoked early on by abc.ABCMeta.__subclasscheck__(). It should return True, False or NotImplemented. If it returns NotImplemented, the normal algorithm is used. Otherwise, it overrides the normal algorithm (and the outcome is cached).

get_idle_request_id(self: openvino._pyopenvino.AsyncInferQueue) → int

Returns next free id of InferRequest from queue’s pool. Function waits for any request to complete and then returns this request’s id.

GIL is released while running this function.

Return type:

int

is_ready(self: openvino._pyopenvino.AsyncInferQueue) → bool

One of ‘flow control’ functions. Returns True if any free request in the pool, otherwise False.

GIL is released while running this function.

Returns:

If there is at least one free InferRequest in a pool, returns True.

Return type:

bool

set_callback(self: openvino._pyopenvino.AsyncInferQueue, arg0: Callable) → None

Sets unified callback on all InferRequests from queue’s pool. Signature of such function should have two arguments, where first one is InferRequest object and second one is userdata connected to InferRequest from the AsyncInferQueue’s pool.

def f(request, userdata):
    result = request.output_tensors[0]
    print(result + userdata)

async_infer_queue.set_callback(f)

Parameters:

callback (function) – Any Python defined function that matches callback’s requirements.

start_async(inputs: Any | None = None, userdata: Any | None = None, share_inputs: bool = False) → None

Run asynchronous inference using the next available InferRequest from the pool.

The allowed types of keys in the inputs dictionary are:

1. int

2. str

3. openvino.runtime.ConstOutput

The allowed types of values in the inputs are:

1. numpy.ndarray and all the types that are castable to it, e.g. torch.Tensor

2. openvino.runtime.Tensor

Can be called with only one openvino.runtime.Tensor or numpy.ndarray, it will work only with one-input models. When model has more inputs, function throws error.

Parameters:

• inputs (Any, optional) – Data to be set on input tensors of the next available InferRequest.

• userdata (Any, optional) – Any data that will be passed to a callback.

• share_inputs (bool, optional) –

  Enables share_inputs mode. Controls memory usage on inference’s inputs.

  If set to False inputs the data dispatcher will safely copy data to existing Tensors (including up- or down-casting according to data type, resizing of the input Tensor). Keeps Tensor inputs “as-is”.

  If set to True the data dispatcher tries to provide “zero-copy” Tensors for every input in form of: * numpy.ndarray and all the types that are castable to it, e.g. torch.Tensor Data that is going to be copied: * numpy.ndarray which are not C contiguous and/or not writable (WRITEABLE flag is set to False) * inputs which data types are mismatched from Infer Request’s inputs * inputs that should be in BF16 data type * scalar inputs (i.e. np.float_/str/bytes/int/float) * lists of simple data types (i.e. str/bytes/int/float) Keeps Tensor inputs “as-is”.

  Note: Use with extra care, shared data can be modified during runtime! Note: Using share_inputs may result in extra memory overhead.

  Default value: False

property userdata

Returns:

List of all passed userdata. List is filled with None if the data wasn’t passed yet.

Return type:

List[Any]

wait_all(self: openvino._pyopenvino.AsyncInferQueue) → None

One of ‘flow control’ functions. Blocking call. Waits for all InferRequests in a pool to finish scheduled work.

GIL is released while running this function.