Group Asynchronous Inference Request base classes

group ov_dev_api_async_infer_request_api

A set of base and helper classes to implement asynchronous inference request class.

class IAsyncInferRequest : public ov::IInferRequest
#include <iasync_infer_request.hpp>

Base class with default implementation of asynchronous multi staged inference request. To customize pipeline stages derived class should change the content of IAsyncInferRequest::m_pipeline member container. It consists of pairs of tasks and executors which will run the task. The class is recommended to be used by plugins as a base class for asynchronous inference request implementation.

Example

Here is an example of asynchronous inference request implementation for some accelerator device. It uses 5 different executors to run different stages of a synchronous inference request.

Note

To synchronize derived context with stages derived class should call IAsyncInferRequest::stop_and_wait() function in destructor.

Public Functions

virtual void start_async()

Start inference of specified input(s) in asynchronous mode.

Note

The method returns immediately. Inference starts also immediately.

virtual void wait()

Waits for the result to become available.

virtual bool wait_for(const std::chrono::milliseconds &timeout)

Waits for the result to become available. Blocks until specified timeout has elapsed or the result becomes available, whichever comes first.

Parameters

timeout – - maximum duration in milliseconds to block for

Returns

A true if results are ready.

virtual void cancel()

Cancel current inference request execution.

virtual void set_callback(std::function<void(std::exception_ptr)> callback)

Set callback function which will be called on success or failure of asynchronous request.

Parameters

callback – - function to be called with the following description:

virtual void infer() override

Infers specified input(s) in synchronous mode.

Note

blocks all method of InferRequest while request is ongoing (running or waiting in queue)

virtual std::vector<ov::ProfilingInfo> get_profiling_info() const override

Queries performance measures per layer to identify the most time consuming operation.

Note

Not all plugins provide meaningful data.

Returns

Vector of profiling information for operations in a model.

virtual ov::SoPtr<ov::ITensor> get_tensor(const ov::Output<const ov::Node> &port) const override

Gets an input/output tensor for inference.

Note

If the tensor with the specified port is not found, an exception is thrown.

Parameters

port – Port of the tensor to get.

Returns

Tensor for the port port.

virtual void set_tensor(const ov::Output<const ov::Node> &port, const ov::SoPtr<ov::ITensor> &tensor) override

Sets an input/output tensor to infer.

Parameters

  • port – Port of the input or output tensor.

  • tensor – Reference to a tensor. The element_type and shape of a tensor must match the model’s input/output element_type and size.

virtual std::vector<ov::SoPtr<ov::ITensor>> get_tensors(const ov::Output<const ov::Node> &port) const override

Gets a batch of tensors for input data to infer by input port. Model input must have batch dimension, and the number of tensors must match the batch size. The current version supports setting tensors to model inputs only. If port is associated with output (or any other non-input node), an exception is thrown.

Parameters

  • port – Port of the input tensor.

  • tensors – Input tensors for batched infer request. The type of each tensor must match the model input element type and shape (except batch dimension). Total size of tensors must match the input size.

Returns

vector of tensors

virtual void set_tensors(const ov::Output<const ov::Node> &port, const std::vector<ov::SoPtr<ov::ITensor>> &tensors) override

Sets a batch of tensors for input data to infer by input port. Model input must have batch dimension, and the number of tensors must match the batch size. The current version supports setting tensors to model inputs only. If port is associated with output (or any other non-input node), an exception is thrown.

Parameters

  • port – Port of the input tensor.

  • tensors – Input tensors for batched infer request. The type of each tensor must match the model input element type and shape (except batch dimension). Total size of tensors must match the input size.

virtual std::vector<ov::SoPtr<ov::IVariableState>> query_state() const override

Gets state control interface for the given infer request.

State control essential for recurrent models.

Returns

Vector of Variable State objects.

virtual const std::shared_ptr<const ov::ICompiledModel> &get_compiled_model() const override

Gets pointer to compiled model (usually synchronous request holds the compiled model)

Returns

Pointer to the compiled model

virtual const std::vector<ov::Output<const ov::Node>> &get_inputs() const override

Gets inputs for infer request.

Returns

vector of input ports

virtual const std::vector<ov::Output<const ov::Node>> &get_outputs() const override

Gets outputs for infer request.

Returns

vector of output ports