class ov::IInferRequest¶

Overview¶

An internal API of inference request to be implemented by plugin. More…

#include <iinfer_request.hpp>

class IInferRequest
{
public:
    // methods

    virtual void infer() = 0;
    virtual std::vector<ov::ProfilingInfo> get_profiling_info() const = 0;
    virtual ov::Tensor get_tensor(const ov::Output<const ov::Node>& port) const = 0;

    virtual void set_tensor(
        const ov::Output<const ov::Node>& port,
        const ov::Tensor& tensor
        ) = 0;

    virtual std::vector<ov::Tensor> get_tensors(const ov::Output<const ov::Node>& port) const = 0;

    virtual void set_tensors(
        const ov::Output<const ov::Node>& port,
        const std::vector<ov::Tensor>& tensors
        ) = 0;

    virtual std::vector<std::shared_ptr<ov::IVariableState>> query_state() const = 0;
    virtual const std::shared_ptr<const ov::ICompiledModel>& get_compiled_model() const = 0;
    virtual const std::vector<ov::Output<const ov::Node>>& get_inputs() const = 0;
    virtual const std::vector<ov::Output<const ov::Node>>& get_outputs() const = 0;
};

// direct descendants

class IAsyncInferRequest;
class ISyncInferRequest;

Detailed Documentation¶

An internal API of inference request to be implemented by plugin.

Methods¶

virtual void infer() = 0

Infers specified input(s) in synchronous mode.

blocks all method of InferRequest while request is ongoing (running or waiting in queue)

virtual std::vector<ov::ProfilingInfo> get_profiling_info() const = 0

Queries performance measures per layer to identify the most time consuming operation.

Not all plugins provide meaningful data.

Returns:

Vector of profiling information for operations in a model.

virtual ov::Tensor get_tensor(const ov::Output<const ov::Node>& port) const = 0

Gets an input/output tensor for inference.

If the tensor with the specified port is not found, an exception is thrown.

Parameters:

port	Port of the tensor to get.

Returns:

Tensor for the port port.

virtual void set_tensor(
    const ov::Output<const ov::Node>& port,
    const ov::Tensor& tensor
    ) = 0

Sets an input/output tensor to infer.

Parameters:

port	Port of the input or output tensor.
tensor	Reference to a tensor. The element_type and shape of a tensor must match the model’s input/output element_type and size.

virtual std::vector<ov::Tensor> get_tensors(const ov::Output<const ov::Node>& port) const = 0

Gets a batch of tensors for input data to infer by input port. Model input must have batch dimension, and the number of tensors must match the batch size. The current version supports setting tensors to model inputs only. If port is associated with output (or any other non-input node), an exception is thrown.

Parameters:

port	Port of the input tensor.
tensors	Input tensors for batched infer request. The type of each tensor must match the model input element type and shape (except batch dimension). Total size of tensors must match the input size.

Returns:

vector of tensors

virtual void set_tensors(
    const ov::Output<const ov::Node>& port,
    const std::vector<ov::Tensor>& tensors
    ) = 0

Sets a batch of tensors for input data to infer by input port. Model input must have batch dimension, and the number of tensors must match the batch size. The current version supports setting tensors to model inputs only. If port is associated with output (or any other non-input node), an exception is thrown.

Parameters:

port	Port of the input tensor.
tensors	Input tensors for batched infer request. The type of each tensor must match the model input element type and shape (except batch dimension). Total size of tensors must match the input size.

virtual std::vector<std::shared_ptr<ov::IVariableState>> query_state() const = 0

Gets state control interface for the given infer request.

State control essential for recurrent models.

Returns:

Vector of Variable State objects.

virtual const std::shared_ptr<const ov::ICompiledModel>& get_compiled_model() const = 0

Gets pointer to compiled model (usually synchronous request holds the compiled model)

Returns:

Pointer to the compiled model

virtual const std::vector<ov::Output<const ov::Node>>& get_inputs() const = 0

Gets inputs for infer request.

Returns:

vector of input ports

virtual const std::vector<ov::Output<const ov::Node>>& get_outputs() const = 0

Gets outputs for infer request.

Returns:

vector of output ports