Infer Request

Overview

The definitions & operations about infer request. More…

// structs

struct ov_ProfilingInfo_t;
struct ov_callback_t;
struct ov_infer_request_t;
struct ov_profiling_info_list_t;

// global functions

ov_infer_request_set_tensor(
    ov_infer_request_t \* infer_request,
    const char \* tensor_name,
    const ov_tensor_t \* tensor
    );

ov_infer_request_set_tensor_by_port(
    ov_infer_request_t \* infer_request,
    const ov_output_port_t \* port,
    const ov_tensor_t \* tensor
    );

ov_infer_request_set_tensor_by_const_port(
    ov_infer_request_t \* infer_request,
    const ov_output_const_port_t \* port,
    const ov_tensor_t \* tensor
    );

ov_infer_request_set_input_tensor_by_index(
    ov_infer_request_t \* infer_request,
    const size_t idx,
    const ov_tensor_t \* tensor
    );

ov_infer_request_set_input_tensor(
    ov_infer_request_t \* infer_request,
    const ov_tensor_t \* tensor
    );

ov_infer_request_set_output_tensor_by_index(
    ov_infer_request_t \* infer_request,
    const size_t idx,
    const ov_tensor_t \* tensor
    );

ov_infer_request_set_output_tensor(
    ov_infer_request_t \* infer_request,
    const ov_tensor_t \* tensor
    );

ov_infer_request_get_tensor(
    const ov_infer_request_t \* infer_request,
    const char \* tensor_name,
    ov_tensor_t \*\* tensor
    );

ov_infer_request_get_tensor_by_const_port(
    const ov_infer_request_t \* infer_request,
    const ov_output_const_port_t \* port,
    ov_tensor_t \*\* tensor
    );

ov_infer_request_get_tensor_by_port(
    const ov_infer_request_t \* infer_request,
    const ov_output_port_t \* port,
    ov_tensor_t \*\* tensor
    );

ov_infer_request_get_input_tensor_by_index(
    const ov_infer_request_t \* infer_request,
    const size_t idx,
    ov_tensor_t \*\* tensor
    );

ov_infer_request_get_input_tensor(
    const ov_infer_request_t \* infer_request,
    ov_tensor_t \*\* tensor
    );

ov_infer_request_get_output_tensor_by_index(
    const ov_infer_request_t \* infer_request,
    const size_t idx,
    ov_tensor_t \*\* tensor
    );

ov_infer_request_get_output_tensor(
    const ov_infer_request_t \* infer_request,
    ov_tensor_t \*\* tensor
    );

ov_infer_request_infer(ov_infer_request_t \* infer_request);
ov_infer_request_cancel(ov_infer_request_t \* infer_request);
ov_infer_request_start_async(ov_infer_request_t \* infer_request);
ov_infer_request_wait(ov_infer_request_t \* infer_request);

ov_infer_request_wait_for(
    ov_infer_request_t \* infer_request,
    const int64_t timeout
    );

ov_infer_request_set_callback(
    ov_infer_request_t \* infer_request,
    const ov_callback_t \* callback
    );

ov_infer_request_free(ov_infer_request_t \* infer_request);

ov_infer_request_get_profiling_info(
    const ov_infer_request_t \* infer_request,
    ov_profiling_info_list_t \* profiling_infos
    );

ov_profiling_info_list_free(ov_profiling_info_list_t \* profiling_infos);

Detailed Documentation

The definitions & operations about infer request.

Global Functions

ov_infer_request_set_tensor(
    ov_infer_request_t \* infer_request,
    const char \* tensor_name,
    const ov_tensor_t \* tensor
    )

Set an input/output tensor to infer on by the name of tensor.

Parameters:

infer_request	A pointer to the ov_infer_request_t.
tensor_name	Name of the input or output tensor.
tensor	Reference to the tensor.

Returns:

Status code of the operation: OK(0) for success.

ov_infer_request_set_tensor_by_port(
    ov_infer_request_t \* infer_request,
    const ov_output_port_t \* port,
    const ov_tensor_t \* tensor
    )

Set an input/output tensor to infer request for the port.

Parameters:

infer_request	A pointer to the ov_infer_request_t.
port	Port of the input or output tensor, which can be got by calling ov_model_t/ov_compiled_model_t interface.
tensor	Reference to the tensor.

Returns:

Status code of the operation: OK(0) for success.

ov_infer_request_set_tensor_by_const_port(
    ov_infer_request_t \* infer_request,
    const ov_output_const_port_t \* port,
    const ov_tensor_t \* tensor
    )

Set an input/output tensor to infer request for the port.

Parameters:

infer_request	A pointer to the ov_infer_request_t.
port	Const port of the input or output tensor, which can be got by call interface from ov_model_t/ov_compiled_model_t.
tensor	Reference to the tensor.

Returns:

Status code of the operation: OK(0) for success.

ov_infer_request_set_input_tensor_by_index(
    ov_infer_request_t \* infer_request,
    const size_t idx,
    const ov_tensor_t \* tensor
    )

Set an input tensor to infer on by the index of tensor.

Parameters:

infer_request	A pointer to the ov_infer_request_t.
idx	Index of the input port. If idx is greater than the number of model inputs, an error will return.
tensor	Reference to the tensor.

Returns:

Status code of the operation: OK(0) for success.

ov_infer_request_set_input_tensor(
    ov_infer_request_t \* infer_request,
    const ov_tensor_t \* tensor
    )

Set an input tensor for the model with single input to infer on.

If model has several inputs, an error will return.

Parameters:

infer_request	A pointer to the ov_infer_request_t.
tensor	Reference to the tensor.

Returns:

Status code of the operation: OK(0) for success.

ov_infer_request_set_output_tensor_by_index(
    ov_infer_request_t \* infer_request,
    const size_t idx,
    const ov_tensor_t \* tensor
    )

Set an output tensor to infer by the index of output tensor.

Index of the output preserved accross ov_model_t, ov_compiled_model_t.

Parameters:

infer_request	A pointer to the ov_infer_request_t.
idx	Index of the output tensor.
tensor	Reference to the tensor.

Returns:

Status code of the operation: OK(0) for success.

ov_infer_request_set_output_tensor(
    ov_infer_request_t \* infer_request,
    const ov_tensor_t \* tensor
    )

Set an output tensor to infer models with single output.

Parameters:

infer_request	A pointer to the ov_infer_request_t.
tensor	Reference to the tensor.

Returns:

Status code of the operation: OK(0) for success.

ov_infer_request_get_tensor(
    const ov_infer_request_t \* infer_request,
    const char \* tensor_name,
    ov_tensor_t \*\* tensor
    )

Get an input/output tensor by the name of tensor.

If model has several outputs, an error will return.

Parameters:

infer_request	A pointer to the ov_infer_request_t.
tensor_name	Name of the input or output tensor to get.
tensor	Reference to the tensor.

Returns:

Status code of the operation: OK(0) for success.

ov_infer_request_get_tensor_by_const_port(
    const ov_infer_request_t \* infer_request,
    const ov_output_const_port_t \* port,
    ov_tensor_t \*\* tensor
    )

Get an input/output tensor by const port.

Parameters:

infer_request	A pointer to the ov_infer_request_t.
port	Port of the tensor to get. port is not found, an error will return.
tensor	Reference to the tensor.

Returns:

Status code of the operation: OK(0) for success.

ov_infer_request_get_tensor_by_port(
    const ov_infer_request_t \* infer_request,
    const ov_output_port_t \* port,
    ov_tensor_t \*\* tensor
    )

Get an input/output tensor by port.

Parameters:

infer_request	A pointer to the ov_infer_request_t.
port	Port of the tensor to get. port is not found, an error will return.
tensor	Reference to the tensor.

Returns:

Status code of the operation: OK(0) for success.

ov_infer_request_get_input_tensor_by_index(
    const ov_infer_request_t \* infer_request,
    const size_t idx,
    ov_tensor_t \*\* tensor
    )

Get an input tensor by the index of input tensor.

Parameters:

infer_request	A pointer to the ov_infer_request_t.
idx	Index of the tensor to get. idx. If the tensor with the specified idx is not found, an error will return.
tensor	Reference to the tensor.

Returns:

Status code of the operation: OK(0) for success.

ov_infer_request_get_input_tensor(
    const ov_infer_request_t \* infer_request,
    ov_tensor_t \*\* tensor
    )

Get an input tensor from the model with only one input tensor.

Parameters:

infer_request	A pointer to the ov_infer_request_t.
tensor	Reference to the tensor.

Returns:

Status code of the operation: OK(0) for success.

ov_infer_request_get_output_tensor_by_index(
    const ov_infer_request_t \* infer_request,
    const size_t idx,
    ov_tensor_t \*\* tensor
    )

Get an output tensor by the index of output tensor.

Parameters:

infer_request	A pointer to the ov_infer_request_t.
idx	Index of the tensor to get. idx. If the tensor with the specified idx is not found, an error will return.
tensor	Reference to the tensor.

Returns:

Status code of the operation: OK(0) for success.

ov_infer_request_get_output_tensor(
    const ov_infer_request_t \* infer_request,
    ov_tensor_t \*\* tensor
    )

Get an output tensor from the model with only one output tensor.

If model has several outputs, an error will return.

Parameters:

infer_request	A pointer to the ov_infer_request_t.
tensor	Reference to the tensor.

Returns:

Status code of the operation: OK(0) for success.

ov_infer_request_infer(ov_infer_request_t \* infer_request)

Infer specified input(s) in synchronous mode.

Parameters:

infer_request

A pointer to the ov_infer_request_t.

Returns:

Status code of the operation: OK(0) for success.

ov_infer_request_cancel(ov_infer_request_t \* infer_request)

Cancel inference request.

Parameters:

infer_request

A pointer to the ov_infer_request_t.

Returns:

Status code of the operation: OK(0) for success.

ov_infer_request_start_async(ov_infer_request_t \* infer_request)

Start inference of specified input(s) in asynchronous mode.

Parameters:

infer_request

A pointer to the ov_infer_request_t.

Returns:

Status code of the operation: OK(0) for success.

ov_infer_request_wait(ov_infer_request_t \* infer_request)

Wait for the result to become available.

Parameters:

infer_request

A pointer to the ov_infer_request_t.

Returns:

Status code of the operation: OK(0) for success.

ov_infer_request_wait_for(
    ov_infer_request_t \* infer_request,
    const int64_t timeout
    )

Waits for the result to become available. Blocks until the specified timeout has elapsed or the result becomes available, whichever comes first.

Parameters:

infer_request	A pointer to the ov_infer_request_t.
timeout	Maximum duration, in milliseconds, to block for.

Returns:

Status code of the operation: OK(0) for success.

ov_infer_request_set_callback(
    ov_infer_request_t \* infer_request,
    const ov_callback_t \* callback
    )

Set callback function, which will be called when inference is done.

Parameters:

infer_request	A pointer to the ov_infer_request_t.
callback	A function to be called.

Returns:

Status code of the operation: OK(0) for success.

ov_infer_request_free(ov_infer_request_t \* infer_request)

Release the memory allocated by ov_infer_request_t.

Parameters:

infer_request

A pointer to the ov_infer_request_t to free memory.

ov_infer_request_get_profiling_info(
    const ov_infer_request_t \* infer_request,
    ov_profiling_info_list_t \* profiling_infos
    )

Query performance measures per layer to identify the most time consuming operation.

Parameters:

infer_request	A pointer to the ov_infer_request_t.
profiling_infos	Vector of profiling information for operations in a model.

Returns:

Status code of the operation: OK(0) for success.

ov_profiling_info_list_free(ov_profiling_info_list_t \* profiling_infos)

Release the memory allocated by ov_profiling_info_list_t.

Parameters:

profiling_infos

A pointer to the ov_profiling_info_list_t to free memory.