interface InferenceEngine::IInferRequestInternal¶

Overview¶

An internal API of synchronous inference request to be implemented by plugin, which is used in InferRequestBase forwarding mechanism. More…

#include <ie_iinfer_request_internal.hpp>

template IInferRequestInternal: public std::enable_shared_from_this< IInferRequestInternal >
{
    // typedefs

    typedef std::shared_ptr<IInferRequestInternal> Ptr;
    typedef std::function<void(std::exception_ptr)> Callback;

    // construction

    IInferRequestInternal();

    IInferRequestInternal(
        const InputsDataMap& networkInputs,
        const OutputsDataMap& networkOutputs
        );

    // methods

    virtual void Infer();
    virtual void InferImpl();
    virtual void Cancel();
    virtual std::map<std::string, InferenceEngineProfileInfo> GetPerformanceCounts() const;
    virtual void SetBlob(const std::string& name, const Blob::Ptr& data);
    virtual Blob::Ptr GetBlob(const std::string& name);

    virtual void SetBlob(
        const std::string& name,
        const Blob::Ptr& data,
        const PreProcessInfo& info
        );

    virtual const PreProcessInfo& GetPreProcess(const std::string& name) const;
    virtual void SetBatch(int batch);
    virtual std::vector<std::shared_ptr<IVariableStateInternal>> QueryState();
    virtual void StartAsync();
    virtual void StartAsyncImpl();
    virtual StatusCode Wait(int64_t millis_timeout);
    virtual void SetCallback(Callback callback);

    void checkBlob(
        const Blob::Ptr& blob,
        const std::string& name,
        bool isInput,
        const SizeVector& refDims = {}
        ) const;

    virtual void checkBlobs();
    void setPointerToExecutableNetworkInternal(const std::shared_ptr<IExecutableNetworkInternal>& exeNetwork);
    void* GetUserData();
    void SetUserData(void* userData);

protected:
};

// direct descendants

class AsyncInferRequestThreadSafeDefault;

Detailed Documentation¶

An internal API of synchronous inference request to be implemented by plugin, which is used in InferRequestBase forwarding mechanism.

Typedefs¶

typedef std::shared_ptr<IInferRequestInternal> Ptr

A shared pointer to a IInferRequestInternal interface.

typedef std::function<void(std::exception_ptr)> Callback

Alias for callback type.

Construction¶

IInferRequestInternal(
    const InputsDataMap& networkInputs,
    const OutputsDataMap& networkOutputs
    )

Constructs a new instance.

Parameters:

networkInputs	The network inputs info
networkOutputs	The network outputs data

Methods¶

virtual void Infer()

Infers specified input(s) in synchronous mode.

blocks all method of InferRequest while request is ongoing (running or waiting in queue)

virtual void InferImpl()

The minimal infer function to be implemented by plugins. It infers specified input(s) in synchronous mode.

This method is used in IInferRequestInternal::Infer, which calls the common code first and after uses this plugin dependent implementation.
Blocks all method of InferRequest while request is ongoing (running or waiting in queue)

virtual void Cancel()

Cancel current inference request execution.

virtual std::map<std::string, InferenceEngineProfileInfo> GetPerformanceCounts() const

Queries performance measures per layer to get feedback of what is the most time consuming layer. Note: not all plugins may provide meaningful data.

Returns:

a map of layer names to profiling information for that layer.

virtual void SetBlob(const std::string& name, const Blob::Ptr& data)

Set input/output data to infer.

Memory allocation doesn’t happen

Parameters:

name	a name of input or output blob.
data	a reference to input or output blob. The type of Blob must correspond to the network input precision and size.

virtual Blob::Ptr GetBlob(const std::string& name)

Get input/output data to infer.

Memory allocation doesn’t happen

Parameters:

name	a name of input or output blob.
data	a reference to input or output blob. The type of Blob must correspond to the network input precision and size.

virtual void SetBlob(
    const std::string& name,
    const Blob::Ptr& data,
    const PreProcessInfo& info
    )

Sets pre-process for input data.

Parameters:

name	Name of input blob.
data	a reference to input or output blob. The type of Blob must correspond to the network input precision and size.
info	Preprocess info for blob.

virtual const PreProcessInfo& GetPreProcess(const std::string& name) const

Gets pre-process for input data.

Parameters:

name	Name of input blob.
info	pointer to a pointer to PreProcessInfo structure

virtual void SetBatch(int batch)

Sets new batch size when dynamic batching is enabled in executable network that created this request.

Parameters:

batch

new batch size to be used by all the following inference calls for this request.

virtual std::vector<std::shared_ptr<IVariableStateInternal>> QueryState()

Queries memory states.

Returns:

Returns memory states

virtual void StartAsync()

Start inference of specified input(s) in asynchronous mode.

The method returns immediately. Inference starts also immediately.

virtual void StartAsyncImpl()

The minimal asynchronous inference function to be implemented by plugins. It starts inference of specified input(s) in asynchronous mode.

The methos is used in AsyncInferRequestInternal::StartAsync which performs common steps first and calls plugin dependent implementation of this method after.
It returns immediately. Inference starts also immediately.

virtual StatusCode Wait(int64_t millis_timeout)

Waits for the result to become available. Blocks until specified millis_timeout has elapsed or the result becomes available, whichever comes first.

There are special cases when millis_timeout is equal some value of WaitMode enum:

STATUS_ONLY - immediately returns request status (InferRequest::StatusCode). It doesn’t block or interrupt current thread.
RESULT_READY - waits until inference result becomes available

Parameters:

millis_timeout

maximum duration in milliseconds to block for

Returns:

A status code

virtual void SetCallback(Callback callback)

Set callback function which will be called on success or failure of asynchronous request.

Parameters:

callback

function to be called with the following description:

void checkBlob(
    const Blob::Ptr& blob,
    const std::string& name,
    bool isInput,
    const SizeVector& refDims = {}
    ) const

Check that blob is valid. Throws an exception if it’s not.

Parameters:

blob	The blob to check
name	The name of input or output depending of if the `blob` is input or output
isInput	Indicates if `is` input
refDims	The reference dims, empty if not specified

virtual void checkBlobs()

Check that all of the blobs is valid. Throws an exception if it’s not.

void setPointerToExecutableNetworkInternal(const std::shared_ptr<IExecutableNetworkInternal>& exeNetwork)

Sets the pointer to executable network internal.

Needed to correctly handle ownership between objects.

Parameters:

exeNetwork

The executable network

void* GetUserData()

Gets the pointer to userData.

Returns:

Pointer to user data

void SetUserData(void* userData)

Sets the pointer to userData.

Parameters:

Pointer

to user data

Prev Next