class InferenceEngine::AsyncInferRequestThreadSafeDefault

Overview

Base class with default implementation of asynchronous multi staged inference request. To customize pipeline stages derived class should change the content of AsyncInferRequestThreadSafeDefault::_pipeline member container. It consists of pairs of tasks and executors which will run the task. The class is recommended to be used by plugins as a base class for asynchronous inference request implementation. More…

#include <ie_infer_async_request_thread_safe_default.hpp>

class AsyncInferRequestThreadSafeDefault: public IInferRequestInternal
{
public:
    // typedefs

    typedef std::shared_ptr<AsyncInferRequestThreadSafeDefault> Ptr;

    // structs

    struct DisableCallbackGuard;
    struct ImmediateStreamsExecutor;

    // construction

    AsyncInferRequestThreadSafeDefault(
        const IInferRequestInternal::Ptr& request,
        const ITaskExecutor::Ptr& taskExecutor,
        const ITaskExecutor::Ptr& callbackExecutor
        );

    // methods

    StatusCode Wait(int64_t millis_timeout);
    void StartAsync();
    void Infer();
    std::map<std::string, InferenceEngineProfileInfo> GetPerformanceCounts() const;
    void SetBlob(const std::string& name, const Blob::Ptr& data);
    void SetBlobs(const std::string& name, const std::vector<Blob::Ptr>& blobs);
    BatchedBlob::Ptr GetBlobs(const std::string& name);
    Blob::Ptr GetBlob(const std::string& name);
    const PreProcessInfo& GetPreProcess(const std::string& name) const;
    void SetCallback(Callback callback);
    std::vector<std::shared_ptr<InferenceEngine::IVariableStateInternal>> QueryState();
    void ThrowIfCanceled() const;
    void Cancel();

    void setModelInputsOutputs(
        const std::vector<std::shared_ptr<const ov::Node>>& inputs,
        const std::vector<std::shared_ptr<const ov::Node>>& outputs
        );
};

Detailed Documentation

Base class with default implementation of asynchronous multi staged inference request. To customize pipeline stages derived class should change the content of AsyncInferRequestThreadSafeDefault::_pipeline member container. It consists of pairs of tasks and executors which will run the task. The class is recommended to be used by plugins as a base class for asynchronous inference request implementation.

To synchronize derived context with stages derived class should call AsyncInferRequestThreadSafeDefault::StopAndWait() function in destructor.

Here is an example of asynchronous inference request implementation for some accelerator device. It uses 5 different executors to run different stages of a synchronous inference request.

Typedefs

typedef std::shared_ptr<AsyncInferRequestThreadSafeDefault> Ptr

A shared pointer to AsyncInferRequestThreadSafeDefault.

Construction

AsyncInferRequestThreadSafeDefault(
    const IInferRequestInternal::Ptr& request,
    const ITaskExecutor::Ptr& taskExecutor,
    const ITaskExecutor::Ptr& callbackExecutor
    )

Wraps a IInferRequestInternal::Ptr implementation and constructs a AsyncInferRequestThreadSafeDefault::_pipeline where taskExecutor is used to run IInferRequestInternal::Infer asynchronously.

Parameters:

request	The synchronous request
taskExecutor	The task executor
callbackExecutor	The callback executor

Methods

StatusCode Wait(int64_t millis_timeout)

Waits for completion of all pipeline stages If the pipeline raises an exception it will be rethrown here.

Parameters:

millis_timeout

A timeout is ms to wait or special enum value of InferRequest::WaitMode

Returns:

A status code