Base class with default implementation of asynchronous multi staged inference request. To customize pipeline stages derived class should change the content of AsyncInferRequestThreadSafeDefault::_pipeline member container. It consists of pairs of tasks and executors which will run the task. The class is recommended to be used by plugins as a base class for asynchronous inference request implementation. More...

#include <ie_infer_async_request_thread_safe_default.hpp>

Inheritance diagram for InferenceEngine::AsyncInferRequestThreadSafeDefault:

Public Types
using	Ptr = std::shared_ptr< AsyncInferRequestThreadSafeDefault >
	A shared pointer to AsyncInferRequestThreadSafeDefault.

Public Types inherited from InferenceEngine::IInferRequestInternal
using	Ptr = std::shared_ptr< IInferRequestInternal >
	A shared pointer to a IInferRequestInternal interface.

using	Callback = std::function< void(std::exception_ptr)>
	Alias for callback type.

Public Member Functions
	AsyncInferRequestThreadSafeDefault (const IInferRequestInternal::Ptr &request, const ITaskExecutor::Ptr &taskExecutor, const ITaskExecutor::Ptr &callbackExecutor)
	Wraps a IInferRequestInternal::Ptr implementation and constructs a AsyncInferRequestThreadSafeDefault::_pipeline where `taskExecutor` is used to run IInferRequestInternal::Infer asynchronously. More...

	~AsyncInferRequestThreadSafeDefault ()
	Destroys the object, stops AsyncInferRequestThreadSafeDefault::_pipeline and waits for a finish.

StatusCode	Wait (int64_t millis_timeout) override
	Waits for completion of all pipeline stages If the pipeline raises an exception it will be rethrown here. More...

void	StartAsync () override
	Start inference of specified input(s) in asynchronous mode. More...

void	Infer () override
	Infers specified input(s) in synchronous mode. More...

std::map< std::string, InferenceEngineProfileInfo >	GetPerformanceCounts () const override
	Queries performance measures per layer to get feedback of what is the most time consuming layer. Note: not all plugins may provide meaningful data. More...

void	SetBlob (const std::string &name, const Blob::Ptr &data) override
	Set input/output data to infer. More...

void	SetBlob (const std::string &name, const Blob::Ptr &data, const PreProcessInfo &info) override
	Sets pre-process for input data. More...

Blob::Ptr	GetBlob (const std::string &name) override
	Get input/output data to infer. More...

const PreProcessInfo &	GetPreProcess (const std::string &name) const override
	Gets pre-process for input data. More...

void	SetBatch (int batch) override
	Sets new batch size when dynamic batching is enabled in executable network that created this request. More...

void	SetCallback (Callback callback) override
	Set callback function which will be called on success or failure of asynchronous request. More...

std::vector< std::shared_ptr< InferenceEngine::IVariableStateInternal > >	QueryState () override
	Queries memory states. More...

void	ThrowIfCanceled () const

void	Cancel () override
	Cancel current inference request execution.

Public Member Functions inherited from InferenceEngine::IInferRequestInternal
	IInferRequestInternal (const InputsDataMap &networkInputs, const OutputsDataMap &networkOutputs)
	Constructs a new instance. More...

virtual void	InferImpl ()
	The minimal infer function to be implemented by plugins. It infers specified input(s) in synchronous mode. More...

virtual void	StartAsyncImpl ()
	The minimal asynchronous inference function to be implemented by plugins. It starts inference of specified input(s) in asynchronous mode. More...

void	checkBlob (const Blob::Ptr &blob, const std::string &name, bool isInput, const SizeVector &refDims={}) const
	Check that `blob` is valid. Throws an exception if it's not. More...

virtual void	checkBlobs ()
	Check that all of the blobs is valid. Throws an exception if it's not.

void	setPointerToExecutableNetworkInternal (const std::shared_ptr< IExecutableNetworkInternal > &exeNetwork)
	Sets the pointer to executable network internal. More...

The method will be removed void *	GetUserData () noexcept
	Gets the pointer to userData. More...

The method will be removed void	SetUserData (void *userData) noexcept
	Sets the pointer to userData. More...

Protected Types
using	Stage = std::pair< ITaskExecutor::Ptr, Task >
	Each pipeline stage is a Task that is executed by specified ITaskExecutor implementation.

using	Pipeline = std::vector< Stage >
	Pipeline is vector of stages.

Protected Member Functions
void	CheckState () const
	Throws exception if inference request is busy or canceled.

void	RunFirstStage (const Pipeline::iterator itBeginStage, const Pipeline::iterator itEndStage, const ITaskExecutor::Ptr callbackExecutor={})
	Creates and run the first stage task. If destructor was not called add a new std::future to the AsyncInferRequestThreadSafeDefault::_futures list that would be used to wait AsyncInferRequestThreadSafeDefault::_pipeline finish. More...

void	StopAndWait ()
	Forbids pipeline start and wait for all started pipelines. More...

virtual void	StartAsync_ThreadUnsafe ()
	Starts an asynchronous pipeline thread unsafe. More...

virtual void	Infer_ThreadUnsafe ()
	Performs inference of pipeline in syncronous mode. More...

void	InferUsingAsync ()
	Implements Infer() using StartAsync() and Wait()

Protected Member Functions inherited from InferenceEngine::IInferRequestInternal
	~IInferRequestInternal ()
	Destroys the object.

void	execDataPreprocessing (InferenceEngine::BlobMap &preprocessedBlobs, bool serial=false)
	Checks and executes input data pre-processing if needed. More...

bool	findInputAndOutputBlobByName (const std::string &name, InputInfo::Ptr &foundInput, DataPtr &foundOutput) const
	Helper function to find input or output blob by name. More...

bool	preProcessingRequired (const InputInfo::Ptr &info, const Blob::Ptr &userBlob, const Blob::Ptr &deviceBlob=nullptr)
	Checks whether pre-processing step is required for a given input. More...

void	addInputPreProcessingFor (const std::string &name, Blob::Ptr const &from, const Blob::Ptr &to)

Protected Attributes
ITaskExecutor::Ptr	_requestExecutor
	Used to run inference CPU tasks.

ITaskExecutor::Ptr	_callbackExecutor
	Used to run post inference callback in asynchronous pipline.

ITaskExecutor::Ptr	_syncCallbackExecutor
	Used to run post inference callback in synchronous pipline.

Pipeline	_pipeline
	Pipeline variable that should be filled by inherited class.

Pipeline	_syncPipeline
	Synchronous pipeline variable that should be filled by inherited class.

Protected Attributes inherited from InferenceEngine::IInferRequestInternal
InferenceEngine::InputsDataMap	_networkInputs
	Holds information about network inputs info.

InferenceEngine::OutputsDataMap	_networkOutputs
	Holds information about network outputs data.

InferenceEngine::BlobMap	_inputs
	A map of user passed blobs for network inputs.

InferenceEngine::BlobMap	_deviceInputs
	A map of actual network inputs, in plugin specific format.

InferenceEngine::BlobMap	_outputs
	A map of user passed blobs for network outputs.

std::map< std::string, PreProcessDataPtr >	_preProcData
	A map of pre-process data per input.

int	m_curBatch = -1
	Current batch value used in dynamic batching.

std::shared_ptr< IExecutableNetworkInternal >	_exeNetwork
	A shared pointer to IInferRequestInternal. More...

Callback	_callback
	A callback.

Friends
struct	DisableCallbackGuard

Detailed Description

Base class with default implementation of asynchronous multi staged inference request. To customize pipeline stages derived class should change the content of AsyncInferRequestThreadSafeDefault::_pipeline member container. It consists of pairs of tasks and executors which will run the task. The class is recommended to be used by plugins as a base class for asynchronous inference request implementation.

Note: To synchronize derived context with stages derived class should call AsyncInferRequestThreadSafeDefault::StopAndWait() function in destructor.

Example: Here is an example of asynchronous inference request implementation for some accelerator device. It uses 5 different executors to run different stages of a synchronous inference request.

// Inherits from AsyncInferRequestThreadSafeDefault
class AcceleratorAsyncInferRequest : public AsyncInferRequestThreadSafeDefault {
    // Store the pointer to the synchronous request and five executors
    AcceleratorAsyncInferRequest(const AcceleratorSyncRequest::Ptr& syncRequest,
                                 const ITaskExecutor::Ptr& preprocessExecutor,
                                 const ITaskExecutor::Ptr& writeToDeviceExecutor,
                                 const ITaskExecutor::Ptr& runOnDeviceExecutor,
                                 const ITaskExecutor::Ptr& readFromDeviceExecutor,
                                 const ITaskExecutor::Ptr& postProcessExecutor) :
    AsyncInferRequestThreadSafeDefault(syncRequest, nullptr, nullptr),
    _accSyncRequest{syncRequest},
    _preprocessExecutor{preprocessExecutor},
    _writeToDeviceExecutor{writeToDeviceExecutor},
    _runOnDeviceExecutor{runOnDeviceExecutor},
    _readFromDeviceExecutor{readFromDeviceExecutor},
    _postProcessExecutor{postProcessExecutor}
    {
        // Five pipeline stages of synchronous infer request are run by different executors
        _pipeline = {
            { _preprocessExecutor , [this] {
                _accSyncRequest->Preprocess();
            }},
            { _writeToDeviceExecutor , [this] {
                _accSyncRequest->WriteToDevice();
            }},
            { _runOnDeviceExecutor , [this] {
                _accSyncRequest->RunOnDevice();
            }},
            { _readFromDeviceExecutor , [this] {
                _accSyncRequest->ReadFromDevice();
            }},
            { _postProcessExecutor , [this] {
                _accSyncRequest->PostProcess();
            }},
        };
    }
 
    // As all stages use _accSyncRequest member we should wait for all stages tasks before the destructor destroy this member.
    ~AcceleratorAsyncInferRequest() {
        StopAndWait();
    }
 
    AcceleratorSyncRequest::Ptr _accSyncRequest;
    ITaskExecutor::Ptr _preprocessExecutor, _writeToDeviceExecutor, _runOnDeviceExecutor, _readFromDeviceExecutor, _postProcessExecutor;
};

Constructor & Destructor Documentation

◆ AsyncInferRequestThreadSafeDefault()

InferenceEngine::AsyncInferRequestThreadSafeDefault::AsyncInferRequestThreadSafeDefault	(	const IInferRequestInternal::Ptr &	request,
		const ITaskExecutor::Ptr &	taskExecutor,
		const ITaskExecutor::Ptr &	callbackExecutor
	)

inline

Wraps a IInferRequestInternal::Ptr implementation and constructs a AsyncInferRequestThreadSafeDefault::_pipeline where taskExecutor is used to run IInferRequestInternal::Infer asynchronously.

Parameters

[in]	request	The synchronous request
[in]	taskExecutor	The task executor
[in]	callbackExecutor	The callback executor

Member Function Documentation

◆ GetBlob()

Blob::Ptr InferenceEngine::AsyncInferRequestThreadSafeDefault::GetBlob ( const std::string & name )

inlineoverridevirtual

Get input/output data to infer.

Note: Memory allocation doesn't happen

Parameters

name	- a name of input or output blob.
data	- a reference to input or output blob. The type of Blob must correspond to the network input precision and size.

Implements InferenceEngine::IInferRequestInternal.

◆ GetPerformanceCounts()

std::map<std::string, InferenceEngineProfileInfo> InferenceEngine::AsyncInferRequestThreadSafeDefault::GetPerformanceCounts ( ) const

inlineoverridevirtual

Queries performance measures per layer to get feedback of what is the most time consuming layer. Note: not all plugins may provide meaningful data.

Returns: - a map of layer names to profiling information for that layer.

Implements InferenceEngine::IInferRequestInternal.

◆ GetPreProcess()

const PreProcessInfo& InferenceEngine::AsyncInferRequestThreadSafeDefault::GetPreProcess ( const std::string & name ) const

inlineoverridevirtual

Gets pre-process for input data.

Parameters

name	Name of input blob.
info	pointer to a pointer to PreProcessInfo structure

Implements InferenceEngine::IInferRequestInternal.

◆ Infer()

void InferenceEngine::AsyncInferRequestThreadSafeDefault::Infer ( )

inlineoverridevirtual

Infers specified input(s) in synchronous mode.

Note: blocks all method of InferRequest while request is ongoing (running or waiting in queue)

Implements InferenceEngine::IInferRequestInternal.

◆ Infer_ThreadUnsafe()

virtual void InferenceEngine::AsyncInferRequestThreadSafeDefault::Infer_ThreadUnsafe ( )

inlineprotectedvirtual

Performs inference of pipeline in syncronous mode.

Note: Used by Infer which ensures thread-safety and calls this method after.

◆ QueryState()

std::vector<std::shared_ptr<InferenceEngine::IVariableStateInternal> > InferenceEngine::AsyncInferRequestThreadSafeDefault::QueryState ( )

inlineoverridevirtual

Queries memory states.

Returns: Returns memory states

Implements InferenceEngine::IInferRequestInternal.

◆ RunFirstStage()

void InferenceEngine::AsyncInferRequestThreadSafeDefault::RunFirstStage	(	const Pipeline::iterator	itBeginStage,
		const Pipeline::iterator	itEndStage,
		const ITaskExecutor::Ptr	callbackExecutor = `{}`
	)

inlineprotected

Creates and run the first stage task. If destructor was not called add a new std::future to the AsyncInferRequestThreadSafeDefault::_futures list that would be used to wait AsyncInferRequestThreadSafeDefault::_pipeline finish.

Parameters

[in]	itBeginStage	Iterator to begin of pipeline
[in]	itEndStage	End pipeline iterator
[in]	callbackExecutor	Final or error stage executor

◆ SetBatch()

void InferenceEngine::AsyncInferRequestThreadSafeDefault::SetBatch ( int batch )

inlineoverridevirtual

Sets new batch size when dynamic batching is enabled in executable network that created this request.

Parameters

batch - new batch size to be used by all the following inference calls for this request.

Implements InferenceEngine::IInferRequestInternal.

◆ SetBlob() [1/2]

void InferenceEngine::AsyncInferRequestThreadSafeDefault::SetBlob	(	const std::string &	name,
		const Blob::Ptr &	data
	)

inlineoverridevirtual

Set input/output data to infer.

Note: Memory allocation doesn't happen

Parameters

name	- a name of input or output blob.
data	- a reference to input or output blob. The type of Blob must correspond to the network input precision and size.

Implements InferenceEngine::IInferRequestInternal.

◆ SetBlob() [2/2]

void InferenceEngine::AsyncInferRequestThreadSafeDefault::SetBlob	(	const std::string &	name,
		const Blob::Ptr &	data,
		const PreProcessInfo &	info
	)

inlineoverridevirtual

Sets pre-process for input data.

Parameters

name	Name of input blob.
data	- a reference to input or output blob. The type of Blob must correspond to the network input precision and size.
info	Preprocess info for blob.

Implements InferenceEngine::IInferRequestInternal.

◆ SetCallback()

void InferenceEngine::AsyncInferRequestThreadSafeDefault::SetCallback ( Callback callback )

inlineoverridevirtual

Set callback function which will be called on success or failure of asynchronous request.

Parameters

callback - function to be called with the following description:

Implements InferenceEngine::IInferRequestInternal.

◆ StartAsync()

void InferenceEngine::AsyncInferRequestThreadSafeDefault::StartAsync ( )

inlineoverridevirtual

Start inference of specified input(s) in asynchronous mode.

Note: The method returns immediately. Inference starts also immediately.

Implements InferenceEngine::IInferRequestInternal.

◆ StartAsync_ThreadUnsafe()

virtual void InferenceEngine::AsyncInferRequestThreadSafeDefault::StartAsync_ThreadUnsafe ( )

inlineprotectedvirtual

Starts an asynchronous pipeline thread unsafe.

Note: Used by StartAsync which ensures thread-safety and calls this method after.

◆ StopAndWait()

void InferenceEngine::AsyncInferRequestThreadSafeDefault::StopAndWait ( )

inlineprotected

Forbids pipeline start and wait for all started pipelines.

Note: Should be called in derived class destructor to wait for completion of usage of derived context captured by pipeline tasks

◆ Wait()

StatusCode InferenceEngine::AsyncInferRequestThreadSafeDefault::Wait ( int64_t millis_timeout )

inlineoverridevirtual

Waits for completion of all pipeline stages If the pipeline raises an exception it will be rethrown here.

Parameters

millis_timeout A timeout is ms to wait or special enum value of InferRequest::WaitMode

Returns: A status code

Implements InferenceEngine::IInferRequestInternal.

The documentation for this class was generated from the following file:

cpp_interfaces/impl/ie_infer_async_request_thread_safe_default.hpp

Public Types

Public Member Functions

Protected Types

Protected Member Functions

Protected Attributes

Friends

Detailed Description

Constructor & Destructor Documentation

◆ AsyncInferRequestThreadSafeDefault()

Member Function Documentation

◆ GetBlob()

◆ GetPerformanceCounts()

◆ GetPreProcess()

◆ Infer()

◆ Infer_ThreadUnsafe()

◆ QueryState()

◆ RunFirstStage()

◆ SetBatch()

◆ SetBlob() [1/2]

◆ SetBlob() [2/2]

◆ SetCallback()

◆ StartAsync()

◆ StartAsync_ThreadUnsafe()

◆ StopAndWait()

◆ Wait()