Base class with default implementation of asynchronous multi staged inference request. To customize pipeline stages derived class should change the content of AsyncInferRequestThreadSafeDefault::_pipeline member container. It consists of pairs of tasks and executors which will run the task. The class is recommended to be used by plugins as a base class for asynchronous inference request implementation. More...

#include <ie_infer_async_request_thread_safe_default.hpp>

Inheritance diagram for InferenceEngine::AsyncInferRequestThreadSafeDefault:

Public Types
using	Ptr = std::shared_ptr< AsyncInferRequestThreadSafeDefault >
	A shared pointer to AsyncInferRequestThreadSafeDefault.

Public Types inherited from InferenceEngine::IAsyncInferRequestInternal
typedef std::shared_ptr< IAsyncInferRequestInternal >	Ptr
	A shared pointer to IAsyncInferRequestInternal interface.

Public Types inherited from InferenceEngine::IInferRequestInternal
typedef std::shared_ptr< IInferRequestInternal >	Ptr
	A shared pointer to a IInferRequestInternal interface.

Public Member Functions
	AsyncInferRequestThreadSafeDefault (const InferRequestInternal::Ptr &request, const ITaskExecutor::Ptr &taskExecutor, const ITaskExecutor::Ptr &callbackExecutor)
	Wraps a InferRequestInternal::Ptr implementation and constructs a AsyncInferRequestThreadSafeDefault::_pipeline where `taskExecutor` is used to run InferRequestInternal::Infer asynchronously. More...

	~AsyncInferRequestThreadSafeDefault ()
	Destroys the object, stops AsyncInferRequestThreadSafeDefault::_pipeline and waits for a finish.

StatusCode	Wait (int64_t millis_timeout) override
	Waits for completion of all pipeline stages If the pipeline raises an exception it will be rethrown here. More...

void	StartAsync () override
	Start inference of specified input(s) in asynchronous mode. More...

void	Infer () override
	Infers specified input(s) in synchronous mode. More...

std::map< std::string, InferenceEngineProfileInfo >	GetPerformanceCounts () const override
	Queries performance measures per layer to get feedback of what is the most time consuming layer. Note: not all plugins may provide meaningful data. More...

void	SetBlob (const std::string &name, const Blob::Ptr &data) override
	Set input/output data to infer. More...

void	SetBlob (const std::string &name, const Blob::Ptr &data, const PreProcessInfo &info) override
	Sets pre-process for input data. More...

Blob::Ptr	GetBlob (const std::string &name) override
	Get input/output data to infer. More...

const PreProcessInfo &	GetPreProcess (const std::string &name) const override
	Gets pre-process for input data. More...

void	SetBatch (int batch) override
	Sets new batch size when dynamic batching is enabled in executable network that created this request. More...

void	GetUserData (void **data) override
	Get arbitrary data for the request. More...

void	SetUserData (void *data) override
	Set arbitrary data for the request. More...

void	SetCompletionCallback (IInferRequest::CompletionCallback callback) override
	Set callback function which will be called on success or failure of asynchronous request. More...

void	SetPointerToPublicInterface (InferenceEngine::IInferRequest::Ptr ptr)
	Sets the pointer to public interface. More...

std::vector< InferenceEngine::IVariableStateInternal::Ptr >	QueryState () override
	Queries memory states. More...

void	ThrowIfCanceled () const

void	Cancel () override
	Cancel current inference request execution.

Public Member Functions inherited from InferenceEngine::IAsyncInferRequestInternal
virtual	~IAsyncInferRequestInternal ()=default
	A virtual destructor.

Public Member Functions inherited from InferenceEngine::IInferRequestInternal
virtual	~IInferRequestInternal ()=default
	Destroys the object.

Protected Types
using	Stage = std::pair< ITaskExecutor::Ptr, Task >
	Each pipeline stage is a Task that is executed by specified ITaskExecutor implementation.

using	Pipeline = std::vector< Stage >
	Pipeline is vector of stages.

Protected Member Functions
void	CheckState () const
	Throws exception if inference request is busy or canceled.

void	RunFirstStage (const Pipeline::iterator itBeginStage, const Pipeline::iterator itEndStage, const ITaskExecutor::Ptr callbackExecutor={})
	Creates and run the first stage task. If destructor was not called add a new std::future to the AsyncInferRequestThreadSafeDefault::_futures list that would be used to wait AsyncInferRequestThreadSafeDefault::_pipeline finish. More...

void	StopAndWait ()
	Forbids pipeline start and wait for all started pipelines. More...

virtual void	StartAsync_ThreadUnsafe ()
	Starts an asynchronous pipeline thread unsafe. More...

virtual void	Infer_ThreadUnsafe ()
	Performs inference of pipeline in syncronous mode. More...

void	InferUsingAsync ()
	Implements Infer() using StartAsync() and Wait()

Protected Attributes
ITaskExecutor::Ptr	_requestExecutor
	Used to run inference CPU tasks.

ITaskExecutor::Ptr	_callbackExecutor
	Used to run post inference callback in asynchronous pipline.

ITaskExecutor::Ptr	_syncCallbackExecutor
	Used to run post inference callback in synchronous pipline.

Pipeline	_pipeline
	Pipeline variable that should be filled by inherited class.

Pipeline	_syncPipeline
	Synchronous pipeline variable that should be filled by inherited class.

Friends
struct	DisableCallbackGuard

Detailed Description

Base class with default implementation of asynchronous multi staged inference request. To customize pipeline stages derived class should change the content of AsyncInferRequestThreadSafeDefault::_pipeline member container. It consists of pairs of tasks and executors which will run the task. The class is recommended to be used by plugins as a base class for asynchronous inference request implementation.

Note: To synchronize derived context with stages derived class should call AsyncInferRequestThreadSafeDefault::StopAndWait() function in destructor.

Example: Here is an example of asynchronous inference request implementation for some accelerator device. It uses 5 different executors to run different stages of a synchronous inference request.

// Inherits from AsyncInferRequestThreadSafeDefault
class AcceleratorAsyncInferRequest : public AsyncInferRequestThreadSafeDefault {
    // Store the pointer to the synchronous request and five executors
    AcceleratorAsyncInferRequest(const AcceleratorSyncRequest::Ptr& syncRequest,
                                 const ITaskExecutor::Ptr& preprocessExecutor,
                                 const ITaskExecutor::Ptr& writeToDeviceExecutor,
                                 const ITaskExecutor::Ptr& runOnDeviceExecutor,
                                 const ITaskExecutor::Ptr& readFromDeviceExecutor,
                                 const ITaskExecutor::Ptr& postProcessExecutor) :
    AsyncInferRequestThreadSafeDefault(syncRequest, nullptr, nullptr),
    _accSyncRequest{syncRequest},
    _preprocessExecutor{preprocessExecutor},
    _writeToDeviceExecutor{writeToDeviceExecutor},
    _runOnDeviceExecutor{runOnDeviceExecutor},
    _readFromDeviceExecutor{readFromDeviceExecutor},
    _postProcessExecutor{postProcessExecutor}
    {
        // Five pipeline stages of synchronous infer request are run by different executors
        _pipeline = {
            { _preprocessExecutor , [this] {
                _accSyncRequest->Preprocess();
            }},
            { _writeToDeviceExecutor , [this] {
                _accSyncRequest->WriteToDevice();
            }},
            { _runOnDeviceExecutor , [this] {
                _accSyncRequest->RunOnDevice();
            }},
            { _readFromDeviceExecutor , [this] {
                _accSyncRequest->ReadFromDevice();
            }},
            { _postProcessExecutor , [this] {
                _accSyncRequest->PostProcess();
            }},
        };
    }
 
    // As all stages use _accSyncRequest member we should wait for all stages tasks before the destructor destroy this member.
    ~AcceleratorAsyncInferRequest() {
        StopAndWait();
    }
 
    AcceleratorSyncRequest::Ptr _accSyncRequest;
    ITaskExecutor::Ptr _preprocessExecutor, _writeToDeviceExecutor, _runOnDeviceExecutor, _readFromDeviceExecutor, _postProcessExecutor;
};

Constructor & Destructor Documentation

◆ AsyncInferRequestThreadSafeDefault()

InferenceEngine::AsyncInferRequestThreadSafeDefault::AsyncInferRequestThreadSafeDefault	(	const InferRequestInternal::Ptr &	request,
		const ITaskExecutor::Ptr &	taskExecutor,
		const ITaskExecutor::Ptr &	callbackExecutor
	)

inline

Wraps a InferRequestInternal::Ptr implementation and constructs a AsyncInferRequestThreadSafeDefault::_pipeline where taskExecutor is used to run InferRequestInternal::Infer asynchronously.

Parameters

[in]	request	The synchronous request
[in]	taskExecutor	The task executor
[in]	callbackExecutor	The callback executor

Member Function Documentation

◆ GetBlob()

Blob::Ptr InferenceEngine::AsyncInferRequestThreadSafeDefault::GetBlob ( const std::string & name )

inlineoverridevirtual

Get input/output data to infer.

Note: Memory allocation doesn't happen

Parameters

name	- a name of input or output blob.

Returns: Returns input or output blob. The type of Blob must correspond to the network input precision and size.

Implements InferenceEngine::IInferRequestInternal.

◆ GetPerformanceCounts()

std::map<std::string, InferenceEngineProfileInfo> InferenceEngine::AsyncInferRequestThreadSafeDefault::GetPerformanceCounts ( ) const

inlineoverridevirtual

Queries performance measures per layer to get feedback of what is the most time consuming layer. Note: not all plugins may provide meaningful data.

Returns: Returns a map of layer names to profiling information for that layer.

Implements InferenceEngine::IInferRequestInternal.

◆ GetPreProcess()

const PreProcessInfo& InferenceEngine::AsyncInferRequestThreadSafeDefault::GetPreProcess ( const std::string & name ) const

inlineoverridevirtual

Gets pre-process for input data.

Parameters

name	Name of input blob.

Returns: Returns constant reference to PreProcessInfo structure

Implements InferenceEngine::IInferRequestInternal.

◆ GetUserData()

void InferenceEngine::AsyncInferRequestThreadSafeDefault::GetUserData ( void ** data )

inlineoverridevirtual

Get arbitrary data for the request.

Parameters

data	A pointer to a pointer to arbitrary data

Implements InferenceEngine::IAsyncInferRequestInternal.

◆ Infer()

void InferenceEngine::AsyncInferRequestThreadSafeDefault::Infer ( )

inlineoverridevirtual

Infers specified input(s) in synchronous mode.

Note: blocks all method of IInferRequest while request is ongoing (running or waiting in queue)

Implements InferenceEngine::IInferRequestInternal.

◆ Infer_ThreadUnsafe()

virtual void InferenceEngine::AsyncInferRequestThreadSafeDefault::Infer_ThreadUnsafe ( )

inlineprotectedvirtual

Performs inference of pipeline in syncronous mode.

Note: Used by Infer which ensures thread-safety and calls this method after.

◆ QueryState()

std::vector<InferenceEngine::IVariableStateInternal::Ptr> InferenceEngine::AsyncInferRequestThreadSafeDefault::QueryState ( )

inlineoverridevirtual

Queries memory states.

Returns: Returns memory states

Implements InferenceEngine::IInferRequestInternal.

◆ RunFirstStage()

void InferenceEngine::AsyncInferRequestThreadSafeDefault::RunFirstStage	(	const Pipeline::iterator	itBeginStage,
		const Pipeline::iterator	itEndStage,
		const ITaskExecutor::Ptr	callbackExecutor = `{}`
	)

inlineprotected

Creates and run the first stage task. If destructor was not called add a new std::future to the AsyncInferRequestThreadSafeDefault::_futures list that would be used to wait AsyncInferRequestThreadSafeDefault::_pipeline finish.

Parameters

[in]	itBeginStage	Iterator to begin of pipeline
[in]	itEndStage	End pipeline iterator
[in]	callbackExecutor	Final or error stage executor

◆ SetBatch()

void InferenceEngine::AsyncInferRequestThreadSafeDefault::SetBatch ( int batch )

inlineoverridevirtual

Sets new batch size when dynamic batching is enabled in executable network that created this request.

Parameters

batch - new batch size to be used by all the following inference calls for this request.

Implements InferenceEngine::IInferRequestInternal.

◆ SetBlob() [1/2]

void InferenceEngine::AsyncInferRequestThreadSafeDefault::SetBlob	(	const std::string &	name,
		const Blob::Ptr &	data
	)

inlineoverridevirtual

Set input/output data to infer.

Note: Memory allocation doesn't happen

Parameters

name	- a name of input or output blob.
data	- a reference to input or output blob. The type of Blob must correspond to the network input precision and size.

Implements InferenceEngine::IInferRequestInternal.

◆ SetBlob() [2/2]

void InferenceEngine::AsyncInferRequestThreadSafeDefault::SetBlob	(	const std::string &	name,
		const Blob::Ptr &	data,
		const PreProcessInfo &	info
	)

inlineoverridevirtual

Sets pre-process for input data.

Parameters

name	Name of input blob.
data	- a reference to input or output blob. The type of Blob must correspond to the network input precision and size.
info	Preprocess info for blob.

Implements InferenceEngine::IInferRequestInternal.

◆ SetCompletionCallback()

void InferenceEngine::AsyncInferRequestThreadSafeDefault::SetCompletionCallback ( IInferRequest::CompletionCallback callback )

inlineoverridevirtual

Set callback function which will be called on success or failure of asynchronous request.

Parameters

callback - function to be called with the following description:

Implements InferenceEngine::IAsyncInferRequestInternal.

◆ SetPointerToPublicInterface()

void InferenceEngine::AsyncInferRequestThreadSafeDefault::SetPointerToPublicInterface ( InferenceEngine::IInferRequest::Ptr ptr )

inline

Sets the pointer to public interface.

Note: Needed to correctly handle ownership between objects

Parameters

[in] ptr A shared pointer to a public IInferRequest interface.

◆ SetUserData()

void InferenceEngine::AsyncInferRequestThreadSafeDefault::SetUserData ( void * data )

inlineoverridevirtual

Set arbitrary data for the request.

Parameters

data	A pointer to a pointer to arbitrary data

Implements InferenceEngine::IAsyncInferRequestInternal.

◆ StartAsync()

void InferenceEngine::AsyncInferRequestThreadSafeDefault::StartAsync ( )

inlineoverridevirtual

Start inference of specified input(s) in asynchronous mode.

Note: The method returns immediately. Inference starts also immediately.

Implements InferenceEngine::IAsyncInferRequestInternal.

◆ StartAsync_ThreadUnsafe()

virtual void InferenceEngine::AsyncInferRequestThreadSafeDefault::StartAsync_ThreadUnsafe ( )

inlineprotectedvirtual

Starts an asynchronous pipeline thread unsafe.

Note: Used by StartAsync which ensures thread-safety and calls this method after.

◆ StopAndWait()

void InferenceEngine::AsyncInferRequestThreadSafeDefault::StopAndWait ( )

inlineprotected

Forbids pipeline start and wait for all started pipelines.

Note: Should be called in derived class destructor to wait for completion of usage of derived context captured by pipeline tasks

◆ Wait()

StatusCode InferenceEngine::AsyncInferRequestThreadSafeDefault::Wait ( int64_t millis_timeout )

inlineoverridevirtual

Waits for completion of all pipeline stages If the pipeline raises an exception it will be rethrown here.

Parameters

millis_timeout A timeout is ms to wait or special enum value of IInferRequest::WaitMode

Returns: A status code

Implements InferenceEngine::IAsyncInferRequestInternal.

The documentation for this class was generated from the following file:

cpp_interfaces/impl/ie_infer_async_request_thread_safe_default.hpp

Public Types

Public Member Functions

Protected Types

Protected Member Functions

Protected Attributes

Friends

Detailed Description

Constructor & Destructor Documentation

◆ AsyncInferRequestThreadSafeDefault()

Member Function Documentation

◆ GetBlob()

◆ GetPerformanceCounts()

◆ GetPreProcess()

◆ GetUserData()

◆ Infer()

◆ Infer_ThreadUnsafe()

◆ QueryState()

◆ RunFirstStage()

◆ SetBatch()

◆ SetBlob() [1/2]

◆ SetBlob() [2/2]

◆ SetCompletionCallback()

◆ SetPointerToPublicInterface()

◆ SetUserData()

◆ StartAsync()

◆ StartAsync_ThreadUnsafe()

◆ StopAndWait()

◆ Wait()