Group Plugin base classes#

group Plugin base classes

A set of base and helper classes to implement a plugin class.

Defines

OV_CREATE_PLUGIN#

Defines a name of a function creating plugin instance.

OV_ENUMERATE_DEVICES#

Defines a name of a function running the device-dispatch enumeration probe. A static build links every plugin into one binary, so the name is made per-device there (as for OV_CREATE_PLUGIN) to keep the definitions from colliding.

OV_DEFINE_PLUGIN_CREATE_FUNCTION(PluginType, version, ...)#

Defines the exported OV_CREATE_PLUGIN function which is used to create a plugin instance.

OV_DEFINE_PLUGIN_ENUMERATE_FUNCTION(enumerate_fn)#

Defines the exported ov_enumerate_dispatch_devices device-dispatch probe by forwarding to enumerate_fn (a callable with the ov::EnumerateDevicesFunc signature). Used by plugins that participate in device-name dispatch (e.g. the GPU plugin). The symbol is resolved and called by ov::Core BEFORE the plugin engine is constructed, so enumerate_fn must not build an engine/context. See ov::EnumerateDevicesFunc.

OV_DEFINE_PLUGIN_ENUMERATE_STUB()#

Defines the exported ov_enumerate_dispatch_devices probe as a no-op that reports no devices. Used by every plugin that does NOT participate in device-name dispatch, so the symbol is uniformly present across the plugin ABI. ov::Core never calls it for a single-candidate device name, so a stubbed plugin keeps exactly today’s behavior.

Typedefs

using DeviceCompatibilityScore = int32_t#

A score describing how well a plugin library can serve a physical device during device-name dispatch. Higher wins; 0 means “cannot serve” and is excluded.

Variables

static constexpr Property<std::vector<PropertyName>, PropertyMutability::RO> caching_properties = {"CACHING_PROPERTIES"}#

Read-only property to get a std::vector<PropertyName> of properties which should affect the hash calculation for model cache.

static constexpr Property<bool, PropertyMutability::RO> caching_with_mmap = {"CACHING_WITH_MMAP"}#

Read-only property to get a std::vector<PropertyName> of properties which should affect the loading time from cache.

static constexpr Property<bool, PropertyMutability::RW> exclusive_async_requests = {"EXCLUSIVE_ASYNC_REQUESTS"}#

Allow to create exclusive_async_requests with one executor.

static constexpr Property<std::string, PropertyMutability::WO> config_device_id = {"CONFIG_DEVICE_ID"}#

the property for setting of required device for which config to be updated values: device id starts from “0” - first device, “1” - second device, etc note: plugin may have different devices naming convention

static constexpr Property<std::map<std::string, std::string>, PropertyMutability::WO> device_id_map{"DISPATCH_DEVICE_ID_MAP"}#

Renames a plugin’s devices to the ids Core exposes for them, as {plugin id -> Core id}.

Several libraries may be registered under one device name (a “dispatch group”), in which case Core owns the numbering: it merges what the candidates enumerate and assigns the “.N” ids users address. A candidate’s own enumeration order may differ, so Core pushes this mapping to each group member before any other property, and the member must address those devices by the given ids from then on - including in the names it reports back (remote contexts, execution devices).

The map covers every device the member enumerated, so its id set stays as dense as its own enumeration; it is not necessarily zero-based. Members must report support for this property in ov::internal::supported_properties - Core refuses to put a library that does not into a group. It is never set for a device name served by a single library, whose ids Core does not rename.

static constexpr Property<int32_t, PropertyMutability::RW> threads_per_stream = {"THREADS_PER_STREAM"}#

Limit #threads that are used by IStreamsExecutor to execute parallel_for calls.

static constexpr Property<std::string, PropertyMutability::RO> compiled_model_runtime_properties{"COMPILED_MODEL_RUNTIME_PROPERTIES"}#

It contains compiled_model_runtime_properties information to make plugin runtime can check whether it is compatible with the cached compiled model, the result is returned by get_property() calling.

The information details are defined by plugin itself, each plugin may require different runtime contents. For example, CPU plugin will contain OV version, while GPU plugin will contain OV and GPU driver version, etc. Core doesn’t understand its content and only read it from plugin and write it into blob header.

static constexpr Property<bool, PropertyMutability::RO> compiled_model_runtime_properties_supported{"COMPILED_MODEL_RUNTIME_PROPERTIES_SUPPORTED"}#

Check whether the attached compiled_model_runtime_properties is supported by this device runtime.

static constexpr Property<float, PropertyMutability::RW> query_model_ratio = {"QUERY_MODEL_RATIO"}#

Read-write property to set the percentage of the estimated model size which is used to determine the query model results for further processing.

static constexpr Property<bool, PropertyMutability::RW> enable_lp_transformations = {"LP_TRANSFORMS_MODE"}#

Allow execution of low precision transformations in plugin’s pipelines.

struct EnumeratedDevice#
#include <iplugin.hpp>

One physical device a plugin library reports it can serve, during device-name dispatch. Produced by the enumeration probe (EnumerateDevicesFunc), consumed by ov::Core to reconcile candidate libraries for a shared device name.

Public Members

std::string internal_id = {}#

The device id THIS library uses internally (“.N”); may differ across libraries.

std::vector<uint8_t> fingerprint = {}#

Opaque cross-library identity token. Core compares it by equality only, never interprets it. Two libraries that build it over the same fields yield equal bytes for the same physical device.

DeviceCompatibilityScore score = PROBE_SCORE_INCOMPATIBLE#

How well this library serves the device (see PROBE_SCORE_* constants).

interface ICore#
#include <icore.hpp>

Minimal ICore interface to allow plugin to get information from Core OpenVINO class.

Public Functions

virtual std::shared_ptr<ov::Model> read_model(const std::string &model, const ov::Tensor &weights, bool frontend_mode = false) const = 0#

Reads IR xml and bin (with the same name) files.

Parameters:

  • model – string with IR

  • weights – shared pointer to constant blob with weights

  • frontend_mode – read network without post-processing or other transformations

Returns:

shared pointer to ov::Model

virtual std::shared_ptr<ov::Model> read_model(const std::shared_ptr<AlignedBuffer> &model, const std::shared_ptr<AlignedBuffer> &weights) const = 0#

Reads IR xml and bin from buffer. This method is not exposed to public API.

Parameters:

  • model – shared pointer to aligned buffer with IR

  • weights – shared pointer to aligned buffer with weights

Returns:

shared pointer to ov::Model

virtual std::shared_ptr<ov::Model> read_model(const std::filesystem::path &model_path, const std::filesystem::path &bin_path, const ov::AnyMap &properties) const = 0#

Reads IR xml and bin files.

Parameters:

  • model_path – path to IR file

  • bin_path – path to bin file, if path is empty, will try to read bin file with the same name as xml and if bin file with the same name was not found, will load IR without weights.

  • properties – Optional map of pairs: (property name, property value) relevant only for this read operation.

Returns:

shared pointer to ov::Model

virtual ov::SoPtr<ov::ICompiledModel> compile_model(const std::shared_ptr<const ov::Model> &model, const std::string &device_name, const ov::AnyMap &config = {}) const = 0#

Creates a compiled mdel from a model object.

Users can create as many models as they need and use them simultaneously (up to the limitation of the hardware resources)

Parameters:

  • model – OpenVINO Model

  • device_name – Name of device to load model to

  • config – Optional map of pairs: (config parameter name, config parameter value) relevant only for this load operation

Returns:

A pointer to compiled model

virtual ov::SoPtr<ov::ICompiledModel> compile_model(const std::shared_ptr<const ov::Model> &model, const ov::SoPtr<ov::IRemoteContext> &context, const ov::AnyMap &config = {}) const = 0#

Creates a compiled model from a model object.

Users can create as many models as they need and use them simultaneously (up to the limitation of the hardware resources)

Parameters:

  • model – OpenVINO Model

  • context – “Remote” (non-CPU) accelerator device-specific execution context to use

  • config – Optional map of pairs: (config parameter name, config parameter value) relevant only for this load operation

Returns:

A pointer to compiled model

virtual ov::SoPtr<ov::ICompiledModel> compile_model(const std::filesystem::path &model_path, const std::string &device_name, const ov::AnyMap &config) const = 0#

Creates a compiled model from a model file.

Users can create as many models as they need and use them simultaneously (up to the limitation of the hardware resources)

Parameters:

  • model_path – Path to model

  • device_name – Name of device to load model to

  • config – Optional map of pairs: (config parameter name, config parameter value) relevant only for this load operation

Returns:

A pointer to compiled model

virtual ov::SoPtr<ov::ICompiledModel> compile_model(const std::string &model_str, const ov::Tensor &weights, const std::string &device_name, const ov::AnyMap &config) const = 0#

Creates a compiled model from a model memory.

Users can create as many models as they need and use them simultaneously (up to the limitation of the hardware resources)

Parameters:

  • model_str – String data of model

  • weights – Model’s weights

  • device_name – Name of device to load model to

  • config – Optional map of pairs: (config parameter name, config parameter value) relevant only for this load operation

Returns:

A pointer to compiled model

virtual ov::SoPtr<ov::ICompiledModel> import_model(std::istream &model, const std::string &device_name, const ov::AnyMap &config = {}) const = 0#

Creates a compiled model from a previously exported model.

Parameters:

  • model – model stream

  • device_name – Name of device load executable model on

  • config – Optional map of pairs: (config parameter name, config parameter value) relevant only for this load operation*

Returns:

A pointer to compiled model

virtual ov::SoPtr<ov::ICompiledModel> import_model(std::istream &modelStream, const ov::SoPtr<ov::IRemoteContext> &context, const ov::AnyMap &config = {}) const = 0#

Creates a compiled model from a previously exported model.

Parameters:

  • model – model stream

  • context – Remote context

  • config – Optional map of pairs: (config parameter name, config parameter value) relevant only for this load operation*

Returns:

A pointer to compiled model

virtual ov::SoPtr<ov::ICompiledModel> import_model(const ov::Tensor &compiled_blob, const std::string &device_name, const ov::AnyMap &config = {}) const = 0#

Creates a compiled model from a previously exported model.

Parameters:

  • compiled_blob – model blob

  • device_name – Name of device load executable model on

  • config – Optional map of pairs: (config parameter name, config parameter value) relevant only for this load operation*

Returns:

A pointer to compiled model

virtual ov::SoPtr<ov::ICompiledModel> import_model(const ov::Tensor &compiled_blob, const ov::SoPtr<ov::IRemoteContext> &context, const ov::AnyMap &config = {}) const = 0#

Creates a compiled model from a previously exported model.

Parameters:

  • compiled_blob – model blob

  • context – Remote context

  • config – Optional map of pairs: (config parameter name, config parameter value) relevant only for this load operation*

Returns:

A pointer to compiled model

virtual ov::SupportedOpsMap query_model(const std::shared_ptr<const ov::Model> &model, const std::string &device_name, const ov::AnyMap &config) const = 0#

Query device if it supports specified network with specified configuration.

Parameters:

  • model – OpenVINO Model

  • device_name – A name of a device to query

  • config – Optional map of pairs: (config parameter name, config parameter value)

Returns:

An object containing a map of pairs a layer name -> a device name supporting this layer.

virtual std::vector<std::string> get_available_devices() const = 0#

Returns devices available for neural networks inference.

Returns:

A vector of devices. The devices are returned as { CPU, GPU.0, GPU.1, MYRIAD } If there more than one device of specific type, they are enumerated with .# suffix.

virtual ov::SoPtr<ov::IRemoteContext> create_context(const std::string &device_name, const AnyMap &args) const = 0#

Create a new shared context object on specified accelerator device using specified plugin-specific low level device API parameters (device handle, pointer, etc.)

Parameters:

  • device_name – Name of a device to create new shared context on.

  • params – Map of device-specific shared context parameters.

Returns:

A shared pointer to a created remote context.

virtual ov::SoPtr<ov::IRemoteContext> get_default_context(const std::string &device_name) const = 0#

Get a pointer to default shared context object for the specified device.

Parameters:

device_name – - A name of a device to get create shared context from.

Returns:

A shared pointer to a default remote context.

virtual Any get_property(const std::string &device_name, const std::string &name, const AnyMap &arguments) const = 0#

Gets properties related to device behaviour.

Parameters:

  • device_name – Name of a device to get a property value.

  • name – Property name.

  • arguments – Additional arguments to get a property.

Returns:

Value of a property corresponding to the property name.

template<typename T, PropertyMutability M>
inline T get_property(const std::string &device_name, const Property<T, M> &property) const#

Gets properties related to device behaviour.

Template Parameters:

  • T – Type of a returned value.

  • M – Property mutability.

Parameters:

  • deviceName – Name of a device to get a property value.

  • property – Property object.

Returns:

Property value.

template<typename T, PropertyMutability M>
inline T get_property(const std::string &device_name, const Property<T, M> &property, const AnyMap &arguments) const#

Gets properties related to device behaviour.

Template Parameters:

  • T – Type of a returned value.

  • M – Property mutability.

Parameters:

  • deviceName – Name of a device to get a property value.

  • property – Property object.

  • arguments – Additional arguments to get a property.

Returns:

Property value.

virtual AnyMap get_supported_property(const std::string &full_device_name, const AnyMap &properties, const bool keep_core_property = true) const = 0#

Get only properties that are supported by specified device.

Parameters:

  • full_device_name – Name of a device (can be either virtual or hardware)

  • properties – Properties that can contains configs that are not supported by device

  • keep_core_property – Whether to return core-level properties

Returns:

map of properties that are supported by device

virtual ~ICore()#

Default virtual destructor.