Custom OpenVINO Operations#
OpenVINO™ Extension API allows you to register custom operations to support models with operations which OpenVINO™ does not support out-of-the-box. This capability requires writing code in C++, so if you are using Python to develop your application you need to build a separate shared library implemented in C++ first and load it in Python using add_extension
API. Please refer to Create library with extensions for more details on library creation and usage. The remaining part of this document describes how to implement an operation class.
Operation Class#
To add your custom operation, create a new class that extends ov::Op
, which is in turn derived from ov::Node
, the base class for all graph operations in OpenVINO™. To add ov::Op
, include the next file:
#include <openvino/op/op.hpp>
Follow the steps below to add a custom operation:
Add the
OPENVINO_OP
macro which defines aNodeTypeInfo
object that identifies the type of the operation to the graph users and helps with dynamic type resolution. The type info of an operation currently consists of a string operation identifier and a string for operation version.Implement default constructor and constructors that optionally take the operation inputs and attributes as parameters.
Override the shape inference method
validate_and_infer_types
. This method is called multiple times during graph manipulations to determine the shapes and element types of the operations outputs. To access the input shapes and input element types, use theget_input_partial_shape()
andget_input_element_type()
methods ofov::Node
. Set the inferred shape and element type of the output usingset_output_type
.Override the
clone_with_new_inputs
method, which enables graph manipulation routines to create copies of this operation and connect it to different nodes during optimization.Override the
visit_attributes
method, which enables serialization and deserialization of operation attributes. AnAttributeVisitor
is passed to the method, and the implementation is expected to walk over all the attributes in the op using the type-awareon_attribute
helper. Helpers are already implemented for standard C++ types likeint64_t
,float
,bool
,vector
, and for existing OpenVINO defined types.Override
evaluate
, which is an optional method that enables fallback of some devices to this implementation and the application of constant folding if there is a custom operation on the constant branch. If your operation containsevaluate
method you also need to override thehas_evaluate
method, this method allows to get information about availability ofevaluate
method for the operation.
Based on that, declaration of an operation class can look as follows:
Operation Constructors#
OpenVINO™ operation contains two constructors:
Default constructor, which enables you to create an operation without attributes
Constructor that creates and validates an operation with specified inputs and attributes
Identity::Identity(const ov::Output<ov::Node>& arg) : Op({arg}) {
constructor_validate_and_infer_types();
}
validate_and_infer_types()
#
ov::Node::validate_and_infer_types
method validates operation attributes and calculates output shapes using attributes of the operation.
void Identity::validate_and_infer_types() {
// Operation doesn't change shapes end element type
set_output_type(0, get_input_element_type(0), get_input_partial_shape(0));
}
clone_with_new_inputs()
#
ov::Node::clone_with_new_inputs
method creates a copy of the operation with new inputs.
std::shared_ptr<ov::Node> Identity::clone_with_new_inputs(const ov::OutputVector& new_args) const {
OPENVINO_ASSERT(new_args.size() == 1, "Incorrect number of new arguments");
return std::make_shared<Identity>(new_args.at(0));
}
visit_attributes()
#
ov::Node::visit_attributes
method enables you to visit all operation attributes.
bool Identity::visit_attributes(ov::AttributeVisitor& visitor) {
return true;
}
evaluate() and has_evaluate()
#
ov::Node::evaluate
method enables you to apply constant folding to an operation.
bool Identity::evaluate(ov::TensorVector& outputs, const ov::TensorVector& inputs) const {
const auto& in = inputs[0];
auto& out = outputs[0];
if (out.data() == in.data()) // Nothing to do
return true;
out.set_shape(in.get_shape());
memcpy(out.data(), in.data(), in.get_byte_size());
return true;
}
bool Identity::has_evaluate() const {
return true;
}