Custom nGraph Operations¶
Inference Engine Extension API allows you to register operation sets (opsets) with custom nGraph operations to support models with operations which OpenVINO™ does not support out-of-the-box.
Besides creating custom nGraph operations, to support custom operations in your model you must also create a Model Optimizer extension for the custom operations and an Inference Engine device plugin extension for the device you will use for inference.
Operation Class¶
To add your custom nGraph operation, create a new class that extends ngraph::Op
, which is in turn derived from ngraph::Node
, the base class for all graph operations in nGraph. Follow the steps below to add a custom nGraph operation:
Add the
NGRAPH_RTTI_DECLARATION
andNGRAPH_RTTI_DEFINITION
macros which define aNodeTypeInfo
object that identifies the type of the operation to the graph users and helps with dynamic type resolution. The type info of an nGraph operation currently consists of a string identifier and a version number, but this may change in the future.Implement constructors that optionally take the operation inputs and attributes as parameters.
Override the shape inference method
validate_and_infer_types
. This method is called multiple times during graph manipulations to determine the shapes and element types of the operations outputs. To access the input shapes and input element types, use theget_input_partial_shape()
andget_input_element_type()
methods ofngraph::Node
. Set the inferred shape and element type of the output usingset_output_type
.Override the
clone_with_new_inputs
method, which enables graph manipulation routines to create copies of this operation and connect it to different nodes during optimization.Override the
visit_attributes
method, which enables serialization and deserialization of operation attributes. AnAttributeVisitor
is passed to the method, and the implementation is expected to walk over all the attributes in the op using the type-awareon_attribute
helper. Helpers are already implemented for standard C++ types likeint64_t
,float
,bool
,vector
, and for existing nGraph defined types.Override
evaluate
, which is an optional method that enables the application of constant folding if there is a custom operation on the constant branch. If your operation containsevaluate
method you also need to override thehas_evaluate
method, this method allow to get information about availability ofevaluate
method for the operation.
Based on that, declaration of an operation class can look as follows:
namespace TemplateExtension {
class Operation : public ngraph::op::Op {
public:
NGRAPH_RTTI_DECLARATION;
Operation() = default;
Operation(const ngraph::Output<ngraph::Node>& arg, int64_t add);
void validate_and_infer_types() override;
std::shared_ptr<ngraph::Node> clone_with_new_inputs(const ngraph::OutputVector& new_args) const override;
bool visit_attributes(ngraph::AttributeVisitor& visitor) override;
int64_t getAddAttr() const {
return add;
}
bool evaluate(const ngraph::HostTensorVector& outputs, const ngraph::HostTensorVector& inputs) const override;
bool has_evaluate() const override;
private:
int64_t add;
};
Class Fields¶
The provided implementation has several fields:
add
of typeint64_t
is an attribute of a custom operationtype_info
of typengraph::NodeTypeInfo
defines type and version of an operation
Operation Constructors¶
nGraph operation contains two constructors:
Default constructor, which enables you to create an operation without attributes
Constructor that creates and validates an operation with specified inputs and attributes
NGRAPH_RTTI_DEFINITION(TemplateExtension::Operation, "Template", 0);
Operation::Operation(const ngraph::Output<ngraph::Node>& arg, int64_t add): Op({arg}), add(add) {
constructor_validate_and_infer_types();
}
ngraph::Node::validate_and_infer_types
method validates operation attributes and calculates output shapes using attributes of the operation.
void Operation::validate_and_infer_types() {
// Operation doesn't change shapes end element type
set_output_type(0, get_input_element_type(0), get_input_partial_shape(0));
}
ngraph::Node::clone_with_new_inputs
method creates a copy of the nGraph operation with new inputs.
std::shared_ptr<ngraph::Node> Operation::clone_with_new_inputs(const ngraph::OutputVector& new_args) const {
if (new_args.size() != 1) {
throw ngraph::ngraph_error("Incorrect number of new arguments");
}
return std::make_shared<Operation>(new_args.at(0), add);
}
ngraph::Node::visit_attributes
method enables you to visit all operation attributes.
bool Operation::visit_attributes(ngraph::AttributeVisitor& visitor) {
visitor.on_attribute("add", add);
return true;
}
and¶
ngraph::Node::evaluate
method enables you to apply constant folding to an operation.
namespace {
template <class T>
void implementation(const T* input, T* output, int64_t add, size_t size) {
for (size_t i = 0; i < size; i++) {
output[i] = input[i] + add;
}
}
template <ngraph::element::Type_t ET>
bool evaluate_op(const ngraph::HostTensorPtr& arg0, const ngraph::HostTensorPtr& out, int64_t add) {
size_t size = ngraph::shape_size(arg0->get_shape());
implementation(arg0->get_data_ptr<ET>(), out->get_data_ptr<ET>(), add, size);
return true;
}
} // namespace
bool Operation::evaluate(const ngraph::HostTensorVector& outputs, const ngraph::HostTensorVector& inputs) const {
switch (inputs[0]->get_element_type()) {
case ngraph::element::Type_t::i8:
return evaluate_op<ngraph::element::Type_t::i8>(inputs[0], outputs[0], getAddAttr());
case ngraph::element::Type_t::i16:
return evaluate_op<ngraph::element::Type_t::i16>(inputs[0], outputs[0], getAddAttr());
case ngraph::element::Type_t::i32:
return evaluate_op<ngraph::element::Type_t::i32>(inputs[0], outputs[0], getAddAttr());
case ngraph::element::Type_t::i64:
return evaluate_op<ngraph::element::Type_t::i64>(inputs[0], outputs[0], getAddAttr());
case ngraph::element::Type_t::u8:
return evaluate_op<ngraph::element::Type_t::u8>(inputs[0], outputs[0], getAddAttr());
case ngraph::element::Type_t::u16:
return evaluate_op<ngraph::element::Type_t::u16>(inputs[0], outputs[0], getAddAttr());
case ngraph::element::Type_t::u32:
return evaluate_op<ngraph::element::Type_t::u32>(inputs[0], outputs[0], getAddAttr());
case ngraph::element::Type_t::u64:
return evaluate_op<ngraph::element::Type_t::u8>(inputs[0], outputs[0], getAddAttr());
case ngraph::element::Type_t::bf16:
return evaluate_op<ngraph::element::Type_t::bf16>(inputs[0], outputs[0], getAddAttr());
case ngraph::element::Type_t::f16:
return evaluate_op<ngraph::element::Type_t::f16>(inputs[0], outputs[0], getAddAttr());
case ngraph::element::Type_t::f32:
return evaluate_op<ngraph::element::Type_t::f32>(inputs[0], outputs[0], getAddAttr());
default:
break;
}
return false;
}
bool Operation::has_evaluate() const {
switch (get_input_element_type(0)) {
case ngraph::element::Type_t::i8:
case ngraph::element::Type_t::i16:
case ngraph::element::Type_t::i32:
case ngraph::element::Type_t::i64:
case ngraph::element::Type_t::u8:
case ngraph::element::Type_t::u16:
case ngraph::element::Type_t::u32:
case ngraph::element::Type_t::u64:
case ngraph::element::Type_t::bf16:
case ngraph::element::Type_t::f16:
case ngraph::element::Type_t::f32:
return true;
default:
break;
}
return false;
}
Register Custom Operations in Extension Class¶
To add custom operations to the Extension class, create an operation set with custom operations and implement the InferenceEngine::IExtension::getOpSets
method:
std::map<std::string, ngraph::OpSet> Extension::getOpSets() {
std::map<std::string, ngraph::OpSet> opsets;
ngraph::OpSet opset;
opset.insert<Operation>();
#ifdef OPENCV_IMPORT_ENABLED
opset.insert<FFTOp>();
#endif
opsets["custom_opset"] = opset;
return opsets;
}
This method returns a map of opsets that exist in the extension library. nGraph provides an opset mechanism to group operations into clusters. Different opsets distinguish between different versions of one operation.
When specifying opset names, follow the rules below:
Use unique opset names.
Do not use the following built-in opset names:
extension
,experimental
,opset1
,opset2
,opset3
, … ,opsetN
.Make sure that the Model Optimizer and your extension use the same opset names.
IR v10 operations have the mandatory
version
attribute specifying the opset. Operations from the default opset cannot be redefined.
Use a custom opset to create a new operation or extend functionality of an existing operation from another opset.