Model Representation in OpenVINO™ Runtime#
In OpenVINO™ Runtime, a model is represented by the ov::Model
class.
The ov::Model
object stores shared pointers to ov::op::v0::Parameter
, ov::op::v0::Result
, and ov::op::Sink
operations,
which are inputs, outputs, and sinks of the graph. Sinks of the graph have no consumers and are not included in the results vector.
All other operations hold each other via shared pointers, in which a child operation holds its parent via a hard link. If an operation
has no consumers and is neither the Result
nor the Sink
operation whose shared pointer counter is zero, the operation will be
destructed and not be accessible anymore.
Each operation in ov::Model
has the std::shared_ptr<ov::Node>
type.
How OpenVINO Runtime Works with Models#
OpenVINO™ Runtime enables you to use different approaches to work with model inputs/outputs:
The
ov::Model::inputs()
/ov::Model::outputs()
methods are used to get vectors of all input/output ports.inputs = model.inputs outputs = model.outputs
/* Take information about all topology inputs */ auto inputs = model->inputs(); /* Take information about all topology outputs */ auto outputs = model->outputs();
For a model that has only one input or output, you can use the
ov::Model::input()
orov::Model::output()
methods without any arguments to get input or output port respectively.The
ov::Model::input()
andov::Model::output()
methods can be used with the index of inputs or outputs from the framework model to get specific ports by index.ov_model_input = model.input(index) ov_model_output = model.output(index)
auto ov_model_input = ov_model->input(index); auto ov_model_output = ov_model->output(index);
You can use the tensor name of input or output from the original framework model together with the
ov::Model::input()
orov::Model::output()
methods to get specific ports. It means that you do not need to have any additional mapping of names from framework to OpenVINO as it was before. OpenVINO Runtime allows the usage of native framework tensor names, for example:
Warning
All inputs/outputs of ov::Model
are numbered, so the preferred way to retrieve them is to use indices.
Using tensor names can potentially be a less reliable approach, since the mandatory
presence of tensor names for inputs and outputs is not guaranteed in the original frameworks.
Therefore ov::Model
may contain empty list of tensor_names
for inputs/outputs.
To get all tensor names which are associated with the corresponding input/output, OpenVINO
Runtime has get_names
method. To get some name from all names associated with a given input/output,
the get_any_name
method was introduced. These methods may return empty names list/empty name
if the names are not present.
ov_model_input = model.input(original_fw_in_tensor_name)
ov_model_output = model.output(original_fw_out_tensor_name)
auto ov_model_input = ov_model->input(original_fw_in_tensor_name);
auto ov_model_output = ov_model->output(original_fw_out_tensor_name);
For details on how to build a model in OpenVINO™ Runtime, see the Build a Model in OpenVINO Runtime section.
OpenVINO™ Runtime model representation uses special classes to work with model data types and shapes. The ov::element::Type
is used for data types.
ov_input.get_element_type()
ov_input->get_element_type();
Representation of Shapes#
OpenVINO™ Runtime provides two types for shape representation:
ov::Shape
- Represents static (fully defined) shapes.ov::PartialShape
- Represents dynamic shapes. This means that the rank or some of dimensions are dynamic (dimension defines an interval or undefined).
ov::PartialShape
can be converted to ov::Shape
by using the get_shape()
method if all dimensions are static; otherwise,
the conversion will throw an exception. For example:
partial_shape = node.output(0).get_partial_shape() # get zero output partial shape
if not partial_shape.is_dynamic: # or partial_shape.is_static
static_shape = partial_shape.get_shape()
ov::Shape static_shape;
ov::PartialShape partial_shape = node->output(0).get_partial_shape(); // get zero output partial shape
if (!partial_shape.is_dynamic() /* or partial_shape.is_static() */) {
static_shape = partial_shape.get_shape();
}
However, in most cases, before getting static shape using the get_shape()
method, you need to check if that shape is static.
Representation of Operations#
The ov::Op
class represents any abstract operation in the model representation. Use this class to create
custom operations.
Representation of Operation Sets#
An operation set (opset) is a collection of operations that can be used to construct a model. The ov::OpSet
class provides
the functionality to work with operation sets.
For each operation set, OpenVINO™ Runtime provides a separate namespace, for example opset8
.
Each OpenVINO™ Release introduces new operations and adds them to new operation sets, within which the new operations would change the behavior of previous operations. Using operation sets helps you avoid changing your application when new operations are introduced. For a complete list of operation sets supported in OpenVINO™ toolkit, see the Available Operations Sets. To add the support for custom operations, see OpenVINO Extensibility Mechanism.
Building a Model in OpenVINO™ Runtime#
You can create a model from source. This section illustrates how to construct a model composed of operations from an available operation set.
Operation set opsetX
integrates a list of pre-compiled operations that work for this purpose. In other words, opsetX
defines a set of operations for building a graph.
To build an ov::Model
instance from opset8
operations, include the following files:
import openvino as ov
#include <openvino/core/model.hpp>
#include <openvino/opsets/opset8.hpp>
The following code demonstrates how to create a simple model:
def create_simple_model():
# This example shows how to create ov::Function
#
# Parameter--->Multiply--->Add--->Result
# Constant---' /
# Constant---'
data = ops.parameter([3, 1, 2], ov.Type.f32)
mul_constant = ops.constant([1.5], ov.Type.f32)
mul = ops.multiply(data, mul_constant)
add_constant = ops.constant([0.5], ov.Type.f32)
add = ops.add(mul, add_constant)
res = ops.result(add)
return ov.Model([res], [data], "model")
std::shared_ptr<ov::Model> create_simple_model() {
// This example shows how to create ov::Model
//
// Parameter--->Multiply--->Add--->Result
// Constant---' /
// Constant---'
// Create opset8::Parameter operation with static shape
auto data = std::make_shared<ov::opset8::Parameter>(ov::element::f32, ov::Shape{3, 1, 2});
auto mul_constant = ov::opset8::Constant::create(ov::element::f32, ov::Shape{1}, {1.5});
auto mul = std::make_shared<ov::opset8::Multiply>(data, mul_constant);
auto add_constant = ov::opset8::Constant::create(ov::element::f32, ov::Shape{1}, {0.5});
auto add = std::make_shared<ov::opset8::Add>(mul, add_constant);
// Create opset8::Result operation
auto res = std::make_shared<ov::opset8::Result>(mul);
// Create OpenVINO function
return std::make_shared<ov::Model>(ov::ResultVector{res}, ov::ParameterVector{data});
}
The following code creates a model with several outputs:
def create_advanced_model():
# Advanced example with multi output operation
#
# Parameter->Split---0-->Result
# | `--1-->Relu-->Result
# `----2-->Result
data = ops.parameter(ov.Shape([1, 3, 64, 64]), ov.Type.f32)
# Create Constant for axis value
axis_const = ops.constant(1, dtype=ov.Type.i64)
# Create opset12::Split operation that splits input to three slices across 1st dimension
split = ops.split(data, axis_const, 3)
# Create opset12::Relu operation that takes 1st Split output as input
relu = ops.relu(split.output(1))
# Results operations will be created automatically based on provided OutputVector
return ov.Model([split.output(0), relu.output(0), split.output(2)], [data], "model")
std::shared_ptr<ov::Model> create_advanced_model() {
// Advanced example with multi output operation
//
// Parameter->Split---0-->Result
// | `--1-->Relu-->Result
// `----2-->Result
auto data = std::make_shared<ov::opset8::Parameter>(ov::element::f32, ov::Shape{1, 3, 64, 64});
// Create Constant for axis value
auto axis_const = ov::opset8::Constant::create(ov::element::i64, ov::Shape{} /*scalar shape*/, {1});
// Create opset8::Split operation that splits input to three slices across 1st dimension
auto split = std::make_shared<ov::opset8::Split>(data, axis_const, 3);
// Create opset8::Relu operation that takes 1st Split output as input
auto relu = std::make_shared<ov::opset8::Relu>(split->output(1) /*specify explicit output*/);
// Results operations will be created automatically based on provided OutputVector
return std::make_shared<ov::Model>(ov::OutputVector{split->output(0), relu, split->output(2)},
ov::ParameterVector{data});
}
Model Debugging Capabilities#
OpenVINO™ provides several debug capabilities:
To receive additional messages about applied model modifications, rebuild the OpenVINO™ Runtime library with the
-DENABLE_OPENVINO_DEBUG=ON
option.Model can be visualized to image from the xDot format:
def visualize_example(m : ov.Model):
# Need import:
# * import openvino.runtime.passes as passes
pass_manager = passes.Manager()
pass_manager.register_pass(passes.VisualizeTree(file_name='image.svg'))
pass_manager.run_passes(m)
void visualize_example(const std::shared_ptr<ov::Model>& m) {
// Need include:
// * openvino/pass/manager.hpp
// * openvino/pass/visualize_tree.hpp
ov::pass::Manager manager;
// Serialize ov::Model to before.svg file before transformation
manager.register_pass<ov::pass::VisualizeTree>("image.svg");
manager.run_passes(m);
}
`ov::pass::VisualizeTree` can be parametrized via environment variables:
OV_VISUALIZE_TREE_OUTPUT_SHAPES=1 - visualize shapes
OV_VISUALIZE_TREE_OUTPUT_TYPES=1 - visualize types
OV_VISUALIZE_TREE_MIN_MAX_DENORMAL=1 - pretty denormal values
OV_VISUALIZE_TREE_RUNTIME_INFO=1 - print runtime information
OV_VISUALIZE_TREE_IO=1 - print I/O ports
OV_VISUALIZE_TREE_MEMBERS_NAME=1 - print member names
Also model can be serialized to IR:
def serialize_example(m : ov.Model):
ov.serialize(m, xml_path='model.xml', bin_path='model.bin')
void serialize_example(const std::shared_ptr<ov::Model>& model) {
ov::serialize(model, "/path/to/file/model.xml", "/path/to/file/model.bin");
}