OpenVINO™ provides the following methods for runtime model reshaping:
InferenceEngine::CNNNetwork::reshape
method.InferenceEngine::CNNNetwork::reshape
method updates input shapes and propagates them down to the outputs of the model through all intermediate layers.NOTES:
- Starting with the 2021.1 release, the Model Optimizer converts topologies keeping shape-calculating sub-graphs by default, which enables correct shape propagation during reshaping in most cases.
- Older versions of IRs are not guaranteed to reshape successfully. Please regenerate them with the Model Optimizer of the latest version of OpenVINO™.
- If an ONNX model does not have a fully defined input shape and the model was imported with the ONNX importer, reshape the model before loading it to the plugin.
Set a new batch dimension value with the InferenceEngine::CNNNetwork::setBatchSize
method.
The meaning of a model batch may vary depending on the model design. This method does not deduce batch placement for inputs from the model architecture. It assumes that the batch is placed at the zero index in the shape for all inputs and uses the InferenceEngine::CNNNetwork::reshape
method to propagate updated shapes through the model.
The method transforms the model before a new shape propagation to relax a hard-coded batch dimension in the model, if any.
Use InferenceEngine::CNNNetwork::reshape
instead of InferenceEngine::CNNNetwork::setBatchSize
to set new input shapes for the model in case the model has:
The InferenceEngine::CNNNetwork::setBatchSize
method is a high-level API method that wraps the InferenceEngine::CNNNetwork::reshape
method call and works for trivial models from the batch placement standpoint. Use InferenceEngine::CNNNetwork::reshape
for other models.
Using the InferenceEngine::CNNNetwork::setBatchSize
method for models with a non-zero index batch placement or for models with inputs that do not have a batch dimension may lead to undefined behaviour.
You can change input shapes multiple times using the InferenceEngine::CNNNetwork::reshape
and InferenceEngine::CNNNetwork::setBatchSize
methods in any order. If a model has a hard-coded batch dimension, use InferenceEngine::CNNNetwork::setBatchSize
first to change the batch, then call InferenceEngine::CNNNetwork::reshape
to update other dimensions, if needed.
Inference Engine takes three kinds of a model description as an input, which are converted into an InferenceEngine::CNNNetwork
object:
InferenceEngine::Core::ReadNetwork
InferenceEngine::Core::ReadNetwork
InferenceEngine::CNNNetwork
InferenceEngine::CNNNetwork
keeps an ngraph::Function
object with the model description internally. The object should have fully defined input shapes to be successfully loaded to the Inference Engine plugins. To resolve undefined input dimensions of a model, call the CNNNetwork::reshape
method providing new input shapes before loading to the Inference Engine plugin.
Run the following code right after InferenceEngine::CNNNetwork
creation to explicitly check for model input names and shapes:
To feed input data of a shape that is different from the model input shape, reshape the model first.
Once the input shape of InferenceEngine::CNNNetwork
is set, call the InferenceEngine::Core::LoadNetwork
method to get an InferenceEngine::ExecutableNetwork
object for inference with updated shapes.
There are other approaches to reshape the model during the stage of IR generation or nGraph::Function creation.
Practically, some models are not ready to be reshaped. In this case, a new input shape cannot be set with the Model Optimizer or the InferenceEngine::CNNNetwork::reshape
method.
Operation semantics may impose restrictions on input shapes of the operation. Shape collision during shape propagation may be a sign that a new shape does not satisfy the restrictions. Changing the model input shape may result in intermediate operations shape collision.
Examples of such operations:
Const
second input cannot be resized by spatial dimensions due to operation semanticsModel structure and logic should not change significantly after model reshaping.
Pooling
operation with the fixed kernel size [H, W]. During spatial reshape, having the input of the shape [N, C, H1, W1], Pooling with the fixed kernel size [H, W] returns the output of the shape [N, C, H2, W2], where H2 and W2 are commonly not equal to 1
. It breaks the classification model structure. For example, publicly available Inception family models from TensorFlow* have this issue.pipeline.config
file. For details, refer to the Tensorflow Object Detection API models resizing techniques.The primary method of the feature is InferenceEngine::CNNNetwork::reshape
. It gets new input shapes and propagates it from input to output for all intermediates layers of the given network. The method takes InferenceEngine::ICNNNetwork::InputShapes
- a map of pairs: name of input data and its dimension.
The algorithm for resizing network is the following:
1) Collect the map of input names and shapes from Intermediate Representation (IR) using helper method InferenceEngine::CNNNetwork::getInputShapes
2) Set new input shapes
3) Call reshape
Here is a code example:
Shape Inference feature is used in Smart classroom sample.
Inference Engine provides a special mechanism that allows to add the support of shape inference for custom operations. This mechanism is described in the Extensibility documentation