Inference Engine takes three kinds of a model description as an input, which are converted into an
InferenceEngine::CNNNetwork keeps an
ngraph::Function object with the model description internally. The object should have fully defined input shapes to be successfully loaded to the Inference Engine plugins. To resolve undefined input dimensions of a model, call the
CNNNetwork::reshape method providing new input shapes before loading to the Inference Engine plugin.
Run the following code right after
InferenceEngine::CNNNetwork creation to explicitly check for model input names and shapes:
To feed input data of a shape that is different from the model input shape, reshape the model first.
OpenVINO™ provides the following methods for runtime model reshaping:
InferenceEngine::CNNNetwork::reshapemethod updates input shapes and propagates them down to the outputs of the model through all intermediate layers. You can reshape a model multiple times like in this application scheme:
- Starting with the 2021.1 release, the Model Optimizer converts topologies keeping shape-calculating sub-graphs by default, which enables correct shape propagation during reshaping.
- Older versions of IRs are not guaranteed to reshape successfully. Please regenerate them with the Model Optimizer of the latest version of OpenVINO™.
- If an ONNX model does not have a fully defined input shape and the model was imported with the ONNX importer, reshape the model before loading it to the plugin.
InferenceEngine::CNNNetwork::setBatchSizemethod deduces the index of a batch dimension based only on the input rank. This method does not work for models with a non-zero index batch placement or models with inputs without a batch dimension. The batch-setting algorithm does not involve the shape inference mechanism. Batch of input and output shapes for all layers is set to a new batch value without layer validation. It may cause both positive and negative side effects. Due to the limitations described above, the current method is not recommended to use. If you need to set a new batch size for the model, use the
Do not use runtime reshaping methods simultaneously, especially do not call the
CNNNetwork::reshape method after you use
InferenceEngine::CNNNetwork::setBatchSize method causes irreversible conversion of the internal model representation into the legacy model representation. The method does not use nGraph for shape inference which leads to reduced reshape opportunities and may affect the performance of the model.
Practically, some models are not ready to be reshaped. In this case, a new input shape cannot be set with the Model Optimizer or the
Operation semantics may impose restrictions on input shapes of the operation. Shape collision during shape propagation may be a sign that a new shape does not satisfy the restrictions. Changing the model input shape may result in intermediate operations shape collision.
Examples of such operations:
Reshapeoperation with a hard-coded output shape value
MatMuloperation with the
Constsecond input cannot be resized by spatial dimensions due to operation semantics
Model structure and logic should not change significantly after model reshaping.
Poolingoperation with the fixed kernel size [H, W]. During spatial reshape, having the input of the shape [N, C, H1, W1], Pooling with the fixed kernel size [H, W] returns the output of the shape [N, C, H2, W2], where H2 and W2 are commonly not equal to
1. It breaks the classification model structure. For example, publicly available Inception family models from TensorFlow* have this issue.
pipeline.configfile. For details, refer to the Tensorflow Object Detection API models resizing techniques.
The primary method of the feature is
InferenceEngine::CNNNetwork::reshape. It gets new input shapes and propagates it from input to output for all intermediates layers of the given network. The method takes
InferenceEngine::ICNNNetwork::InputShapes - a map of pairs: name of input data and its dimension.
The algorithm for resizing network is the following:
1) Collect the map of input names and shapes from Intermediate Representation (IR) using helper method
2) Set new input shapes
3) Call reshape
Here is a code example:
Shape Inference feature is used in Smart classroom sample.
Inference Engine provides a special mechanism that allows to add the support of shape inference for custom operations. This mechanism is described in the Extensibility documentation