Dynamic Shapes#

As it was demonstrated in the Changing Input Shapes article, there are models that support changing input shapes before model compilation in Core::compile_model. Reshaping models provides an ability to customize the model input shape for the exact size required in the end application. This article explains how the ability of model to reshape can further be leveraged in more dynamic scenarios.

Applying Dynamic Shapes#

Conventional “static” model reshaping works well when it can be done once per many model inference calls with the same shape. However, this approach does not perform efficiently if the input tensor shape is changed on every inference call. Calling the reshape() and compile_model() methods each time a new size comes is extremely time-consuming. A popular example would be inference of natural language processing models (like BERT) with arbitrarily-sized user input sequences. In this case, the sequence length cannot be predicted and may change every time inference is called. Dimensions that can be frequently changed are called dynamic dimensions. Dynamic shapes should be considered, when a real shape of input is not known at the time of the compile_model() method call.

Below are several examples of dimensions that can be naturally dynamic:

Sequence length dimension for various sequence processing models, like BERT
Spatial dimensions in segmentation and style transfer models
Batch dimension
Arbitrary number of detections in object detection models output

There are various methods to address input dynamic dimensions through combining multiple pre-reshaped models and input data padding. The methods are sensitive to model internals, do not always give optimal performance and are cumbersome. For a short overview of the methods, refer to the When Dynamic Shapes API is Not Applicable page. Apply those methods only if native dynamic shape API described in the following sections does not work or does not perform as expected.

The decision about using dynamic shapes should be based on proper benchmarking of a real application with real data. Unlike statically shaped models, dynamically shaped ones require different inference time, depending on input data shape or input tensor content. Furthermore, using the dynamic shapes can bring more overheads in memory and running time of each inference call depending on hardware plugin and model used.

Handling Dynamic Shapes#

This section describes how to handle dynamically shaped models with OpenVINO Runtime API version 2022.1 and higher. When using dynamic shapes, there are three main differences in the workflow than with static shapes:

Configuring the model
Preparing and inferencing dynamic data
Dynamic shapes in outputs

Preparing and Inferencing Dynamic Data#

After configuring a model with the reshape method, the next steps are to create tensors with the appropriate data shape and pass them to the model as an inference request. This is similar to the regular steps described in Integrate OpenVINO™ with Your Application. However, tensors can now be passed into the model with different shapes.

The sample below shows how a model can accept different input shapes. In the first case, the model runs inference on a 1x128 input shape and returns a result. In the second case, a 1x200 input shape is used, which the model can still handle because it is dynamically shaped.

Python

# For first inference call, prepare an input tensor with 1x128 shape and run inference request
input_data1 = np.ones(shape=[1,128])
infer_request.infer({input_tensor_name: input_data1})

# Get resulting outputs
output_tensor1 = infer_request.get_output_tensor()
output_data1 = output_tensor1.data[:]

# For second inference call, prepare a 1x200 input tensor and run inference request
input_data2 = np.ones(shape=[1,200])
infer_request.infer({input_tensor_name: input_data2})

# Get resulting outputs
output_tensor2 = infer_request.get_output_tensor()
output_data2 = output_tensor2.data[:]

C++

// The first inference call

// Create tensor compatible with the model input
// Shape {1, 128} is compatible with any reshape statements made in previous examples
auto input_tensor_1 = ov::Tensor(model->input().get_element_type(), {1, 128});
// ... write values to input_tensor_1

// Set the tensor as an input for the infer request
infer_request.set_input_tensor(input_tensor_1);

// Do the inference
infer_request.infer();

// Retrieve a tensor representing the output data
ov::Tensor output_tensor = infer_request.get_output_tensor();

// For dynamic models output shape usually depends on input shape,
// that means shape of output tensor is initialized after the first inference only
// and has to be queried after every infer request
auto output_shape_1 = output_tensor.get_shape();

// Take a pointer of an appropriate type to tensor data and read elements according to the shape
// Assuming model output is f32 data type
auto data_1 = output_tensor.data<float>();
// ... read values

// The second inference call, repeat steps:

// Create another tensor (if the previous one cannot be utilized)
// Notice, the shape is different from input_tensor_1
auto input_tensor_2 = ov::Tensor(model->input().get_element_type(), {1, 200});
// ... write values to input_tensor_2

infer_request.set_input_tensor(input_tensor_2);

infer_request.infer();

// No need to call infer_request.get_output_tensor() again
// output_tensor queried after the first inference call above is valid here.
// But it may not be true for the memory underneath as shape changed, so re-take a pointer:
auto data_2 = output_tensor.data<float>();

// and new shape as well
auto output_shape_2 = output_tensor.get_shape();

// ... read values in data_2 according to the shape output_shape_2

ov_output_const_port_t* input_port = NULL;
ov_element_type_e type = UNDEFINED;
ov_shape_t input_shape_1;
ov_tensor_t* input_tensor_1 = NULL;
ov_tensor_t* output_tensor = NULL;
ov_shape_t output_shape_1;
void* data_1 = NULL;
ov_shape_t input_shape_2;
ov_tensor_t* input_tensor_2 = NULL;
ov_shape_t output_shape_2;
void* data_2 = NULL;
// The first inference call

// Create tensor compatible with the model input
// Shape {1, 128} is compatible with any reshape statements made in previous examples
{
ov_model_const_input(model, &input_port);
ov_port_get_element_type(input_port, &type);
int64_t dims[2] = {1, 128};
ov_shape_create(2, dims, &input_shape_1);
ov_tensor_create(type, input_shape_1, &input_tensor_1);
// ... write values to input_tensor
}

// Set the tensor as an input for the infer request
ov_infer_request_set_input_tensor(infer_request, input_tensor_1);

// Do the inference
ov_infer_request_infer(infer_request);

// Retrieve a tensor representing the output data
ov_infer_request_get_output_tensor(infer_request, &output_tensor);

// For dynamic models output shape usually depends on input shape,
// that means shape of output tensor is initialized after the first inference only
// and has to be queried after every infer request
ov_tensor_get_shape(output_tensor, &output_shape_1);

// Take a pointer of an appropriate type to tensor data and read elements according to the shape
// Assuming model output is f32 data type
ov_tensor_data(output_tensor, &data_1);
// ... read values

// The second inference call, repeat steps:

// Create another tensor (if the previous one cannot be utilized)
// Notice, the shape is different from input_tensor_1
{
int64_t dims[2] = {1, 200};
ov_shape_create(2, dims, &input_shape_2);
ov_tensor_create(type, input_shape_2, &input_tensor_2);
// ... write values to input_tensor_2
}

ov_infer_request_set_input_tensor(infer_request, input_tensor_2);
ov_infer_request_infer(infer_request);

// No need to call infer_request.get_output_tensor() again
// output_tensor queried after the first inference call above is valid here.
// But it may not be true for the memory underneath as shape changed, so re-take a pointer:
ov_tensor_data(output_tensor, &data_2);

// and new shape as well
ov_tensor_get_shape(output_tensor, &output_shape_2);
// ... read values in data_2 according to the shape output_shape_2

// free resource
ov_output_const_port_free(input_port);
ov_shape_free(&input_shape_1);
ov_tensor_free(input_tensor_1);
ov_shape_free(&output_shape_1);
ov_shape_free(&input_shape_2);
ov_tensor_free(input_tensor_2);
ov_shape_free(&output_shape_2);
ov_tensor_free(output_tensor);

For more information on how to apply input data to a model and run inference, see OpenVINO™ Inference Request.

Dynamic Shapes in Outputs#

When using dynamic dimensions in the input of a model, one or more output dimensions may also be dynamic depending on how the dynamic inputs are propagated through the model. For example, the batch dimension in an input shape is usually propagated through the whole model and appears in the output shape. It also applies to other dimensions, like sequence length for NLP models or spatial dimensions for segmentation models, that are propagated through the entire network.

To determine if the output has dynamic dimensions, the partial_shape property of the model’s output layers can be queried after the model has been read or reshaped. The same property can be queried for model inputs. For example:

Python

# Print output partial shape
print(model.output().partial_shape)

# Print input partial shape
print(model.input().partial_shape)

C++

// Print output partial shape
std::cout << model->output().get_partial_shape() << "\n";

// Print input partial shape
std::cout << model->input().get_partial_shape() << "\n";

ov_output_const_port_t* output_port = NULL;
ov_output_const_port_t* input_port = NULL;
ov_partial_shape_t partial_shape;
const char * str_partial_shape = NULL;

// Print output partial shape
{
ov_model_const_output(model, &output_port);
ov_port_get_partial_shape(output_port, &partial_shape);
str_partial_shape = ov_partial_shape_to_string(partial_shape);
printf("The output partial shape: %s", str_partial_shape);
}

// Print input partial shape
{
ov_model_const_input(model, &input_port);
ov_port_get_partial_shape(input_port, &partial_shape);
str_partial_shape = ov_partial_shape_to_string(partial_shape);
printf("The input partial shape: %s", str_partial_shape);
}

// free allocated resource
ov_free(str_partial_shape);
ov_partial_shape_free(&partial_shape);
ov_output_const_port_free(output_port);
ov_output_const_port_free(input_port);

If the output has any dynamic dimensions, they will be reported as ? or as a range (e.g. 1..10).

Output layers can also be checked for dynamic dimensions using the partial_shape.is_dynamic() property. This can be used on an entire output layer, or on an individual dimension, as shown in these examples:

Python

if model.input(0).partial_shape.is_dynamic:
    # input is dynamic
    pass

if model.output(0).partial_shape.is_dynamic:
    # output is dynamic
    pass

if model.output(0).partial_shape[1].is_dynamic:
    # 1-st dimension of output is dynamic
    pass

C++

auto model = core.read_model("model.xml");

if (model->input(0).get_partial_shape().is_dynamic()) {
    // input is dynamic
}

if (model->output(0).get_partial_shape().is_dynamic()) {
    // output is dynamic
}

if (model->output(0).get_partial_shape()[1].is_dynamic()) {
    // 1-st dimension of output is dynamic
}

ov_model_t* model = NULL;
ov_output_const_port_t* input_port = NULL;
ov_output_const_port_t* output_port = NULL;
ov_partial_shape_t partial_shape;

ov_core_read_model(core, "model.xml", NULL, &model);

// for input
{
ov_model_const_input_by_index(model, 0, &input_port);
ov_port_get_partial_shape(input_port, &partial_shape);
if (ov_partial_shape_is_dynamic(partial_shape)) {
    // input is dynamic
}
}

// for output
{
ov_model_const_output_by_index(model, 0, &output_port);
ov_port_get_partial_shape(output_port, &partial_shape);
if (ov_partial_shape_is_dynamic(partial_shape)) {
    // output is dynamic
}
}

// free allocated resource
ov_partial_shape_free(&partial_shape);
ov_output_const_port_free(input_port);
ov_output_const_port_free(output_port);

If at least one dynamic dimension exists in the output layer of a model, the actual shape of the output tensor will be determined during inference. Before the first inference, the output tensor’s memory is not allocated and has a shape of [0].

To pre-allocate space in memory for the output tensor, use the set_output_tensor method with the expected shape of the output. This will call the set_shape method internally, which will cause the initial shape to be replaced by the calculated shape.

Dynamic Shapes#

Applying Dynamic Shapes#

Handling Dynamic Shapes#

Configuring the Model#

Undefined Dimensions “Out Of the Box”#

Dimension Bounds#

Preparing and Inferencing Dynamic Data#

Dynamic Shapes in Outputs#