Optimize Preprocessing

This tutorial is also available as a Jupyter notebook that can be cloned directly from GitHub. See the installation guide for instructions to run this tutorial locally on Windows, Linux or macOS.


When input data does not fit the model input tensor perfectly, additional operations/steps are needed to transform the data to the format expected by the model. This tutorial demonstrates how it could be performed with Preprocessing API. Preprocessing API is an easy-to-use instrument, that enables integration of preprocessing steps into an execution graph and performing it on a selected device, which can improve device utilization. For more information about Preprocessing API, see this overview and details

This tutorial include following steps:

  • Downloading the model.

  • Setup preprocessing with Preprocessing API, loading the model and inference with original image.

  • Fitting image to the model input type and inference with prepared image.

  • Comparing results on one picture.

  • Comparing performance.

Table of contents:


# Install openvino package
!pip install -q "openvino==2023.1.0.dev20230811" tensorflow opencv-python matplotlib


import time
from pathlib import Path

import cv2
import matplotlib.pyplot as plt
import numpy as np
import openvino as ov
import tensorflow as tf
2023-09-08 23:04:01.488557: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0.
2023-09-08 23:04:01.524594: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-09-08 23:04:02.060166: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT

Setup image and device

image_path = "../data/image/coco.jpg"
import ipywidgets as widgets

core = ov.Core()
device = widgets.Dropdown(
    options=core.available_devices + ["AUTO"],

Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')

Downloading the model

This tutorial uses the InceptionResNetV2. The InceptionResNetV2 model is the second of the Inception family of models designed to perform image classification. Like other Inception models, InceptionResNetV2 has been pre-trained on the ImageNet data set. For more details about this family of models, see the research paper.

Load the model by using tf.keras.applications api and save it to the disk.

model_name = "InceptionResNetV2"

model_dir = Path("model")

model_path = model_dir / model_name

model = tf.keras.applications.InceptionV3()
2023-09-08 23:04:03.032233: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1956] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. model.compile_metrics will be empty until you train or evaluate the model.
WARNING:absl:Found untraced functions such as _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op while saving (showing 5 of 94). These functions will not be directly callable after loading.
INFO:tensorflow:Assets written to: model/InceptionResNetV2/assets
INFO:tensorflow:Assets written to: model/InceptionResNetV2/assets

Create core

core = ov.Core()

Check the original parameters of image

image = cv2.imread(image_path)
plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB));
print(f"The original shape of the image is {image.shape}")
print(f"The original data type of the image is {image.dtype}")
The original shape of the image is (577, 800, 3)
The original data type of the image is uint8

Setup preprocessing steps with Preprocessing API and perform inference

Intuitively, preprocessing API consists of the following parts:

  • Tensor - declares user data format, like shape, layout, precision, color format from actual user’s data.

  • Steps - describes sequence of preprocessing steps which need to be applied to user data.

  • Model - specifies model data format. Usually, precision and shape are already known for model, only additional information, like layout can be specified.

Graph modifications of a model shall be performed after the model is read from a drive and before it is loaded on the actual device.

Pre-processing support following operations (please, see more details here)

  • Mean/Scale Normalization

  • Converting Precision

  • Converting layout (transposing)

  • Resizing Image

  • Color Conversion

  • Custom Operations

Convert model to OpenVINO IR with model conversion API

The options for preprocessing are not required.

ir_path = model_dir / "ir_model" / f"{model_name}.xml"

ppp_model = None

if ir_path.exists():
    ppp_model = core.read_model(model=ir_path)
    print(f"Model in OpenVINO format already exists: {ir_path}")
    ppp_model = ov.convert_model(model_path,
    ov.save_model(ppp_model, str(ir_path))

Create PrePostProcessor Object

The `PrePostProcessor() <https://docs.openvino.ai/2023.0/classov_1_1preprocess_1_1PrePostProcessor.html#doxid-classov-1-1preprocess-1-1-pre-post-processor>`__ class enables specifying the preprocessing and postprocessing steps for a model.

from openvino.preprocess import PrePostProcessor

ppp = PrePostProcessor(ppp_model)

Declare User’s Data Format

To address particular input of a model/preprocessor, use the PrePostProcessor.input(input_name) method. If the model has only one input, then simple PrePostProcessor.input() will get a reference to pre-processing builder for this input (a tensor, the steps, a model). In general, when a model has multiple inputs/outputs, each one can be addressed by a tensor name or by its index. By default, information about user’s input tensor will be initialized to same data (type/shape/etc) as model’s input parameter. User application can override particular parameters according to application’s data. Refer to the following page for more information about parameters for overriding.

Below is all the specified input information:

  • Precision is U8 (unsigned 8-bit integer).

  • Size is non-fixed, setup of one determined shape size can be done with .set_shape([1, 577, 800, 3])

  • Layout is “NHWC”. It means, for example: height=577, width=800, channels=3.

The height and width are necessary for resizing, and channels are needed for mean/scale normalization.

# setup formant of data
<openvino._pyopenvino.preprocess.InputTensorInfo at 0x7fab2c20c9b0>

Declaring Model Layout

Model input already has information about precision and shape. Preprocessing API is not intended to modify this. The only thing that may be specified is input data layout.

input_layer_ir = next(iter(ppp_model.inputs))
print(f"The input shape of the model is {input_layer_ir.shape}")

The input shape of the model is [1,299,299,3]
<openvino._pyopenvino.preprocess.InputModelInfo at 0x7fab2c204070>

Preprocessing Steps

Now, the sequence of preprocessing steps can be defined. For more information about preprocessing steps, see here.

Perform the following:

There is no need to specify conversion layout. If layouts are different, then such conversion will be added explicitly.

from openvino.preprocess import ResizeAlgorithm

ppp.input().preprocess().convert_element_type(ov.Type.f32) \
<openvino._pyopenvino.preprocess.PreProcessSteps at 0x7fab2c143670>

Integrating Steps into a Model

Once the preprocessing steps have been finished, the model can be finally built. It is possible to display PrePostProcessor configuration for debugging purposes.

print(f'Dump preprocessor: {ppp}')
model_with_preprocess = ppp.build()
Dump preprocessor: Input "input_1":
    User's input tensor: [1,?,?,3], [N,H,W,C], u8
    Model's expected tensor: [1,299,299,3], [N,H,W,C], f32
    Pre-processing steps (4):
      convert type (f32): ([1,?,?,3], [N,H,W,C], u8) -> ([1,?,?,3], [N,H,W,C], f32)
      resize to model width/height: ([1,?,?,3], [N,H,W,C], f32) -> ([1,299,299,3], [N,H,W,C], f32)
      mean (127.5,127.5,127.5): ([1,299,299,3], [N,H,W,C], f32) -> ([1,299,299,3], [N,H,W,C], f32)
      scale (127.5,127.5,127.5): ([1,299,299,3], [N,H,W,C], f32) -> ([1,299,299,3], [N,H,W,C], f32)

Load model and perform inference

def prepare_image_api_preprocess(image_path, model=None):
    image = cv2.imread(image_path)
    input_tensor = np.expand_dims(image, 0)
    return input_tensor

compiled_model_with_preprocess_api = core.compile_model(model=ppp_model, device_name=device.value)

ppp_output_layer = compiled_model_with_preprocess_api.output(0)

ppp_input_tensor = prepare_image_api_preprocess(image_path)
results = compiled_model_with_preprocess_api(ppp_input_tensor)[ppp_output_layer][0]

Fit image manually and perform inference

Load the model

model = core.read_model(model=ir_path)
compiled_model = core.compile_model(model=model, device_name=device.value)

Load image and fit it to model input

def manual_image_preprocessing(path_to_image, compiled_model):
    input_layer_ir = next(iter(compiled_model.inputs))

    # N, H, W, C = batch size, height, width, number of channels
    N, H, W, C = input_layer_ir.shape

    # load  image, image will be resized to model input size and converted to RGB
    img = tf.keras.preprocessing.image.load_img(image_path, target_size=(H, W), color_mode='rgb')

    x = tf.keras.preprocessing.image.img_to_array(img)
    x = np.expand_dims(x, axis=0)

    # will scale input pixels between -1 and 1
    input_tensor = tf.keras.applications.inception_resnet_v2.preprocess_input(x)

    return input_tensor

input_tensor = manual_image_preprocessing(image_path, compiled_model)
print(f"The shape of the image is {input_tensor.shape}")
print(f"The data type of the image is {input_tensor.dtype}")
The shape of the image is (1, 299, 299, 3)
The data type of the image is float32

Perform inference

output_layer = compiled_model.output(0)

result = compiled_model(input_tensor)[output_layer]

Compare results

Compare results on one image

def check_results(input_tensor, compiled_model, imagenet_classes):
    output_layer = compiled_model.output(0)

    results = compiled_model(input_tensor)[output_layer][0]

    top_indices = np.argsort(results)[-5:][::-1]
    top_softmax = results[top_indices]

    for index, softmax_probability in zip(top_indices, top_softmax):
        print(f"{imagenet_classes[index]}, {softmax_probability:.5f}")

    return top_indices, top_softmax

# Convert the inference result to a class name.
imagenet_classes = open("../data/datasets/imagenet/imagenet_2012.txt").read().splitlines()
imagenet_classes = ['background'] + imagenet_classes

# get result for inference with preprocessing api
print("Result of inference with Preprocessing API:")
res = check_results(ppp_input_tensor, compiled_model_with_preprocess_api, imagenet_classes)


# get result for inference with the manual preparing of the image
print("Result of inference with manual image setup:")
res = check_results(input_tensor, compiled_model, imagenet_classes)
Result of inference with Preprocessing API:
n02099601 golden retriever, 0.80560
n02098413 Lhasa, Lhasa apso, 0.10039
n02108915 French bulldog, 0.01915
n02111129 Leonberg, 0.00825
n02097047 miniature schnauzer, 0.00294

Result of inference with manual image setup:
n02098413 Lhasa, Lhasa apso, 0.76843
n02099601 golden retriever, 0.19322
n02111129 Leonberg, 0.00720
n02097047 miniature schnauzer, 0.00287
n02100877 Irish setter, red setter, 0.00115

Compare performance

def check_performance(compiled_model, preprocessing_function=None):
    num_images = 1000

    start = time.perf_counter()

    for _ in range(num_images):
        input_tensor = preprocessing_function(image_path, compiled_model)

    end = time.perf_counter()
    time_ir = end - start

    return time_ir, num_images

time_ir, num_images = check_performance(compiled_model, manual_image_preprocessing)
    f"IR model in OpenVINO Runtime/CPU with manual image preprocessing: {time_ir/num_images:.4f} "
    f"seconds per image, FPS: {num_images/time_ir:.2f}"

time_ir, num_images = check_performance(compiled_model_with_preprocess_api, prepare_image_api_preprocess)
    f"IR model in OpenVINO Runtime/CPU with preprocessing API: {time_ir/num_images:.4f} "
    f"seconds per image, FPS: {num_images/time_ir:.2f}"
IR model in OpenVINO Runtime/CPU with manual image preprocessing: 0.0153 seconds per image, FPS: 65.52
IR model in OpenVINO Runtime/CPU with preprocessing API: 0.0187 seconds per image, FPS: 53.40