Use Case - Integrate and Save Preprocessing Steps Into IR

Previous sections covered the topic of the preprocessing steps and the overview of Layout API.

For many applications, it is also important to minimize read/load time of a model. Therefore, performing integration of preprocessing steps every time on application startup, after ov::runtime::Core::read_model, may seem inconvenient. In such cases, once pre and postprocessing steps have been added, it can be useful to store new execution model to OpenVINO Intermediate Representation (OpenVINO IR, .xml format).

Most available preprocessing steps can also be performed via command-line options, using Model Optimizer. For details on such command-line options, refer to the Optimizing Preprocessing Computation.

Code example - Saving Model with Preprocessing to OpenVINO IR

When some preprocessing steps cannot be integrated into the execution graph using Model Optimizer command-line options (for example, YUV->``RGB`` color space conversion, Resize, etc.), it is possible to write a simple code which:

  • Reads the original model (OpenVINO IR, TensorFlow, TensorFlow Lite, ONNX, PaddlePaddle).

  • Adds the preprocessing/postprocessing steps.

  • Saves resulting model as IR (.xml and .bin).

Consider the example, where an original ONNX model takes one float32 input with the {1, 3, 224, 224} shape, the RGB channel order, and mean/scale values applied. In contrast, the application provides BGR image buffer with a non-fixed size and input images as batches of two. Below is the model conversion code that can be applied in the model preparation script for such a case.

  • Includes / Imports

from openvino.preprocess import PrePostProcessor, ColorFormat, ResizeAlgorithm
from openvino.runtime import Core, Layout, Type, set_batch
# First method - imports
from openvino.runtime import serialize
# Second method - imports
from openvino.runtime.passes import Manager, Serialize
 #include <openvino/runtime/core.hpp>
 #include <openvino/core/preprocess/pre_post_process.hpp>
 #include <openvino/pass/serialize.hpp>
  • Preprocessing & Saving to the OpenVINO IR code.

from openvino.preprocess import PrePostProcessor, ColorFormat, ResizeAlgorithm
from openvino.runtime import Core, Layout, Type, set_batch
# First method - imports
from openvino.runtime import serialize
# Second method - imports
from openvino.runtime.passes import Manager, Serialize
 #include <openvino/runtime/core.hpp>
 #include <openvino/core/preprocess/pre_post_process.hpp>
 #include <openvino/pass/serialize.hpp>

Application Code - Load Model to Target Device

After this, the application code can load a saved file and stop preprocessing. In this case, enable model caching to minimize load time when the cached model is available.

core = Core()
core.set_property({'CACHE_DIR': '/path/to/cache/dir'})

# In case that no preprocessing is needed anymore, we can load model on target device directly
# With cached model available, it will also save some time on reading original model
compiled_model = core.compile_model('/path/to/some_model_saved.xml', 'CPU')
 ov::Core core;
 core.set_property(ov::cache_dir("/path/to/cache/dir"));

 // In case that no preprocessing is needed anymore, we can load model on target device directly
 // With cached model available, it will also save some time on reading original model
 ov::CompiledModel compiled_model = core.compile_model("/path/to/some_model_saved.xml", "CPU");

Additional Resources