Edit this page

Use Case - Integrate and Save Preprocessing Steps Into IR

Introduction

In previous sections we’ve covered how to add preprocessing steps and got the overview of Layout API.

For many applications it is also important to minimize model’s read/load time, so performing integration of preprocessing steps every time on application startup after ov::runtime::Core::read_model may look not convenient. In such cases, after adding of Pre- and Post-processing steps it can be useful to store new execution model to Intermediate Representation (IR, .xml format).

Most part of existing preprocessing steps can also be performed via command line options using Model Optimizer tool. Refer to Model Optimizer - Optimize Preprocessing Computation for details os such command line options.

Code example - saving model with preprocessing to IR

In case if you have some preprocessing steps which can’t be integrated into execution graph using Model Optimizer command line options (e.g. YUV->RGB color space conversion, Resize, etc.) it is possible to write simple code which:

  • Reads original model (IR, ONNX, Paddle)

  • Adds preprocessing/postprocessing steps

  • Saves resulting model as IR (.xml/.bin)

Let’s consider the example, there is an original ONNX model which takes one float32 input with shape {1, 3, 224, 224} with RGB channels order, with mean/scale values applied. User’s application can provide BGR image buffer with not fixed size. Additionally, we’ll also imagine that our application provides input images as batches, each batch contains 2 images. Here is how model conversion code may look like in your model preparation script

  • Includes / Imports

#include <openvino/runtime/core.hpp>
#include <openvino/core/preprocess/pre_post_process.hpp>
#include <openvino/pass/serialize.hpp>
from openvino.preprocess import PrePostProcessor, ColorFormat, ResizeAlgorithm
from openvino.runtime import Core, Layout, Type, set_batch
from openvino.runtime.passes import Manager

  • Preprocessing & Saving to IR code

// ========  Step 0: read original model =========
ov::Core core;
std::shared_ptr<ov::Model> model = core.read_model("/path/to/some_model.onnx");

// ======== Step 1: Preprocessing ================
ov::preprocess::PrePostProcessor prep(model);
// Declare section of desired application's input format
prep.input().tensor()
       .set_element_type(ov::element::u8)
       .set_layout("NHWC")
       .set_color_format(ov::preprocess::ColorFormat::BGR)
       .set_spatial_dynamic_shape();
// Specify actual model layout
prep.input().model()
       .set_layout("NCHW");
// Explicit preprocessing steps. Layout conversion will be done automatically as last step
prep.input().preprocess()
       .convert_element_type()
       .convert_color(ov::preprocess::ColorFormat::RGB)
       .resize(ov::preprocess::ResizeAlgorithm::RESIZE_LINEAR)
       .mean({123.675, 116.28, 103.53}) // Subtract mean after color conversion
       .scale({58.624, 57.12, 57.375});
// Dump preprocessor
std::cout << "Preprocessor: " << prep << std::endl;
model = prep.build();

// ======== Step 2: Change batch size ================
// In this example we also want to change batch size to increase throughput
ov::set_batch(model, 2);

// ======== Step 3: Save the model ================
std::string xml = "/path/to/some_model_saved.xml";
std::string bin = "/path/to/some_model_saved.bin";
ov::pass::Serialize(xml, bin).run_on_model(model);
# ========  Step 0: read original model =========
core = Core()
model = core.read_model(model='/path/to/some_model.onnx')

# ======== Step 1: Preprocessing ================
ppp = PrePostProcessor(model)
# Declare section of desired application's input format
ppp.input().tensor() \
    .set_element_type(Type.u8) \
    .set_spatial_dynamic_shape() \
    .set_layout(Layout('NHWC')) \
    .set_color_format(ColorFormat.BGR)

# Specify actual model layout
ppp.input().model().set_layout(Layout('NCHW'))

# Explicit preprocessing steps. Layout conversion will be done automatically as last step
ppp.input().preprocess() \
    .convert_element_type() \
    .convert_color(ColorFormat.RGB) \
    .resize(ResizeAlgorithm.RESIZE_LINEAR) \
    .mean([123.675, 116.28, 103.53]) \
    .scale([58.624, 57.12, 57.375])

# Dump preprocessor
print(f'Dump preprocessor: {ppp}')
model = ppp.build()

# ======== Step 2: Change batch size ================
# In this example we also want to change batch size to increase throughput
set_batch(model, 2)

# ======== Step 3: Save the model ================
pass_manager = Manager()
pass_manager.register_pass(pass_name="Serialize",
                           xml_path='/path/to/some_model_saved.xml',
                           bin_path='/path/to/some_model_saved.bin')
pass_manager.run_passes(model)

Application code - load model to target device

After this, your application’s code can load saved file and don’t perform preprocessing anymore. In this example we’ll also enable model caching to minimize load time when cached model is available

ov::Core core;
core.set_property(ov::cache_dir("/path/to/cache/dir"));

// In case that no preprocessing is needed anymore, we can load model on target device directly
// With cached model available, it will also save some time on reading original model
ov::CompiledModel compiled_model = core.compile_model("/path/to/some_model_saved.xml", "CPU");
core = Core()
core.set_property({'CACHE_DIR': '/path/to/cache/dir'})

# In case that no preprocessing is needed anymore, we can load model on target device directly
# With cached model available, it will also save some time on reading original model
compiled_model = core.compile_model('/path/to/some_model_saved.xml', 'CPU')

See Also

Prev Next