预处理¶

本指南通过与之前的推理引擎 API 中的预处理功能进行比较，介绍 API 2.0 中预处理的工作机制。此外，还通过代码示例演示了如何将预处理场景从推理引擎迁移到 API 2.0。

API 2.0 中预处理的工作机制¶

推理引擎 API 的 InferenceEngine::CNNNetwork 类中包含预处理功能。此类预处理信息不属于由 OpenVINO™ 器件执行的主要推理图的一部分。因此，在推理阶段之前，会单独存储和执行这些信息：

对大多数 OpenVINO™ 推理插件来说，预处理操作在 CPU 上执行。因此，它们会使 CPU 忙于处理计算任务，而不会占用加速器。
重新保存为 OpenVINO™ IR 文件格式后， InferenceEngine::CNNNetwork 中存储的预处理信息会丢失。

API 2.0 引入了一种新方法来向模型中添加预处理操作 - 每个预处理或后期处理操作将直接集成到模型中，并与推理图一起进行编译：

API 2.0 首先通过使用 ov::preprocess::PrePostProcessor 添加预处理操作，
然后通过使用 ov::Core::compile_model 在目标上编译模型。

通过将预处理操作作为 OpenVINO™ 操作集的一部分，即可以 OpenVINO™ IR 文件格式读取并序列化经过预处理的模型。

更重要的是，API 2.0 不会像推理引擎那样恢复任何默认布局。例如，{ 1, 224, 224, 3 } 和 { 1, 3, 224, 224 } 形状均应位于 NCHW 布局中，但只有后者如此。因此，API 中的某些预处理功能需要显式设置布局。如需了解如何操作，请参阅布局概述。例如，要通过偏维 H 和 W 执行图像缩放，预处理需要了解维度 H 和 W 是什么。

Note

请使用模型优化器预处理功能在要优化的模型中插入预处理操作。因此，应用不需要重复读取模型并设置预处理。您可以使用模型缓存功能缩短推理时间。

以下部分演示了如何将预处理场景从推理引擎 API 迁移到 API 2.0。代码片段假定您需要在推理引擎 API 中通过 tensor_name 对模型输入进行预处理，从而使用 operation_name 处理数据。

准备：在 Python 中导入预处理¶

要利用预处理，必须添加以下导入项。

推理引擎 API

from openvino.runtime import Core, Layout, Type
from openvino.preprocess import ColorFormat, PrePostProcessor, ResizeAlgorithm

API 2.0

from openvino.runtime import Core, Layout, Type
from openvino.preprocess import ColorFormat, PrePostProcessor, ResizeAlgorithm

有两个不同的命名空间：
* runtime。其中包含 API 2.0 类；
* 和 preprocess，它提供预处理 API。

使用平均值和标度值¶

推理引擎 API

Python

ppp = PrePostProcessor(model)
input = ppp.input(tensor_name)
# we only need to know where is C dimension
input.model().set_layout(Layout('...C'))
# specify scale and mean values, order of operations is important
input.preprocess().mean([116.78]).scale([57.21, 57.45, 57.73])
# insert preprocessing operations to the 'model'
model = ppp.build()

C++

ov::preprocess::PrePostProcessor ppp(model);
ov::preprocess::InputInfo& input = ppp.input(tensor_name);
// we only need to know where is C dimension
input.model().set_layout("...C");
// specify scale and mean values, order of operations is important
input.preprocess().mean(116.78f).scale({ 57.21f, 57.45f, 57.73f });
// insert preprocessing operations to the 'model'
model = ppp.build();

    // No preprocessing related interfaces provided by C API 1.0

API 2.0

Python

ppp = PrePostProcessor(model)
input = ppp.input(tensor_name)
# we only need to know where is C dimension
input.model().set_layout(Layout('...C'))
# specify scale and mean values, order of operations is important
input.preprocess().mean([116.78]).scale([57.21, 57.45, 57.73])
# insert preprocessing operations to the 'model'
model = ppp.build()

C++

ov::preprocess::PrePostProcessor ppp(model);
ov::preprocess::InputInfo& input = ppp.input(tensor_name);
// we only need to know where is C dimension
input.model().set_layout("...C");
// specify scale and mean values, order of operations is important
input.preprocess().mean(116.78f).scale({ 57.21f, 57.45f, 57.73f });
// insert preprocessing operations to the 'model'
model = ppp.build();

    ov_preprocess_prepostprocessor_t* preprocess = NULL;
    ov_preprocess_input_info_t* input_info = NULL;
    ov_preprocess_input_model_info_t* input_model = NULL;
    ov_preprocess_preprocess_steps_t* input_process = NULL;
    ov_layout_t* layout = NULL;
    ov_model_t* new_model = NULL;

    ov_preprocess_prepostprocessor_create(model, &preprocess);
    ov_preprocess_prepostprocessor_get_input_info_by_index(preprocess, 0, &input_info);
    ov_preprocess_input_info_get_model_info(input_info, &input_model);
    // we only need to know where is C dimension
    ov_layout_create("...C", &layout);
    ov_preprocess_input_model_info_set_layout(input_model, layout);
    // specify scale and mean values, order of operations is important
    ov_preprocess_input_info_get_preprocess_steps(input_info, &input_process);
    ov_preprocess_preprocess_steps_mean(input_process, 116.78f);
    ov_preprocess_preprocess_steps_scale(input_process, 57.21f);
    // insert preprocessing operations to the 'model'
    ov_preprocess_prepostprocessor_build(preprocess, &new_model);

    ov_layout_free(layout);
    ov_model_free(new_model);
    ov_preprocess_input_model_info_free(input_model);
    ov_preprocess_preprocess_steps_free(input_process);
    ov_preprocess_input_info_free(input_info);
    ov_preprocess_prepostprocessor_free(preprocess);

转换精度和布局¶

推理引擎 API

Python

ppp = PrePostProcessor(model)
input = ppp.input(tensor_name)
input.tensor().set_layout(Layout('NCHW')).set_element_type(Type.u8)
input.model().set_layout(Layout('NCHW'))
# layout and precision conversion is inserted automatically,
# because tensor format != model input format
model = ppp.build()

C++

ov::preprocess::PrePostProcessor ppp(model);
ov::preprocess::InputInfo& input = ppp.input(tensor_name);
input.tensor().set_layout("NHWC").set_element_type(ov::element::u8);
input.model().set_layout("NCHW");
// layout and precision conversion is inserted automatically,
// because tensor format != model input format
model = ppp.build();

    // No preprocessing related interfaces provided by C API 1.0

API 2.0

Python

ppp = PrePostProcessor(model)
input = ppp.input(tensor_name)
input.tensor().set_layout(Layout('NCHW')).set_element_type(Type.u8)
input.model().set_layout(Layout('NCHW'))
# layout and precision conversion is inserted automatically,
# because tensor format != model input format
model = ppp.build()

C++

ov::preprocess::PrePostProcessor ppp(model);
ov::preprocess::InputInfo& input = ppp.input(tensor_name);
input.tensor().set_layout("NHWC").set_element_type(ov::element::u8);
input.model().set_layout("NCHW");
// layout and precision conversion is inserted automatically,
// because tensor format != model input format
model = ppp.build();

    ov_preprocess_prepostprocessor_t* preprocess = NULL;
    ov_preprocess_input_info_t* input_info = NULL;
    ov_preprocess_input_tensor_info_t* input_tensor_info = NULL;
    ov_layout_t* layout_nhwc = NULL;
    ov_preprocess_input_model_info_t* input_model = NULL;
    ov_layout_t* layout_nchw = NULL;
    ov_model_t* new_model = NULL;

    ov_preprocess_prepostprocessor_create(model, &preprocess);
    ov_preprocess_prepostprocessor_get_input_info_by_name(preprocess, tensor_name, &input_info);
    ov_preprocess_input_info_get_tensor_info(input_info, &input_tensor_info);

    ov_layout_create("NHWC", &layout_nhwc);
    ov_preprocess_input_tensor_info_set_layout(input_tensor_info, layout_nhwc);
    ov_preprocess_input_tensor_info_set_element_type(input_tensor_info, ov_element_type_e::U8);

    ov_preprocess_input_info_get_model_info(input_info, &input_model);
    ov_layout_create("NCHW", &layout_nchw);
    ov_preprocess_input_model_info_set_layout(input_model, layout_nchw);
    // layout and precision conversion is inserted automatically,
    // because tensor format != model input format
    ov_preprocess_prepostprocessor_build(preprocess, &new_model);

    ov_layout_free(layout_nchw);
    ov_layout_free(layout_nhwc);
    ov_model_free(new_model);
    ov_preprocess_input_model_info_free(input_model);
    ov_preprocess_input_tensor_info_free(input_tensor_info);
    ov_preprocess_input_info_free(input_info);
    ov_preprocess_prepostprocessor_free(preprocess);

使用图像缩放¶

推理引擎 API

Python

ppp = PrePostProcessor(model)
input = ppp.input(tensor_name)
# need to specify H and W dimensions in model, others are not important
input.model().set_layout(Layout('??HW'))
# scale to model shape
input.preprocess().resize(ResizeAlgorithm.RESIZE_LINEAR, 448, 448)
# and insert operations to the model
model = ppp.build()

C++

ov::preprocess::PrePostProcessor ppp(model);
ov::preprocess::InputInfo& input = ppp.input(tensor_name);
// scale from the specified tensor size
input.tensor().set_spatial_static_shape(448, 448);
// need to specify H and W dimensions in model, others are not important
input.model().set_layout("??HW");
// scale to model shape
input.preprocess().resize(ov::preprocess::ResizeAlgorithm::RESIZE_LINEAR);
// and insert operations to the model
model = ppp.build();

    // No preprocessing related interfaces provided by C API 1.0

API 2.0

Python

ppp = PrePostProcessor(model)
input = ppp.input(tensor_name)
# need to specify H and W dimensions in model, others are not important
input.model().set_layout(Layout('??HW'))
# scale to model shape
input.preprocess().resize(ResizeAlgorithm.RESIZE_LINEAR, 448, 448)
# and insert operations to the model
model = ppp.build()

C++

ov::preprocess::PrePostProcessor ppp(model);
ov::preprocess::InputInfo& input = ppp.input(tensor_name);
// scale from the specified tensor size
input.tensor().set_spatial_static_shape(448, 448);
// need to specify H and W dimensions in model, others are not important
input.model().set_layout("??HW");
// scale to model shape
input.preprocess().resize(ov::preprocess::ResizeAlgorithm::RESIZE_LINEAR);
// and insert operations to the model
model = ppp.build();

    ov_preprocess_prepostprocessor_t* preprocess = NULL;
    ov_preprocess_input_info_t* input_info = NULL;
    ov_preprocess_input_tensor_info_t* input_tensor_info = NULL;
    ov_preprocess_input_model_info_t* input_model = NULL;
    ov_layout_t* layout = NULL;
    ov_preprocess_preprocess_steps_t* input_process = NULL;
    ov_model_t* new_model = NULL;

    ov_preprocess_prepostprocessor_create(model, &preprocess);
    ov_preprocess_prepostprocessor_get_input_info_by_name(preprocess, tensor_name, &input_info);
    ov_preprocess_input_info_get_tensor_info(input_info, &input_tensor_info);
    // scale from the specified tensor size
    ov_preprocess_input_tensor_info_set_spatial_static_shape(input_tensor_info, 448, 448);
    // need to specify H and W dimensions in model, others are not important
    ov_preprocess_input_info_get_model_info(input_info, &input_model);
    ov_layout_create("??HW", &layout);
    ov_preprocess_input_model_info_set_layout(input_model, layout);
    // scale to model shape
    ov_preprocess_input_info_get_preprocess_steps(input_info, &input_process);
    ov_preprocess_preprocess_steps_resize(input_process, ov_preprocess_resize_algorithm_e::RESIZE_LINEAR);
    // and insert operations to the model
    ov_preprocess_prepostprocessor_build(preprocess, &new_model);

    ov_layout_free(layout);
    ov_preprocess_preprocess_steps_free(input_process);
    ov_preprocess_input_model_info_free(input_model);
    ov_preprocess_input_tensor_info_free(input_tensor_info);
    ov_preprocess_input_info_free(input_info);
    ov_model_free(new_model);
    ov_preprocess_prepostprocessor_free(preprocess);

转换颜色空间¶

推理引擎 API

Python

ppp = PrePostProcessor(model)
input = ppp.input(tensor_name)
input.tensor().set_color_format(ColorFormat.NV12_TWO_PLANES)
# add NV12 to BGR conversion
input.preprocess().convert_color(ColorFormat.BGR)
# and insert operations to the model
model = ppp.build()

C++

ov::preprocess::PrePostProcessor ppp(model);
ov::preprocess::InputInfo& input = ppp.input(tensor_name);
input.tensor().set_color_format(ov::preprocess::ColorFormat::NV12_TWO_PLANES);
// add NV12 to BGR conversion
input.preprocess().convert_color(ov::preprocess::ColorFormat::BGR);
// and insert operations to the model
model = ppp.build();

    ov_preprocess_prepostprocessor_t* preprocess = NULL;
    ov_preprocess_input_info_t* input_info = NULL;
    ov_preprocess_input_tensor_info_t* input_tensor_info = NULL;
    ov_preprocess_preprocess_steps_t* input_process = NULL;
    ov_model_t* new_model = NULL;

    ov_preprocess_prepostprocessor_create(model, &preprocess);
    ov_preprocess_prepostprocessor_get_input_info_by_name(preprocess, tensor_name, &input_info);
    ov_preprocess_input_info_get_tensor_info(input_info, &input_tensor_info);
    ov_preprocess_input_tensor_info_set_color_format(input_tensor_info, ov_color_format_e::NV12_TWO_PLANES);
    // add NV12 to BGR conversion
    ov_preprocess_input_info_get_preprocess_steps(input_info, &input_process);
    ov_preprocess_preprocess_steps_convert_color(input_process, ov_color_format_e::BGR);
    // and insert operations to the model
    ov_preprocess_prepostprocessor_build(preprocess, &new_model);

    ov_preprocess_input_tensor_info_free(input_tensor_info);
    ov_preprocess_preprocess_steps_free(input_process);
    ov_preprocess_input_info_free(input_info);
    ov_preprocess_prepostprocessor_free(preprocess);
    ov_model_free(new_model);