NOTE: With 2020.2 the extgen tool is removed from the package due to the changes in the extension mechanism. To create extensions, use the following guides: Extending Model Optimizer with New Primitives and Inference Engine Extensibility Mechanism.
OpenVINO™ toolkit provides the extgen
tool that facilitates creating Model Optimizer and Inference Engine extensions. The tool generates extension source files with stubs for the core functions. To get the workable extension, you should only add your implementation of these functions to the generated files.
To generate extension files, run the extgen
tool in one of the available modes:
new
command line option. For example: extgen
tool directory: <INSTALL_DIR>/deployment_tools/extension_generator/config.extgen.json.example
.To run the tool in the interactive mode, specify the following parameters:
mo-op
to generate a Model Optimizer operationmo-caffe-ext
to generate a Model Optimizer Caffe* extractormo-tf-ext
to generate a Model Optimizer TensorFlow* extractormo-mxnet-ext
to generates a Model Optimizer MXNet* extractorie-cpu-ext
to generate an Inference Engine CPU extensionie-gpu-ext
to generate an Inference Engine GPU extensionoutput_dir
to set an output directory. If not specified, the current directory is used by default.You can use any combination of the parameters to generate Model Optimizer and/or Inference Engine extension files. For example:
To generate Model Optimizer extension files, run the tool in the interactive mode with necessary parameters or in the silent mode with a configuration file. For example, to generate operation and extractor files for a Caffe model in the <output_directory>
in the interactive mode, run the following command:
The extension stub files are generated in the <output_dir>/user_mo_extensions
directory, which has the following structure:
/front
/caffe
- Folder with Caffe extractors/mxnet
- Folder with MXNet extractors/tf
- Folder with TensorFlow extractors/ops
- Folder with operation filesSpecific paths to the generated files appear on the screen. For example, for the Caffe* Proposal
layer, the files are <output_dir>/user_mo_extensions/front/caffe/proposal_ext.py
and <output_dir>/user_mo_extensions/ops/proposal.py
.
Usually, you can use an extractor file without changes. Exceptions are the cases when you want to transform parameters from a input file in the IR. In this case, you should add these transformations to the extract
method. Do not forget to add parameter names to the supported_attrs
and backend_attrs
methods to the operation file.
An operation file can be used without changes if your layer does not change the shape. Otherwise, you should implement the shape calculation in the <op_name>_infer
method. Also, you can add default values to the __init__
method. You can find more details in the Extending Model Optimizer with New Primitives.
To generate stub files for GPU and CPU Inference Engine extensions, run the tool and provide input information interactively or in the configuration file. For example, to generate an Inference Engine CPU extension files in the <output_directory>
in the interactive mode:
The extension stub files are generated in the <output_dir>/user_ie_extensions
directory.
For CPU, several files are generated in the cpu
subdirectory. You must change only <output_dir>/user_ie_extensions/cpu/ext_<op_name>.cpp
with adding inference implementation.
For GPU, <op_name>.cl
and <op_name>.xml
are generated in the gpu
subdirectory. You must update both:
<op_name>.cl
, implement an OpenCL™ kernel to infer the model.<op_name>.xml
, fill information about input/output buffers and worksize for your kernel.More details about implementing Inference Engine extensions see in [DEPRECATED] Inference Engine Kernels Extensibility.
This section provides step-by-step examples of extension generation for conversion Caffe* and TensorFlow* models. The Caffe* example describes the Inference Engine extension creation. The TensorFlow* example uses existing Inference Engine operation. If you need Inference Engine extension to infer a TensorFlow-based model, look at steps 6-7 in Caffe* example, because Inference Engine extension generation does not depend on the framework is it based on.
This section provides a sample for generating and implementing Model Optimizer and Inference Engine custom layer extensions for the Proposal
layer of a Caffe* example model. The model (.prototxt
and .caffemodel
) used in the example is described in the Extending Model Optimizer with New Primitives chapter.
extgen
: extgen.py
file with the following parameters to generate extension stub files: ./user_mo_extensions
and ./user_ie_extensions
directories, which have the following structure:/user_mo_extensions
__init__.py
/front
/caffe
__init__.py
proposallayer_ext.py
/mxnet
__init__.py
/ops
__init__.py
proposal.py
/user_ie_extensions
/cpu
CMakeLists.txt
ext_base.cpp
ext_base.hpp
ext_lists.cpp
ext_lists.hpp
ext_proposal.cpp
/gpu
Implement extension functions in the generated files:
a. Extractor proposallayer_ext.py
can be used without changes.
b. Add the shape calculation logic to the operation file proposal.py
. According to IR catalog, the 'Proposal' layer shape dynamically depends on the post_nms_topn parameter.
Add this parameter with the default value in __init__
method:
Then add supported attributes in the method supported_attrs
:
Now add shape calculation in infer
function:
Proposal
layer inference to the execute
method in the ext_proposal.cpp
file. You can find sample code for this extension in the <INSTALL_DIR>/deployment_tools/inference_engine/samples/extension/ext_proposal.cpp
file. For more information about implementation of Inference Engine extensions, refer to [DEPRECATED] Inference Engine Kernels Extensibility.Build a library with CPU extension to use it with the Inference Engine:
a. Create a new build directory:
b. Go to the created build directory:
c. Set the environment variables:
d. Run CMake to generate the Make files:
e. Build the library:
This section provides an example for generating and implementing Model Optimizer extension on TensorFlow* example model.
If you already have a model with unrecognized operation, you can omit Model Preparation and go to Extension Generation chapter.
In example the Pooling
layer will be used to illustrate extension generation. ModelOptimizer already supports this layer but we will remove it and show how it can be created with extgen
tool. This process described in Model Preparation chapter.
Operation and Inference engine extension generation does not depend on framework and was demonstrated already in Caffe example, so here only TensorFlow extractor generation will be done.
ResNet-50
from TensorFlow* Model Zoo. Follow the instructions in Convert Model From TensorFlow* to prepare the model for converting.ResNet-50
model, it will be converted successfully. To demonstrate extension generation, remove the existing implementation of Pooling
layer from the Model Optimizer: Now the sample model is ready for extension generation.
extgen.py
file with the following parameters to generate extension stub files : user_mo_extensions
directory, which has the following structure:/user_mo_extensions
__init__.py
/front
/caffe
__init__.py
/mxnet
__init__.py
/tf
__init__.py
pooling_ext.py
/ops
__init__.py
Implement extension functions in the generated files:
a. The extractor pooling_ext.py
requires additional attribute conversion. Several attributes should be initialized by constants, real values will be calculated during inference. These changes are needed because we use existing operation that was written for several frameworks.
Conversion should finish successfully.