Post-Training Optimization Toolkit

Introduction

Starting with the 2020.1 version, OpenVINO™ toolkit delivers the Post-Training Optimization Tool designed to accelerate the inference of DL models by converting them into a more hardware-friendly representation by applying specific methods that do not require re-training, for example, post-training quantization. For more details about the low-precision flow in OpenVINO™, refer to the Low Precision Optimization Guide.

Post-Training Optimization Toolkit includes standalone command-line tool and Python* API that provide the following key features:

• Two supported post-training quantization algorithms: fast DefaultQuantization and precise AccuracyAwareQuantization. as well as multiple experimental methods including global optimization.
• Symmetric and asymmetric quantization schemes. For more details, see the Quantization section.
• Compression for different hardware targets such as CPU, GPU.
• Per-channel quantization for Convolutional and Fully-Connected layers.
• Multiple domains: Computer Vision, Recommendation Systems.
• Ability to implement custom calibration pipeline via supported API.

Prerequisites

• Python* 3.5 or higher
• OpenVINO 2020.1 or higher

Install Post-Training Optimization Toolkit

The toolkit is distributed as a part of the OpenVINO release package and available after installation from the <INSTALL_DIR>/deployment_tools/tools/post_training_optimization_toolkit. It's recommended to create a separate Python* environment before installing the OpenVINO components. To install the toolkit into your environment, follow the steps below:

1. Before using optimization toolkit, install the AccuracyChecker component:
   1. Navigate to <INSTALL_DIR>/deployment_tools/open_model_zoo/tools/accuracy_checker.
   2. Run the setup.py script:

      python3 setup.py install

2. Install the toolkit:
   1. Navigate to <INSTALL_DIR>/deployment_tools/tools/post_training_optimization_toolkit.
   2. Run the setup.py script:

      python3 setup.py install

Now the toolkit is available in the command line by the pot alias.

Use Post-Training Optimization Command-line Tool

The toolkit works with pre-trained models in the OpenVINO Intermediate Representation (IR) format, which means that you need to convert the model using Model Optimizer before running the optimization tool. In addition, it's highly recommended to make sure that the model can be successfully inferred and achieves the similar accuracy numbers as the reference model from the original framework. In order to do this, the AccuracyChecker tool can be used.

To run the command-line Post-Training Optimization Tool:

1. Activate OpenVINO environment in the command-line shell running the following script:

   source /opt/intel/openvino/bin/setupvars.sh

2. Prepare a configuration file for the tool using the examples in the configs folder. To simplify this step, an AccuracyChecker configuration file for the full-precision model can be used and referred.
3. Launch the command-line tool with the configuration file running the following command:

   pot -c <path_to_config_file>

   For all available usage options, use the -h, --help arguments or refer to the Command-Line Arguments below.
4. By default, the results are dumped into the separate output subfolder inside the results folder that is created in the same directory where the tool is run from. Use the -e option to evaluate the accuracy directly from the tool.

Command-Line Arguments

The following command-line options are available to run the tool:

Argument	Description
-h, --help	Optional. Show help message and exit.
-c CONFIG, --config CONFIG	Path to a config file with task/model-specific parameters.
-e, --evaluate	Optional. Evaluate model on the whole dataset after optimization.
--output-dir OUTPUT_DIR	Optional. A directory where results are saved. Default: ./results.
-sm, --save-model	Optional. Save the original (FP32) model.
-d, --direct-dump	Optional. Save results directly to output directory without additional subfolders.
--log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG}	Optional. Log level to print. INFO is by default.