Inference Engine with low-precision 8-bit integer inference is in a feature preview and requires the following prerequisites to be satisfied:
The 8-bit inference feature was validated on the following topologies:
A lot of investigation was made in the field of deep learning with the idea of using low precision computations during inference in order to boost deep learning pipelines and gather higher performance. For example, one of the popular approaches is to shrink the precision of activations and weights values from fp32
precision to smaller ones, for example, to fp11
or int8
. For more information about this approach, refer to Brief History of Lower Precision in Deep Learning section in this whitepaper.
8-bit computations (referred to as int8
) offer better performance compared to the results of inference in higher precision (for example, fp32
), because they allow to load more data into a single processor instruction. Usually the cost for significant boost is a reduced accuracy. However, it is proved that the drop in accuracy can be negligible and depends on task requirements, so that the application engineer can set up the maximum accuracy drop that is acceptable.
Current Inference Engine solution for low-precision inference uses Intel MKL-DNN, which supports inference of the following layers in 8-bit integer computation mode:
This means that 8-bit inference can only be performed with the CPU plugin on the layers listed above. All other layers are executed in the format supported by the CPU plugin: 32-bit floating point format (fp32
).
For 8-bit integer computations, the original model (or its Intermediate Representation) must be in the fp32
format. In order to perform calculation of layers in the int8
format, the input data (input blob) and weights of the given layer (also biases and/or other blobs of the layer) must be quantized - transitioned from fp32
to int8
format. The quantization process converts model input into a lower-precision format. The precision and accuracy factors are specified by the scale and rounding-mode respectively. Read more about mathematical computations under the hood in the white paper.
8-bit inference pipeline includes two stages (also refer to the figure below):
One of the vital components for successful data quantization is a set of scale factors for each layer that supports 8-bit computations. These scales are obtained from statistics of layers activations collected by the Calibration Tool on a calibration dataset. The calibration dataset contains images and can be a subset of the validation set. A small fraction of images from validation dataset (1-5%) is enough to create a calibration dataset. For more information on the dataset preparation, refer to the Validation Application.
To calibrate a model, the calibration tool preforms the following steps:
fp32
inference. Note that accuracy metric depends on the type of the calibrated model. For classification networks, top-1 metric is used; for object detection models, mAP metric is used.fp32
mode. After eliminating one layer, the Calibration Tool computes the accuracy of this configuration. Until the resulting accuracy satisfies the required level with respect to the accepted accuracy drop delta (which equals 1% by default), the tool continues switching layers back to fp32
computations in the order defined in the step 3. However, calibration of the model with all layers returned to fp32
computations is meaningless, so that this plays a role of hard stop of the whole calibration process.When the calibration completes, the tool writes the resulting statistics and the modified Intermediate Representation (IR) to the .xml
file. The tool does not change the IR structure, so the layers hierarchy is the same. However, the layers that are chosen to be executed in 8-bit format are marked with the appropriate profile attribute, and their statistics is stored at the end of the .xml
file.
When you pass the calibrated IR to the CPU plugin, the plugin automatically recognizes it as calibrated and performs the 8-bit inference. At the same time, other plugins do not support 8-bit inference, so if you pass the calibrated model to them, statistics and additional attributes are ignored and the model is inferred in the precision that this plugin supports.
This is the second stage of the 8-bit integer inference. After you load the calibrated model IR to the CPU plugin, it performs quantization for 8-bit inference:
int8
data type and normalize output layers to unsigned 8-bit integer type, to signed 8-bit integer type, or to 32-bit floating data typeInformation about layer precision is stored in the performance counters that are available from the Inference Engine API. The layers have the following marks:
I8
for layers that had 8-bit data type input and were computed in 8-bit precisionFP32
for layers computed in 32-bit precisionFor example, the performance counters table for the Inception model can look as follows:
The execType
column of the table includes inference primitives with specific suffixes.