INT8 vs FP32 Comparison on Select Networks and Platforms¶
The table below illustrates the speed-up factor for the performance gain by switching from an FP32 representation of an OpenVINO™ supported model to its INT8 representation.
| Intel® Core™ i7-8700T |
Intel® Core™ i7-1185G7 |
Intel® Xeon® W-1290P |
Intel® Xeon® Platinum 8270 |
||
|---|---|---|---|---|---|
| OpenVINO benchmark model name |
Dataset | Throughput speed-up FP16-INT8 vs FP32 | |||
| bert-large- uncased-whole-word- masking-squad-0001 |
SQuAD | 1.6 | 3.1 | 1.5 | 2.5 |
| brain-tumor- segmentation- 0001-MXNET |
BraTS | 1.6 | 2.0 | 1.8 | 1.8 |
| deeplabv3-TF | VOC 2012 Segmentation |
1.9 | 3.0 | 2.8 | 3.1 |
| densenet-121-TF | ImageNet | 1.8 | 3.5 | 1.9 | 3.8 |
| facenet- 20180408- 102900-TF |
LFW | 2.1 | 3.6 | 2.2 | 3.7 |
| faster_rcnn_ resnet50_coco-TF |
MS COCO | 1.9 | 3.7 | 2.0 | 3.4 |
| inception-v3-TF | ImageNet | 1.9 | 3.8 | 2.0 | 4.1 |
| mobilenet- ssd-CF |
VOC2012 | 1.6 | 3.1 | 1.9 | 3.6 |
| mobilenet-v2-1.0- 224-TF |
ImageNet | 1.5 | 2.4 | 1.8 | 3.9 |
| mobilenet-v2- pytorch |
ImageNet | 1.7 | 2.4 | 1.9 | 4.0 |
| resnet-18- pytorch |
ImageNet | 1.9 | 3.7 | 2.1 | 4.2 |
| resnet-50- pytorch |
ImageNet | 1.9 | 3.6 | 2.0 | 3.9 |
| resnet-50- TF |
ImageNet | 1.9 | 3.6 | 2.0 | 3.9 |
| squeezenet1.1- CF |
ImageNet | 1.7 | 3.2 | 1.8 | 3.4 |
| ssd_mobilenet_ v1_coco-tf |
VOC2012 | 1.8 | 3.1 | 2.0 | 3.6 |
| ssd300-CF | MS COCO | 1.8 | 4.2 | 1.9 | 3.9 |
| ssdlite_ mobilenet_ v2-TF |
MS COCO | 1.7 | 2.5 | 2.4 | 3.5 |
| yolo_v4-TF | MS COCO | 1.9 | 3.6 | 2.0 | 3.4 |
| unet-camvid-onnx-0001 | MS COCO | 1.7 | 3.9 | 1.7 | 3.7 |
| ssd-resnet34- 1200-onnx |
MS COCO | 1.7 | 4.0 | 1.7 | 3.4 |
| googlenet-v4-tf | ImageNet | 1.9 | 3.9 | 2.0 | 4.1 |
| vgg19-caffe | ImageNet | 1.9 | 4.7 | 2.0 | 4.5 |
| yolo-v3-tiny-tf | MS COCO | 1.7 | 3.4 | 1.9 | 3.5 |
The following table shows the absolute accuracy drop that is calculated as the difference in accuracy between the FP32 representation of a model and its INT8 representation.
| Intel® Core™ i9-10920X CPU @ 3.50GHZ (VNNI) |
Intel® Core™ i9-9820X CPU @ 3.30GHz (AVX512) |
Intel® Core™ i7-6700K CPU @ 4.0GHz (AVX2) |
Intel® Core™ i7-1185G7 CPU @ 4.0GHz (TGL VNNI) |
|||
|---|---|---|---|---|---|---|
| OpenVINO Benchmark Model Name |
Dataset | Metric Name | Absolute Accuracy Drop, % | |||
| bert-large-uncased-whole-word-masking-squad-0001 | SQuAD | F1 | 0.62 | 0.71 | 0.62 | 0.62 |
| brain-tumor- segmentation- 0001-MXNET |
BraTS | Dice-index@ Mean@ Overall Tumor |
0.08 | 0.10 | 0.10 | 0.08 |
| deeplabv3-TF | VOC 2012 Segmentation |
mean_iou | 0.09 | 0.41 | 0.41 | 0.09 |
| densenet-121-TF | ImageNet | acc@top-1 | 0.49 | 0.56 | 0.56 | 0.49 |
| facenet- 20180408- 102900-TF |
LFW | pairwise_ accuracy _subsets |
0.05 | 0.12 | 0.12 | 0.05 |
| faster_rcnn_ resnet50_coco-TF |
MS COCO | coco_ precision |
0.09 | 0.09 | 0.09 | 0.09 |
| inception-v3-TF | ImageNet | acc@top-1 | 0.02 | 0.01 | 0.01 | 0.02 |
| mobilenet- ssd-CF |
VOC2012 | mAP | 0.06 | 0.04 | 0.04 | 0.06 |
| mobilenet-v2-1.0- 224-TF |
ImageNet | acc@top-1 | 0.40 | 0.76 | 0.76 | 0.40 |
| mobilenet-v2- PYTORCH |
ImageNet | acc@top-1 | 0.36 | 0.52 | 0.52 | 0.36 |
| resnet-18- pytorch |
ImageNet | acc@top-1 | 0.25 | 0.25 | 0.25 | 0.25 |
| resnet-50- PYTORCH |
ImageNet | acc@top-1 | 0.19 | 0.21 | 0.21 | 0.19 |
| resnet-50- TF |
ImageNet | acc@top-1 | 0.11 | 0.11 | 0.11 | 0.11 |
| squeezenet1.1- CF |
ImageNet | acc@top-1 | 0.64 | 0.66 | 0.66 | 0.64 |
| ssd_mobilenet_ v1_coco-tf |
VOC2012 | COCO mAp | 0.17 | 2.96 | 2.96 | 0.17 |
| ssd300-CF | MS COCO | COCO mAp | 0.18 | 3.06 | 3.06 | 0.18 |
| ssdlite_ mobilenet_ v2-TF |
MS COCO | COCO mAp | 0.11 | 0.43 | 0.43 | 0.11 |
| yolo_v4-TF | MS COCO | COCO mAp | 0.06 | 0.03 | 0.03 | 0.06 |
| unet-camvid- onnx-0001 |
MS COCO | COCO mAp | 0.29 | 0.29 | 0.31 | 0.29 |
| ssd-resnet34- 1200-onnx |
MS COCO | COCO mAp | 0.02 | 0.03 | 0.03 | 0.02 |
| googlenet-v4-tf | ImageNet | COCO mAp | 0.08 | 0.06 | 0.06 | 0.06 |
| vgg19-caffe | ImageNet | COCO mAp | 0.02 | 0.04 | 0.04 | 0.02 |
| yolo-v3-tiny-tf | MS COCO | COCO mAp | 0.02 | 0.6 | 0.6 | 0.02 |
For more complete information about performance and benchmark results, visit: www.intel.com/benchmarks and Optimization Notice. Legal Information.