Quantize Speech Recognition Models using NNCF PTQ API

This tutorial is also available as a Jupyter notebook that can be cloned directly from GitHub. See the installation guide for instructions to run this tutorial locally on Windows, Linux or macOS. To run without installing anything, click the “Open in Colab” button.

Google Colab Github

This tutorial demonstrates how to use the NNCF (Neural Network Compression Framework) 8-bit quantization in post-training mode (without the fine-tuning pipeline) to optimize the speech recognition model, known as Data2Vec for the high-speed inference via OpenVINO™ Toolkit. This notebook uses a fine-tuned data2vec-audio-base-960h PyTorch model trained on the LibriSpeech ASR corpus. The tutorial is designed to be extendable to custom models and datasets. It consists of the following steps:

  • Download and prepare model.

  • Define data loading and accuracy validation functionality.

  • Prepare the model for quantization and quantize.

  • Compare performance of the original and quantized models.

  • Compare Accuracy of the Original and Quantized Models.

Table of contents:

Download and prepare model

data2vec is a framework for self-supervised representation learning for images, speech, and text as described in data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language (Baevski et al., 2022). The algorithm uses the same learning mechanism for different modalities.

pre-trained pipeline

pre-trained pipeline

In our case, we will use data2vec-audio-base-960h model, which was finetuned on 960 hours of audio from LibriSpeech Automatic Speech Recognition corpus and distributed as part of HuggingFace transformers.

Obtain Pytorch model representation

For instantiating PyTorch model class, we should use Data2VecAudioForCTC.from_pretrained method with providing model ID for downloading from HuggingFace hub. Model weights and configuration files will be downloaded automatically in first time usage. Keep in mind that downloading the files can take several minutes and depends on your internet connection.

Additionally, we can create processor class which is responsible for model specific pre- and post-processing steps.

!pip install -q "openvino-dev>=2023.0.0" "nncf>=2.5.0"
!pip install -q soundfile librosa transformers onnx
from transformers import Wav2Vec2Processor, Data2VecAudioForCTC

processor = Wav2Vec2Processor.from_pretrained("facebook/data2vec-audio-base-960h")
model = Data2VecAudioForCTC.from_pretrained("facebook/data2vec-audio-base-960h")

Convert model to OpenVINO Intermediate Representation

from pathlib import Path
# Set model directory
MODEL_DIR = Path("model")
MODEL_DIR.mkdir(exist_ok=True)
from openvino.tools import mo
from openvino.runtime import serialize, Core
import torch

core = Core()

BATCH_SIZE = 1
MAX_SEQ_LENGTH = 30480


def export_model_to_onnx(model, path):
    # switch model to evaluation mode
    model.eval()
    # disallow gradient propagation for reducing memory during export
    with torch.no_grad():
        # define dummy input with specific shape
        default_input = torch.zeros([1, MAX_SEQ_LENGTH], dtype=torch.float)
        inputs = {
            "inputs": default_input
        }

        # define names for dynamic dimentions
        symbolic_names = {0: "batch_size", 1: "sequence_len"}
        # export model
        torch.onnx.export(
            model,
            (inputs["inputs"]),
            path,
            opset_version=11,
            input_names=["inputs"],
            output_names=["logits"],
            dynamic_axes={
                "inputs": symbolic_names,
                "logits": symbolic_names,
            },
        )
        print("ONNX model saved to {}".format(path))


onnx_model_path = MODEL_DIR / "data2vec-audo-base.onnx"
ir_model_path = onnx_model_path.with_suffix('.xml')

if not ir_model_path.exists():
    if not onnx_model_path.exists():
        export_model_to_onnx(model, onnx_model_path)
    ov_model = mo.convert_model(onnx_model_path, compress_to_fp16=True)
    serialize(ov_model, str(ir_model_path))
    print("IR model saved to {}".format(ir_model_path))
else:
    print("Read IR model from {}".format(ir_model_path))
    ov_model = core.read_model(ir_model_path)
Read IR model from model/data2vec-audo-base.xml

Prepare inference data

For demonstration purposes, we will use short dummy version of LibriSpeech dataset - patrickvonplaten/librispeech_asr_dummy to speed up model evaluation. Model accuracy can be different from reported in the paper. For reproducing original accuracy, use librispeech_asr dataset.

!pip install -q datasets "torchmetrics>=0.11.0"
from datasets import load_dataset

ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation")


# define preprocessing function for converting audio to input values for model
def map_to_input(batch):
    preprocessed_signal = processor(batch["audio"]["array"], return_tensors="pt", padding="longest", sampling_rate=batch['audio']['sampling_rate'])
    input_values = preprocessed_signal.input_values
    batch['input_values'] = input_values
    return batch


# apply preprocessing function to dataset and remove audio column, to save memory as we do not need it anymore
dataset = ds.map(map_to_input, batched=False, remove_columns=["audio"])

test_sample = ds[0]["audio"]
Found cached dataset librispeech_asr_dummy (/home/adrian/.cache/huggingface/datasets/patrickvonplaten___librispeech_asr_dummy/clean/2.1.0/f2c70a4d03ab4410954901bde48c54b85ca1b7f9bf7d616e7e2a72b5ee6ddbfc)
Loading cached processed dataset at /home/adrian/.cache/huggingface/datasets/patrickvonplaten___librispeech_asr_dummy/clean/2.1.0/f2c70a4d03ab4410954901bde48c54b85ca1b7f9bf7d616e7e2a72b5ee6ddbfc/cache-4e0f4916cd205b24.arrow

Check model inference result

The code below is used for running model inference on a single sample from the dataset. It contains the following steps:

  • Get the input_values tensor as model input.

  • Run model inference and obtain logits.

  • Find logits ids with highest probability, using argmax.

  • Decode predicted token ids, using processor.

For reference, see the same function provided for OpenVINO model.

import numpy as np


# inference function for pytorch
def torch_infer(model, sample):
    logits = model(torch.Tensor(sample['input_values'])).logits
    # take argmax and decode
    predicted_ids = torch.argmax(logits, dim=-1)
    transcription = processor.batch_decode(predicted_ids)
    return transcription


# inference function for openvino
def ov_infer(model, sample):
    output = model.output(0)
    logits = model(np.array(sample['input_values']))[output]
    predicted_ids = np.argmax(logits, axis=-1)
    transcription = processor.batch_decode(torch.from_numpy(predicted_ids))
    return transcription
core = Core()

pt_transcription = torch_infer(model, dataset[0])
compiled_model = core.compile_model(ov_model)
ov_transcription = ov_infer(compiled_model, dataset[0])
import IPython.display as ipd

print(f"[Reference]:     {dataset[0]['text']}")
print(f"[PyTorch]:       {pt_transcription[0]}")
print(f"[OpenVINO FP16]: {ov_transcription[0]}")
ipd.Audio(test_sample["array"], rate=16000)
[Reference]:     MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL
[PyTorch]:       MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL
[OpenVINO FP16]: MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL

Validate model accuracy on dataset

For model accuracy evaluation, Word Error Rate metric can be used. Word Error Rate or WER is the ratio of errors in a transcript to the total words spoken. A lower WER in speech-to-text means better accuracy in recognizing speech.

For WER calculation, we will use torchmetrics library.

from torchmetrics import WordErrorRate
from tqdm.notebook import tqdm


def compute_wer(dataset, model, infer_fn):
    wer = WordErrorRate()
    for sample in tqdm(dataset):
        # run infer function on sample
        transcription = infer_fn(model, sample)
        # update metric on sample result
        wer.update(transcription, [sample['text']])
    # finalize metric calculation
    result = wer.compute()
    return result
pt_result = compute_wer(dataset, model, torch_infer)
ov_result = compute_wer(dataset, compiled_model, ov_infer)
0%|          | 0/73 [00:00<?, ?it/s]
0%|          | 0/73 [00:00<?, ?it/s]
print(f'[PyTorch]   Word Error Rate: {pt_result:.4f}')
print(f'[OpenVino]  Word Error Rate: {ov_result:.4f}')
[PyTorch]   Word Error Rate: 0.0383
[OpenVino]  Word Error Rate: 0.0383

Quantization

NNCF provides a suite of advanced algorithms for Neural Networks inference optimization in OpenVINO with minimal accuracy drop.

Create a quantized model from the pre-trained FP16 model and the calibration dataset. The optimization process contains the following steps:

  1. Create a Dataset for quantization.

2. Run nncf.quantize for getting an optimized model. The nncf.quantize function provides an interface for model quantization. It requires an instance of the OpenVINO Model and quantization dataset. Optionally, some additional parameters for the configuration quantization process (number of samples for quantization, preset, ignored scope, etc.) can be provided. For more accurate results, we should keep the operation in the postprocessing subgraph in floating point precision, using the ignored_scope parameter. advanced_parameters can be used to specify advanced quantization parameters for fine-tuning the quantization algorithm. In this tutorial we pass range estimator parameters for activations. For more information see Tune quantization parameters. 3. Serialize OpenVINO IR model using openvino.runtime.serialize function.

import nncf
from nncf.quantization.advanced_parameters import AdvancedQuantizationParameters
from nncf.quantization.range_estimator import AggregatorType
from nncf.quantization.range_estimator import RangeEstimatorParameters
from nncf.quantization.range_estimator import StatisticsCollectorParameters
from nncf.quantization.range_estimator import StatisticsType
from nncf.parameters import ModelType


def transform_fn(data_item):
    """
    Extract the model's input from the data item.
    The data item here is the data item that is returned from the data source per iteration.
    This function should be passed when the data item cannot be used as model's input.
    """
    return np.array(data_item["input_values"])


calibration_dataset = nncf.Dataset(dataset, transform_fn)


quantized_model = nncf.quantize(
    ov_model,
    calibration_dataset,
    model_type=ModelType.TRANSFORMER,  # specify additional transformer patterns in the model
    subset_size=len(dataset),
    ignored_scope=nncf.IgnoredScope(
        names=[
            "/data2vec_audio/encoder/layers.3/feed_forward/intermediate_dense/MatMul",
            "/data2vec_audio/feature_extractor/conv_layers.2/conv/Conv",
            "/data2vec_audio/encoder/layers.3/Add_1",
            "/data2vec_audio/encoder/layers.2/feed_forward/intermediate_dense/MatMul",
            "/data2vec_audio/feature_extractor/conv_layers.0/conv/Conv",
            "/data2vec_audio/encoder/layers.4/Add_1",
            "/data2vec_audio/encoder/layers.4/feed_forward/intermediate_dense/MatMul",
            "/data2vec_audio/encoder/layers.4/final_layer_norm/Div",
            "/data2vec_audio/encoder/layers.4/feed_forward/output_dense/MatMul",
            "/data2vec_audio/encoder/layers.8/attention/MatMul_1",
            "/data2vec_audio/feature_extractor/conv_layers.1/conv/Conv",
            "/data2vec_audio/encoder/layers.2/Add_1",
            "/data2vec_audio/feature_extractor/conv_layers.0/layer_norm/Div",
            "/data2vec_audio/encoder/layers.1/feed_forward/intermediate_dense/MatMul",
            "/data2vec_audio/encoder/layers.1/Add_1",
            "/data2vec_audio/feature_extractor/conv_layers.3/layer_norm/Div"
        ],
    ),
    advanced_parameters=AdvancedQuantizationParameters(
        activations_range_estimator_params=RangeEstimatorParameters(
            min=StatisticsCollectorParameters(
                statistics_type=StatisticsType.MIN,
                aggregator_type=AggregatorType.MIN
            ),
            max=StatisticsCollectorParameters(
                statistics_type=StatisticsType.QUANTILE,
                aggregator_type=AggregatorType.MEAN,
                quantile_outlier_prob=0.0001
            ),
        )
    )
)
INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
INFO:nncf:16 ignored nodes was found by name in the NNCFGraph
INFO:nncf:220 ignored nodes was found by types in the NNCFGraph
INFO:nncf:24 ignored nodes was found by name in the NNCFGraph
INFO:nncf:Not adding activation input quantizer for operation: 2 /data2vec_audio/feature_extractor/conv_layers.0/conv/Conv
INFO:nncf:Not adding activation input quantizer for operation: 4 /data2vec_audio/feature_extractor/conv_layers.0/layer_norm/Div
5 /data2vec_audio/feature_extractor/conv_layers.0/layer_norm/Mul
6 /data2vec_audio/feature_extractor/conv_layers.0/layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 9 /data2vec_audio/feature_extractor/conv_layers.1/conv/Conv
INFO:nncf:Not adding activation input quantizer for operation: 11 /data2vec_audio/feature_extractor/conv_layers.1/layer_norm/Div
12 /data2vec_audio/feature_extractor/conv_layers.1/layer_norm/Mul
13 /data2vec_audio/feature_extractor/conv_layers.1/layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 16 /data2vec_audio/feature_extractor/conv_layers.2/conv/Conv
INFO:nncf:Not adding activation input quantizer for operation: 18 /data2vec_audio/feature_extractor/conv_layers.2/layer_norm/Div
19 /data2vec_audio/feature_extractor/conv_layers.2/layer_norm/Mul
20 /data2vec_audio/feature_extractor/conv_layers.2/layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 25 /data2vec_audio/feature_extractor/conv_layers.3/layer_norm/Div
26 /data2vec_audio/feature_extractor/conv_layers.3/layer_norm/Mul
27 /data2vec_audio/feature_extractor/conv_layers.3/layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 32 /data2vec_audio/feature_extractor/conv_layers.4/layer_norm/Div
33 /data2vec_audio/feature_extractor/conv_layers.4/layer_norm/Mul
34 /data2vec_audio/feature_extractor/conv_layers.4/layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 39 /data2vec_audio/feature_extractor/conv_layers.5/layer_norm/Div
40 /data2vec_audio/feature_extractor/conv_layers.5/layer_norm/Mul
41 /data2vec_audio/feature_extractor/conv_layers.5/layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 46 /data2vec_audio/feature_extractor/conv_layers.6/layer_norm/Div
47 /data2vec_audio/feature_extractor/conv_layers.6/layer_norm/Mul
48 /data2vec_audio/feature_extractor/conv_layers.6/layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 52 /data2vec_audio/feature_projection/layer_norm/Div
53 /data2vec_audio/feature_projection/layer_norm/Mul
54 /data2vec_audio/feature_projection/layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 70 /data2vec_audio/encoder/pos_conv_embed/layers.0/layer_norm/Div
INFO:nncf:Not adding activation input quantizer for operation: 124 /data2vec_audio/encoder/pos_conv_embed/layers.1/layer_norm/Div
INFO:nncf:Not adding activation input quantizer for operation: 178 /data2vec_audio/encoder/pos_conv_embed/layers.2/layer_norm/Div
INFO:nncf:Not adding activation input quantizer for operation: 221 /data2vec_audio/encoder/pos_conv_embed/layers.3/layer_norm/Div
INFO:nncf:Not adding activation input quantizer for operation: 262 /data2vec_audio/encoder/pos_conv_embed/layers.4/layer_norm/Div
INFO:nncf:Not adding activation input quantizer for operation: 57 /data2vec_audio/encoder/Add
INFO:nncf:Not adding activation input quantizer for operation: 59 /data2vec_audio/encoder/layer_norm/Div
61 /data2vec_audio/encoder/layer_norm/Mul
63 /data2vec_audio/encoder/layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 65 /data2vec_audio/encoder/layers.0/Add
INFO:nncf:Not adding activation input quantizer for operation: 71 /data2vec_audio/encoder/layers.0/layer_norm/Div
83 /data2vec_audio/encoder/layers.0/layer_norm/Mul
94 /data2vec_audio/encoder/layers.0/layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 104 /data2vec_audio/encoder/layers.0/Add_1
INFO:nncf:Not adding activation input quantizer for operation: 115 /data2vec_audio/encoder/layers.0/final_layer_norm/Div
121 /data2vec_audio/encoder/layers.0/final_layer_norm/Mul
125 /data2vec_audio/encoder/layers.0/final_layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 128 /data2vec_audio/encoder/layers.1/Add
INFO:nncf:Not adding activation input quantizer for operation: 135 /data2vec_audio/encoder/layers.1/layer_norm/Div
147 /data2vec_audio/encoder/layers.1/layer_norm/Mul
158 /data2vec_audio/encoder/layers.1/layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 169 /data2vec_audio/encoder/layers.1/feed_forward/intermediate_dense/MatMul
180 /data2vec_audio/encoder/layers.1/feed_forward/intermediate_dense/Add

INFO:nncf:Not adding activation input quantizer for operation: 168 /data2vec_audio/encoder/layers.1/Add_1
INFO:nncf:Not adding activation input quantizer for operation: 179 /data2vec_audio/encoder/layers.1/final_layer_norm/Div
185 /data2vec_audio/encoder/layers.1/final_layer_norm/Mul
189 /data2vec_audio/encoder/layers.1/final_layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 192 /data2vec_audio/encoder/layers.2/Add
INFO:nncf:Not adding activation input quantizer for operation: 199 /data2vec_audio/encoder/layers.2/layer_norm/Div
211 /data2vec_audio/encoder/layers.2/layer_norm/Mul
222 /data2vec_audio/encoder/layers.2/layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 233 /data2vec_audio/encoder/layers.2/feed_forward/intermediate_dense/MatMul
244 /data2vec_audio/encoder/layers.2/feed_forward/intermediate_dense/Add

INFO:nncf:Not adding activation input quantizer for operation: 232 /data2vec_audio/encoder/layers.2/Add_1
INFO:nncf:Not adding activation input quantizer for operation: 243 /data2vec_audio/encoder/layers.2/final_layer_norm/Div
249 /data2vec_audio/encoder/layers.2/final_layer_norm/Mul
253 /data2vec_audio/encoder/layers.2/final_layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 256 /data2vec_audio/encoder/layers.3/Add
INFO:nncf:Not adding activation input quantizer for operation: 263 /data2vec_audio/encoder/layers.3/layer_norm/Div
275 /data2vec_audio/encoder/layers.3/layer_norm/Mul
286 /data2vec_audio/encoder/layers.3/layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 297 /data2vec_audio/encoder/layers.3/feed_forward/intermediate_dense/MatMul
307 /data2vec_audio/encoder/layers.3/feed_forward/intermediate_dense/Add

INFO:nncf:Not adding activation input quantizer for operation: 296 /data2vec_audio/encoder/layers.3/Add_1
INFO:nncf:Not adding activation input quantizer for operation: 306 /data2vec_audio/encoder/layers.3/final_layer_norm/Div
311 /data2vec_audio/encoder/layers.3/final_layer_norm/Mul
314 /data2vec_audio/encoder/layers.3/final_layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 316 /data2vec_audio/encoder/layers.4/Add
INFO:nncf:Not adding activation input quantizer for operation: 322 /data2vec_audio/encoder/layers.4/layer_norm/Div
333 /data2vec_audio/encoder/layers.4/layer_norm/Mul
343 /data2vec_audio/encoder/layers.4/layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 353 /data2vec_audio/encoder/layers.4/feed_forward/intermediate_dense/MatMul
363 /data2vec_audio/encoder/layers.4/feed_forward/intermediate_dense/Add

INFO:nncf:Not adding activation input quantizer for operation: 371 /data2vec_audio/encoder/layers.4/feed_forward/output_dense/MatMul
377 /data2vec_audio/encoder/layers.4/feed_forward/output_dense/Add

INFO:nncf:Not adding activation input quantizer for operation: 352 /data2vec_audio/encoder/layers.4/Add_1
INFO:nncf:Not adding activation input quantizer for operation: 362 /data2vec_audio/encoder/layers.4/final_layer_norm/Div
367 /data2vec_audio/encoder/layers.4/final_layer_norm/Mul
370 /data2vec_audio/encoder/layers.4/final_layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 372 /data2vec_audio/encoder/layers.5/Add
INFO:nncf:Not adding activation input quantizer for operation: 378 /data2vec_audio/encoder/layers.5/layer_norm/Div
389 /data2vec_audio/encoder/layers.5/layer_norm/Mul
399 /data2vec_audio/encoder/layers.5/layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 408 /data2vec_audio/encoder/layers.5/Add_1
INFO:nncf:Not adding activation input quantizer for operation: 418 /data2vec_audio/encoder/layers.5/final_layer_norm/Div
423 /data2vec_audio/encoder/layers.5/final_layer_norm/Mul
426 /data2vec_audio/encoder/layers.5/final_layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 428 /data2vec_audio/encoder/layers.6/Add
INFO:nncf:Not adding activation input quantizer for operation: 434 /data2vec_audio/encoder/layers.6/layer_norm/Div
445 /data2vec_audio/encoder/layers.6/layer_norm/Mul
455 /data2vec_audio/encoder/layers.6/layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 464 /data2vec_audio/encoder/layers.6/Add_1
INFO:nncf:Not adding activation input quantizer for operation: 474 /data2vec_audio/encoder/layers.6/final_layer_norm/Div
479 /data2vec_audio/encoder/layers.6/final_layer_norm/Mul
482 /data2vec_audio/encoder/layers.6/final_layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 484 /data2vec_audio/encoder/layers.7/Add
INFO:nncf:Not adding activation input quantizer for operation: 490 /data2vec_audio/encoder/layers.7/layer_norm/Div
501 /data2vec_audio/encoder/layers.7/layer_norm/Mul
511 /data2vec_audio/encoder/layers.7/layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 520 /data2vec_audio/encoder/layers.7/Add_1
INFO:nncf:Not adding activation input quantizer for operation: 530 /data2vec_audio/encoder/layers.7/final_layer_norm/Div
535 /data2vec_audio/encoder/layers.7/final_layer_norm/Mul
538 /data2vec_audio/encoder/layers.7/final_layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 540 /data2vec_audio/encoder/layers.8/Add
INFO:nncf:Not adding activation input quantizer for operation: 546 /data2vec_audio/encoder/layers.8/layer_norm/Div
557 /data2vec_audio/encoder/layers.8/layer_norm/Mul
567 /data2vec_audio/encoder/layers.8/layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 576 /data2vec_audio/encoder/layers.8/Add_1
INFO:nncf:Not adding activation input quantizer for operation: 586 /data2vec_audio/encoder/layers.8/final_layer_norm/Div
591 /data2vec_audio/encoder/layers.8/final_layer_norm/Mul
594 /data2vec_audio/encoder/layers.8/final_layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 596 /data2vec_audio/encoder/layers.9/Add
INFO:nncf:Not adding activation input quantizer for operation: 602 /data2vec_audio/encoder/layers.9/layer_norm/Div
613 /data2vec_audio/encoder/layers.9/layer_norm/Mul
623 /data2vec_audio/encoder/layers.9/layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 632 /data2vec_audio/encoder/layers.9/Add_1
INFO:nncf:Not adding activation input quantizer for operation: 642 /data2vec_audio/encoder/layers.9/final_layer_norm/Div
647 /data2vec_audio/encoder/layers.9/final_layer_norm/Mul
650 /data2vec_audio/encoder/layers.9/final_layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 652 /data2vec_audio/encoder/layers.10/Add
INFO:nncf:Not adding activation input quantizer for operation: 658 /data2vec_audio/encoder/layers.10/layer_norm/Div
669 /data2vec_audio/encoder/layers.10/layer_norm/Mul
679 /data2vec_audio/encoder/layers.10/layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 688 /data2vec_audio/encoder/layers.10/Add_1
INFO:nncf:Not adding activation input quantizer for operation: 698 /data2vec_audio/encoder/layers.10/final_layer_norm/Div
703 /data2vec_audio/encoder/layers.10/final_layer_norm/Mul
706 /data2vec_audio/encoder/layers.10/final_layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 708 /data2vec_audio/encoder/layers.11/Add
INFO:nncf:Not adding activation input quantizer for operation: 714 /data2vec_audio/encoder/layers.11/layer_norm/Div
725 /data2vec_audio/encoder/layers.11/layer_norm/Mul
735 /data2vec_audio/encoder/layers.11/layer_norm/Add_1

INFO:nncf:Not adding activation input quantizer for operation: 744 /data2vec_audio/encoder/layers.11/Add_1
INFO:nncf:Not adding activation input quantizer for operation: 754 /data2vec_audio/encoder/layers.11/final_layer_norm/Div
759 /data2vec_audio/encoder/layers.11/final_layer_norm/Mul
762 /data2vec_audio/encoder/layers.11/final_layer_norm/Add_1
Statistics collection: 100%|██████████| 73/73 [00:37<00:00,  1.93it/s]
Biases correction: 100%|██████████| 74/74 [00:16<00:00,  4.60it/s]

After quantization is finished, compressed model representation can be saved using serialize function.

MODEL_NAME = 'quantized_data2vec_base'
quantized_model_path = Path(f"{MODEL_NAME}_openvino_model/{MODEL_NAME}_quantized.xml")
serialize(quantized_model, str(quantized_model_path))

Check INT8 model inference result

INT8 model is the same in usage like the original one. We need to read it, using the core.read_model method and load on the device, using core.compile_model. After that, we can reuse the same ov_infer function for getting model inference result on test sample.

int8_compiled_model = core.compile_model(quantized_model)
transcription = ov_infer(int8_compiled_model, dataset[0])
print(f"[Reference]:     {dataset[0]['text']}")
print(f"[OpenVINO INT8]: {transcription[0]}")
ipd.Audio(test_sample["array"], rate=16000)
[Reference]:     MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL
[OpenVINO INT8]: MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL

Compare Performance of the Original and Quantized Models

Benchmark Tool is used to measure the inference performance of the FP16 and INT8 models.

Note

For more accurate performance, it is recommended to run benchmark_app in a terminal/command prompt after closing other applications. Run benchmark_app -m model.xml -d CPU to benchmark async inference on CPU for one minute. Change CPU to GPU to benchmark on GPU. Run benchmark_app --help to see an overview of all command-line options.

# Inference FP16 model (OpenVINO IR)
! benchmark_app -m $ir_model_path -shape [1,30480] -d CPU -api async -t 15
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0
[ INFO ]
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 34.19 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ]     inputs (node: inputs) : f32 / [...] / [?,?]
[ INFO ] Model outputs:
[ INFO ]     logits (node: logits) : f32 / [...] / [?,?,32]
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[ INFO ] Reshaping model: 'inputs': [1,30480]
[ INFO ] Reshape model took 14.25 ms
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ]     inputs (node: inputs) : f32 / [...] / [1,30480]
[ INFO ] Model outputs:
[ INFO ]     logits (node: logits) : f32 / [...] / [1,95,32]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 510.06 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ]   NETWORK_NAME: torch_jit
[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 6
[ INFO ]   NUM_STREAMS: 6
[ INFO ]   AFFINITY: Affinity.HYBRID_AWARE
[ INFO ]   INFERENCE_NUM_THREADS: 18
[ INFO ]   PERF_COUNT: False
[ INFO ]   INFERENCE_PRECISION_HINT: <Type: 'float32'>
[ INFO ]   PERFORMANCE_HINT: PerformanceMode.THROUGHPUT
[ INFO ]   EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE
[ INFO ]   PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ]   ENABLE_CPU_PINNING: True
[ INFO ]   SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
[ INFO ]   ENABLE_HYPER_THREADING: True
[ INFO ]   EXECUTION_DEVICES: ['CPU']
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'inputs'!. This input will be filled with random values!
[ INFO ] Fill input 'inputs' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 6 inference requests, limits: 15000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 110.84 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count:            360 iterations
[ INFO ] Duration:         15162.56 ms
[ INFO ] Latency:
[ INFO ]    Median:        219.94 ms
[ INFO ]    Average:       251.64 ms
[ INFO ]    Min:           154.25 ms
[ INFO ]    Max:           378.39 ms
[ INFO ] Throughput:   23.74 FPS
# Inference INT8 model (OpenVINO IR)
! benchmark_app -m $quantized_model_path -shape [1,30480] -d CPU -api async -t 15
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0
[ INFO ]
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 67.14 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ]     inputs (node: inputs) : f32 / [...] / [?,?]
[ INFO ] Model outputs:
[ INFO ]     logits (node: logits) : f32 / [...] / [?,?,32]
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[ INFO ] Reshaping model: 'inputs': [1,30480]
[ INFO ] Reshape model took 32.77 ms
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ]     inputs (node: inputs) : f32 / [...] / [1,30480]
[ INFO ] Model outputs:
[ INFO ]     logits (node: logits) : f32 / [...] / [1,95,32]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 731.37 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ]   NETWORK_NAME: torch_jit
[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 6
[ INFO ]   NUM_STREAMS: 6
[ INFO ]   AFFINITY: Affinity.HYBRID_AWARE
[ INFO ]   INFERENCE_NUM_THREADS: 18
[ INFO ]   PERF_COUNT: False
[ INFO ]   INFERENCE_PRECISION_HINT: <Type: 'float32'>
[ INFO ]   PERFORMANCE_HINT: PerformanceMode.THROUGHPUT
[ INFO ]   EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE
[ INFO ]   PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ]   ENABLE_CPU_PINNING: True
[ INFO ]   SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
[ INFO ]   ENABLE_HYPER_THREADING: True
[ INFO ]   EXECUTION_DEVICES: ['CPU']
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'inputs'!. This input will be filled with random values!
[ INFO ] Fill input 'inputs' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 6 inference requests, limits: 15000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 72.68 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count:            582 iterations
[ INFO ] Duration:         15220.80 ms
[ INFO ] Latency:
[ INFO ]    Median:        135.06 ms
[ INFO ]    Average:       156.14 ms
[ INFO ]    Min:           124.62 ms
[ INFO ]    Max:           232.68 ms
[ INFO ] Throughput:   38.24 FPS

Compare Accuracy of the Original and Quantized Models

Finally, calculate WER metric for the INT8 model representation and compare it with the FP16 result.

int8_ov_result = compute_wer(dataset, int8_compiled_model, ov_infer)
print(f'[OpenVino FP16] Word Error Rate: {ov_result:.4}')
print(f'[OpenVino INT8] Word Error Rate: {int8_ov_result:.4f}')
0%|          | 0/73 [00:00<?, ?it/s]
[OpenVino FP16] Word Error Rate: 0.03826
[OpenVino INT8] Word Error Rate: 0.0487