Quantize Speech Recognition Models using NNCF PTQ API¶
This tutorial is also available as a Jupyter notebook that can be cloned directly from GitHub. See the installation guide for instructions to run this tutorial locally on Windows, Linux or macOS. To run without installing anything, click the “Open in Colab” button.
This tutorial demonstrates how to use the NNCF (Neural Network Compression Framework) 8-bit quantization in post-training mode (without the fine-tuning pipeline) to optimize the speech recognition model, known as Data2Vec for the high-speed inference via OpenVINO™ Toolkit. This notebook uses a fine-tuned data2vec-audio-base-960h PyTorch model trained on the LibriSpeech ASR corpus. The tutorial is designed to be extendable to custom models and datasets. It consists of the following steps:
Download and prepare model.
Define data loading and accuracy validation functionality.
Prepare the model for quantization and quantize.
Compare performance of the original and quantized models.
Compare Accuracy of the Original and Quantized Models.
Table of contents:
Download and prepare model¶
data2vec is a framework for self-supervised representation learning for images, speech, and text as described in data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language (Baevski et al., 2022). The algorithm uses the same learning mechanism for different modalities.
pre-trained pipeline¶
In our case, we will use data2vec-audio-base-960h model, which was
finetuned on 960 hours of audio from LibriSpeech Automatic Speech
Recognition corpus and distributed as part of HuggingFace transformers.
Obtain Pytorch model representation¶
For instantiating PyTorch model class, we should use
Data2VecAudioForCTC.from_pretrained method with providing model ID
for downloading from HuggingFace hub. Model weights and configuration
files will be downloaded automatically in first time usage. Keep in mind
that downloading the files can take several minutes and depends on your
internet connection.
Additionally, we can create processor class which is responsible for model specific pre- and post-processing steps.
%pip install -q "openvino>=2023.1.0" "nncf>=2.5.0"
%pip install -q datasets "torchmetrics>=0.11.0"
%pip install -q soundfile librosa transformers
from transformers import Wav2Vec2Processor, Data2VecAudioForCTC
processor = Wav2Vec2Processor.from_pretrained("facebook/data2vec-audio-base-960h")
model = Data2VecAudioForCTC.from_pretrained("facebook/data2vec-audio-base-960h")
2023-09-12 19:27:57.776647: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0. 2023-09-12 19:27:57.812053: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-09-12 19:27:58.411557: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Convert model to OpenVINO Intermediate Representation¶
from pathlib import Path
# Set model directory
MODEL_DIR = Path("model")
MODEL_DIR.mkdir(exist_ok=True)
import openvino as ov
import torch
core = ov.Core()
BATCH_SIZE = 1
MAX_SEQ_LENGTH = 30480
ir_model_path = MODEL_DIR / "data2vec-audo-base.xml"
if not ir_model_path.exists():
ov_model = ov.convert_model(model, example_input=torch.zeros([1, MAX_SEQ_LENGTH], dtype=torch.float))
ov.save_model(ov_model, str(ir_model_path))
print("IR model saved to {}".format(ir_model_path))
else:
print("Read IR model from {}".format(ir_model_path))
ov_model = core.read_model(ir_model_path)
Read IR model from model/data2vec-audo-base.xml
Prepare inference data¶
For demonstration purposes, we will use short dummy version of
LibriSpeech dataset - patrickvonplaten/librispeech_asr_dummy to
speed up model evaluation. Model accuracy can be different from reported
in the paper. For reproducing original accuracy, use librispeech_asr
dataset.
from datasets import load_dataset
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation")
# define preprocessing function for converting audio to input values for model
def map_to_input(batch):
preprocessed_signal = processor(batch["audio"]["array"], return_tensors="pt", padding="longest", sampling_rate=batch['audio']['sampling_rate'])
input_values = preprocessed_signal.input_values
batch['input_values'] = input_values
return batch
# apply preprocessing function to dataset and remove audio column, to save memory as we do not need it anymore
dataset = ds.map(map_to_input, batched=False, remove_columns=["audio"])
test_sample = ds[0]["audio"]
Found cached dataset librispeech_asr_dummy (/home/ea/.cache/huggingface/datasets/patrickvonplaten___librispeech_asr_dummy/clean/2.1.0/f2c70a4d03ab4410954901bde48c54b85ca1b7f9bf7d616e7e2a72b5ee6ddbfc)
Loading cached processed dataset at /home/ea/.cache/huggingface/datasets/patrickvonplaten___librispeech_asr_dummy/clean/2.1.0/f2c70a4d03ab4410954901bde48c54b85ca1b7f9bf7d616e7e2a72b5ee6ddbfc/cache-5282243604a7a526.arrow
Check model inference result¶
The code below is used for running model inference on a single sample from the dataset. It contains the following steps:
Get the input_values tensor as model input.
Run model inference and obtain logits.
Find logits ids with highest probability, using argmax.
Decode predicted token ids, using processor.
For reference, see the same function provided for OpenVINO model.
import numpy as np
# inference function for pytorch
def torch_infer(model, sample):
logits = model(torch.Tensor(sample['input_values'])).logits
# take argmax and decode
predicted_ids = torch.argmax(logits, dim=-1)
transcription = processor.batch_decode(predicted_ids)
return transcription
# inference function for openvino
def ov_infer(model, sample):
output = model.output(0)
logits = model(np.array(sample['input_values']))[output]
predicted_ids = np.argmax(logits, axis=-1)
transcription = processor.batch_decode(torch.from_numpy(predicted_ids))
return transcription
core = ov.Core()
pt_transcription = torch_infer(model, dataset[0])
compiled_model = core.compile_model(ov_model)
ov_transcription = ov_infer(compiled_model, dataset[0])
import IPython.display as ipd
print(f"[Reference]: {dataset[0]['text']}")
print(f"[PyTorch]: {pt_transcription[0]}")
print(f"[OpenVINO FP16]: {ov_transcription[0]}")
ipd.Audio(test_sample["array"], rate=16000)
[Reference]: BECAUSE YOU WERE SLEEPING INSTEAD OF CONQUERING THE LOVELY ROSE PRINCESS HAS BECOME A FIDDLE WITHOUT A BOW WHILE POOR SHAGGY SITS THERE A COOING DOVE
[PyTorch]: BECAUSE YOU WERE SLEEPING INSTEAD OF CONQUERING THE LOVELY RUSE PRINCESS HAS BECOME A FIDDLE WITHOUT A BOW A POOR SHAGGY SITS THERE ACCOOING DOVE
[OpenVINO FP16]: BECAUSE YOU WERE SLEEPING INSTEAD OF CONQUERING THE LOVELY RUSE PRINCESS HAS BECOME A FIDDLE WITHOUT A BOW A POOR SHAGGY SITS THERE ACCOOING DOVE
Validate model accuracy on dataset¶
For model accuracy evaluation, Word Error Rate metric can be used. Word Error Rate or WER is the ratio of errors in a transcript to the total words spoken. A lower WER in speech-to-text means better accuracy in recognizing speech.
For WER calculation, we will use torchmetrics library.
from torchmetrics import WordErrorRate
from tqdm.notebook import tqdm
def compute_wer(dataset, model, infer_fn):
wer = WordErrorRate()
for sample in tqdm(dataset):
# run infer function on sample
transcription = infer_fn(model, sample)
# update metric on sample result
wer.update(transcription, [sample['text']])
# finalize metric calculation
result = wer.compute()
return result
pt_result = compute_wer(dataset, model, torch_infer)
ov_result = compute_wer(dataset, compiled_model, ov_infer)
0%| | 0/73 [00:00<?, ?it/s]
0%| | 0/73 [00:00<?, ?it/s]
print(f'[PyTorch] Word Error Rate: {pt_result:.4f}')
print(f'[OpenVino] Word Error Rate: {ov_result:.4f}')
[PyTorch] Word Error Rate: 0.0383
[OpenVino] Word Error Rate: 0.0383
Quantization¶
NNCF provides a suite of advanced algorithms for Neural Networks inference optimization in OpenVINO with minimal accuracy drop.
Create a quantized model from the pre-trained FP16 model and the
calibration dataset. The optimization process contains the following
steps:
Create a Dataset for quantization.
Run
nncf.quantizefor getting an optimized model. Thenncf.quantizefunction provides an interface for model quantization. It requires an instance of the OpenVINO Model and quantization dataset. Optionally, some additional parameters for the configuration quantization process (number of samples for quantization, preset, ignored scope, etc.) can be provided. For more accurate results, we should keep the operation in the postprocessing subgraph in floating point precision, using theignored_scopeparameter. For more information see Tune quantization parameters.Serialize OpenVINO IR model using
ov.save_modelfunction.
import nncf
from nncf.parameters import ModelType
def transform_fn(data_item):
"""
Extract the model's input from the data item.
The data item here is the data item that is returned from the data source per iteration.
This function should be passed when the data item cannot be used as model's input.
"""
return np.array(data_item["input_values"])
calibration_dataset = nncf.Dataset(dataset, transform_fn)
quantized_model = nncf.quantize(
ov_model,
calibration_dataset,
model_type=ModelType.TRANSFORMER, # specify additional transformer patterns in the model
subset_size=len(dataset),
ignored_scope=nncf.IgnoredScope(
names=[
"__module.data2vec_audio.feature_extractor.conv_layers.1.conv/aten::_convolution/Convolution_33",
"__module.data2vec_audio.feature_extractor.conv_layers.0.conv/aten::_convolution/Convolution_3",
"__module.data2vec_audio.encoder.layers.6.feed_forward.output_dense/aten::linear/MatMul_1312",
"__module.data2vec_audio.encoder.layers.7.feed_forward.output_dense/aten::linear/MatMul_1434",
"__module.data2vec_audio.encoder.layers.5.feed_forward.output_dense/aten::linear/MatMul_1190",
"__module.data2vec_audio.encoder.layers.4.feed_forward.output_dense/aten::linear/MatMul_1068",
"__module.data2vec_audio.encoder.layers.8.feed_forward.output_dense/aten::linear/MatMul_1556"
],
),
)
INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
INFO:nncf:7 ignored nodes was found by name in the NNCFGraph
INFO:nncf:220 ignored nodes was found by types in the NNCFGraph
INFO:nncf:24 ignored nodes was found by name in the NNCFGraph
INFO:nncf:Not adding activation input quantizer for operation: 3 __module.data2vec_audio.feature_extractor.conv_layers.0.conv/aten::_convolution/Convolution_3
INFO:nncf:Not adding activation input quantizer for operation: 5 __module.data2vec_audio.feature_extractor.conv_layers.0.layer_norm/aten::layer_norm/MVN
6 __module.data2vec_audio.feature_extractor.conv_layers.0.layer_norm/aten::layer_norm/Multiply
7 __module.data2vec_audio.feature_extractor.conv_layers.0.layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 10 __module.data2vec_audio.feature_extractor.conv_layers.1.conv/aten::_convolution/Convolution_33
INFO:nncf:Not adding activation input quantizer for operation: 12 __module.data2vec_audio.feature_extractor.conv_layers.1.layer_norm/aten::layer_norm/MVN
13 __module.data2vec_audio.feature_extractor.conv_layers.1.layer_norm/aten::layer_norm/Multiply
14 __module.data2vec_audio.feature_extractor.conv_layers.1.layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 19 __module.data2vec_audio.feature_extractor.conv_layers.2.layer_norm/aten::layer_norm/MVN
20 __module.data2vec_audio.feature_extractor.conv_layers.2.layer_norm/aten::layer_norm/Multiply
21 __module.data2vec_audio.feature_extractor.conv_layers.2.layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 26 __module.data2vec_audio.feature_extractor.conv_layers.3.layer_norm/aten::layer_norm/MVN
27 __module.data2vec_audio.feature_extractor.conv_layers.3.layer_norm/aten::layer_norm/Multiply
28 __module.data2vec_audio.feature_extractor.conv_layers.3.layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 33 __module.data2vec_audio.feature_extractor.conv_layers.4.layer_norm/aten::layer_norm/MVN
34 __module.data2vec_audio.feature_extractor.conv_layers.4.layer_norm/aten::layer_norm/Multiply
35 __module.data2vec_audio.feature_extractor.conv_layers.4.layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 40 __module.data2vec_audio.feature_extractor.conv_layers.5.layer_norm/aten::layer_norm/MVN
41 __module.data2vec_audio.feature_extractor.conv_layers.5.layer_norm/aten::layer_norm/Multiply
42 __module.data2vec_audio.feature_extractor.conv_layers.5.layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 47 __module.data2vec_audio.feature_extractor.conv_layers.6.layer_norm/aten::layer_norm/MVN
48 __module.data2vec_audio.feature_extractor.conv_layers.6.layer_norm/aten::layer_norm/Multiply
49 __module.data2vec_audio.feature_extractor.conv_layers.6.layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 53 __module.data2vec_audio.feature_projection.layer_norm/aten::layer_norm/MVN
54 __module.data2vec_audio.feature_projection.layer_norm/aten::layer_norm/Multiply
55 __module.data2vec_audio.feature_projection.layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 66 __module.data2vec_audio.encoder.pos_conv_embed.layers.0.layer_norm/aten::layer_norm/MVN
INFO:nncf:Not adding activation input quantizer for operation: 114 __module.data2vec_audio.encoder.pos_conv_embed.layers.1.layer_norm/aten::layer_norm/MVN
INFO:nncf:Not adding activation input quantizer for operation: 154 __module.data2vec_audio.encoder.pos_conv_embed.layers.2.layer_norm/aten::layer_norm/MVN
INFO:nncf:Not adding activation input quantizer for operation: 191 __module.data2vec_audio.encoder.pos_conv_embed.layers.3.layer_norm/aten::layer_norm/MVN
INFO:nncf:Not adding activation input quantizer for operation: 233 __module.data2vec_audio.encoder.pos_conv_embed.layers.4.layer_norm/aten::layer_norm/MVN
INFO:nncf:Not adding activation input quantizer for operation: 59 __module.data2vec_audio.encoder/aten::add/Add
INFO:nncf:Not adding activation input quantizer for operation: 61 __module.data2vec_audio.encoder.layer_norm/aten::layer_norm/MVN
63 __module.data2vec_audio.encoder.layer_norm/aten::layer_norm/Multiply
65 __module.data2vec_audio.encoder.layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 71 __module.data2vec_audio.encoder.layers.0/aten::add/Add
INFO:nncf:Not adding activation input quantizer for operation: 78 __module.data2vec_audio.encoder.layers.0.layer_norm/aten::layer_norm/MVN
88 __module.data2vec_audio.encoder.layers.0.layer_norm/aten::layer_norm/Multiply
97 __module.data2vec_audio.encoder.layers.0.layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 106 __module.data2vec_audio.encoder.layers.0/aten::add/Add_583
INFO:nncf:Not adding activation input quantizer for operation: 113 __module.data2vec_audio.encoder.layers.0.final_layer_norm/aten::layer_norm/MVN
117 __module.data2vec_audio.encoder.layers.0.final_layer_norm/aten::layer_norm/Multiply
120 __module.data2vec_audio.encoder.layers.0.final_layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 127 __module.data2vec_audio.encoder.layers.1/aten::add/Add
INFO:nncf:Not adding activation input quantizer for operation: 134 __module.data2vec_audio.encoder.layers.1.layer_norm/aten::layer_norm/MVN
144 __module.data2vec_audio.encoder.layers.1.layer_norm/aten::layer_norm/Multiply
153 __module.data2vec_audio.encoder.layers.1.layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 162 __module.data2vec_audio.encoder.layers.1/aten::add/Add_705
INFO:nncf:Not adding activation input quantizer for operation: 169 __module.data2vec_audio.encoder.layers.1.final_layer_norm/aten::layer_norm/MVN
173 __module.data2vec_audio.encoder.layers.1.final_layer_norm/aten::layer_norm/Multiply
176 __module.data2vec_audio.encoder.layers.1.final_layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 183 __module.data2vec_audio.encoder.layers.2/aten::add/Add
INFO:nncf:Not adding activation input quantizer for operation: 190 __module.data2vec_audio.encoder.layers.2.layer_norm/aten::layer_norm/MVN
200 __module.data2vec_audio.encoder.layers.2.layer_norm/aten::layer_norm/Multiply
209 __module.data2vec_audio.encoder.layers.2.layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 218 __module.data2vec_audio.encoder.layers.2/aten::add/Add_827
INFO:nncf:Not adding activation input quantizer for operation: 225 __module.data2vec_audio.encoder.layers.2.final_layer_norm/aten::layer_norm/MVN
229 __module.data2vec_audio.encoder.layers.2.final_layer_norm/aten::layer_norm/Multiply
232 __module.data2vec_audio.encoder.layers.2.final_layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 239 __module.data2vec_audio.encoder.layers.3/aten::add/Add
INFO:nncf:Not adding activation input quantizer for operation: 246 __module.data2vec_audio.encoder.layers.3.layer_norm/aten::layer_norm/MVN
256 __module.data2vec_audio.encoder.layers.3.layer_norm/aten::layer_norm/Multiply
265 __module.data2vec_audio.encoder.layers.3.layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 273 __module.data2vec_audio.encoder.layers.3/aten::add/Add_949
INFO:nncf:Not adding activation input quantizer for operation: 279 __module.data2vec_audio.encoder.layers.3.final_layer_norm/aten::layer_norm/MVN
282 __module.data2vec_audio.encoder.layers.3.final_layer_norm/aten::layer_norm/Multiply
284 __module.data2vec_audio.encoder.layers.3.final_layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 290 __module.data2vec_audio.encoder.layers.4/aten::add/Add
INFO:nncf:Not adding activation input quantizer for operation: 296 __module.data2vec_audio.encoder.layers.4.layer_norm/aten::layer_norm/MVN
305 __module.data2vec_audio.encoder.layers.4.layer_norm/aten::layer_norm/Multiply
313 __module.data2vec_audio.encoder.layers.4.layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 331 __module.data2vec_audio.encoder.layers.4.feed_forward.output_dense/aten::linear/MatMul_1068
333 __module.data2vec_audio.encoder.layers.4.feed_forward.output_dense/aten::linear/Add
INFO:nncf:Not adding activation input quantizer for operation: 321 __module.data2vec_audio.encoder.layers.4/aten::add/Add_1071
INFO:nncf:Not adding activation input quantizer for operation: 327 __module.data2vec_audio.encoder.layers.4.final_layer_norm/aten::layer_norm/MVN
330 __module.data2vec_audio.encoder.layers.4.final_layer_norm/aten::layer_norm/Multiply
332 __module.data2vec_audio.encoder.layers.4.final_layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 338 __module.data2vec_audio.encoder.layers.5/aten::add/Add
INFO:nncf:Not adding activation input quantizer for operation: 344 __module.data2vec_audio.encoder.layers.5.layer_norm/aten::layer_norm/MVN
353 __module.data2vec_audio.encoder.layers.5.layer_norm/aten::layer_norm/Multiply
361 __module.data2vec_audio.encoder.layers.5.layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 379 __module.data2vec_audio.encoder.layers.5.feed_forward.output_dense/aten::linear/MatMul_1190
381 __module.data2vec_audio.encoder.layers.5.feed_forward.output_dense/aten::linear/Add
INFO:nncf:Not adding activation input quantizer for operation: 369 __module.data2vec_audio.encoder.layers.5/aten::add/Add_1193
INFO:nncf:Not adding activation input quantizer for operation: 375 __module.data2vec_audio.encoder.layers.5.final_layer_norm/aten::layer_norm/MVN
378 __module.data2vec_audio.encoder.layers.5.final_layer_norm/aten::layer_norm/Multiply
380 __module.data2vec_audio.encoder.layers.5.final_layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 386 __module.data2vec_audio.encoder.layers.6/aten::add/Add
INFO:nncf:Not adding activation input quantizer for operation: 392 __module.data2vec_audio.encoder.layers.6.layer_norm/aten::layer_norm/MVN
401 __module.data2vec_audio.encoder.layers.6.layer_norm/aten::layer_norm/Multiply
409 __module.data2vec_audio.encoder.layers.6.layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 427 __module.data2vec_audio.encoder.layers.6.feed_forward.output_dense/aten::linear/MatMul_1312
429 __module.data2vec_audio.encoder.layers.6.feed_forward.output_dense/aten::linear/Add
INFO:nncf:Not adding activation input quantizer for operation: 417 __module.data2vec_audio.encoder.layers.6/aten::add/Add_1315
INFO:nncf:Not adding activation input quantizer for operation: 423 __module.data2vec_audio.encoder.layers.6.final_layer_norm/aten::layer_norm/MVN
426 __module.data2vec_audio.encoder.layers.6.final_layer_norm/aten::layer_norm/Multiply
428 __module.data2vec_audio.encoder.layers.6.final_layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 434 __module.data2vec_audio.encoder.layers.7/aten::add/Add
INFO:nncf:Not adding activation input quantizer for operation: 440 __module.data2vec_audio.encoder.layers.7.layer_norm/aten::layer_norm/MVN
449 __module.data2vec_audio.encoder.layers.7.layer_norm/aten::layer_norm/Multiply
457 __module.data2vec_audio.encoder.layers.7.layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 475 __module.data2vec_audio.encoder.layers.7.feed_forward.output_dense/aten::linear/MatMul_1434
477 __module.data2vec_audio.encoder.layers.7.feed_forward.output_dense/aten::linear/Add
INFO:nncf:Not adding activation input quantizer for operation: 465 __module.data2vec_audio.encoder.layers.7/aten::add/Add_1437
INFO:nncf:Not adding activation input quantizer for operation: 471 __module.data2vec_audio.encoder.layers.7.final_layer_norm/aten::layer_norm/MVN
474 __module.data2vec_audio.encoder.layers.7.final_layer_norm/aten::layer_norm/Multiply
476 __module.data2vec_audio.encoder.layers.7.final_layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 482 __module.data2vec_audio.encoder.layers.8/aten::add/Add
INFO:nncf:Not adding activation input quantizer for operation: 488 __module.data2vec_audio.encoder.layers.8.layer_norm/aten::layer_norm/MVN
497 __module.data2vec_audio.encoder.layers.8.layer_norm/aten::layer_norm/Multiply
505 __module.data2vec_audio.encoder.layers.8.layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 523 __module.data2vec_audio.encoder.layers.8.feed_forward.output_dense/aten::linear/MatMul_1556
525 __module.data2vec_audio.encoder.layers.8.feed_forward.output_dense/aten::linear/Add
INFO:nncf:Not adding activation input quantizer for operation: 513 __module.data2vec_audio.encoder.layers.8/aten::add/Add_1559
INFO:nncf:Not adding activation input quantizer for operation: 519 __module.data2vec_audio.encoder.layers.8.final_layer_norm/aten::layer_norm/MVN
522 __module.data2vec_audio.encoder.layers.8.final_layer_norm/aten::layer_norm/Multiply
524 __module.data2vec_audio.encoder.layers.8.final_layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 530 __module.data2vec_audio.encoder.layers.9/aten::add/Add
INFO:nncf:Not adding activation input quantizer for operation: 536 __module.data2vec_audio.encoder.layers.9.layer_norm/aten::layer_norm/MVN
545 __module.data2vec_audio.encoder.layers.9.layer_norm/aten::layer_norm/Multiply
553 __module.data2vec_audio.encoder.layers.9.layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 561 __module.data2vec_audio.encoder.layers.9/aten::add/Add_1681
INFO:nncf:Not adding activation input quantizer for operation: 567 __module.data2vec_audio.encoder.layers.9.final_layer_norm/aten::layer_norm/MVN
570 __module.data2vec_audio.encoder.layers.9.final_layer_norm/aten::layer_norm/Multiply
572 __module.data2vec_audio.encoder.layers.9.final_layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 578 __module.data2vec_audio.encoder.layers.10/aten::add/Add
INFO:nncf:Not adding activation input quantizer for operation: 584 __module.data2vec_audio.encoder.layers.10.layer_norm/aten::layer_norm/MVN
593 __module.data2vec_audio.encoder.layers.10.layer_norm/aten::layer_norm/Multiply
601 __module.data2vec_audio.encoder.layers.10.layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 609 __module.data2vec_audio.encoder.layers.10/aten::add/Add_1803
INFO:nncf:Not adding activation input quantizer for operation: 615 __module.data2vec_audio.encoder.layers.10.final_layer_norm/aten::layer_norm/MVN
618 __module.data2vec_audio.encoder.layers.10.final_layer_norm/aten::layer_norm/Multiply
620 __module.data2vec_audio.encoder.layers.10.final_layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 626 __module.data2vec_audio.encoder.layers.11/aten::add/Add
INFO:nncf:Not adding activation input quantizer for operation: 632 __module.data2vec_audio.encoder.layers.11.layer_norm/aten::layer_norm/MVN
641 __module.data2vec_audio.encoder.layers.11.layer_norm/aten::layer_norm/Multiply
649 __module.data2vec_audio.encoder.layers.11.layer_norm/aten::layer_norm/Add
INFO:nncf:Not adding activation input quantizer for operation: 657 __module.data2vec_audio.encoder.layers.11/aten::add/Add_1925
INFO:nncf:Not adding activation input quantizer for operation: 663 __module.data2vec_audio.encoder.layers.11.final_layer_norm/aten::layer_norm/MVN
666 __module.data2vec_audio.encoder.layers.11.final_layer_norm/aten::layer_norm/Multiply
668 __module.data2vec_audio.encoder.layers.11.final_layer_norm/aten::layer_norm/Add
Statistics collection: 100%|████████████████████| 73/73 [00:19<00:00, 3.70it/s]
Biases correction: 100%|████████████████████████| 74/74 [00:22<00:00, 3.34it/s]
After quantization is finished, compressed model representation can be
saved using ov.save_model function.
MODEL_NAME = 'quantized_data2vec_base'
quantized_model_path = Path(f"{MODEL_NAME}_openvino_model/{MODEL_NAME}_quantized.xml")
ov.save_model(quantized_model, quantized_model_path)
Check INT8 model inference result¶
INT8 model is the same in usage like the original one. We need to
read it, using the core.read_model method and load on the device,
using core.compile_model. After that, we can reuse the same
ov_infer function for getting model inference result on test sample.
int8_compiled_model = core.compile_model(quantized_model)
transcription = ov_infer(int8_compiled_model, dataset[0])
print(f"[Reference]: {dataset[0]['text']}")
print(f"[OpenVINO INT8]: {transcription[0]}")
ipd.Audio(test_sample["array"], rate=16000)
[Reference]: BECAUSE YOU WERE SLEEPING INSTEAD OF CONQUERING THE LOVELY ROSE PRINCESS HAS BECOME A FIDDLE WITHOUT A BOW WHILE POOR SHAGGY SITS THERE A COOING DOVE
[OpenVINO INT8]: BECAUSE YOU WERE SLEEPING INSTEAD OF CONQUERING THE LOVELY RUSE PRINCESS HAS BECOME A FIDDLE WITHOUT A BOW A POORA SHAGGY SITS THERE A COOING DOVE