Quantize Speech Recognition Models with accuracy control using NNCF PTQ API¶
This tutorial is also available as a Jupyter notebook that can be cloned directly from GitHub. See the installation guide for instructions to run this tutorial locally on Windows, Linux or macOS.
This tutorial demonstrates how to apply INT8 quantization with
accuracy control to the speech recognition model, known as
Wav2Vec2,
using the NNCF (Neural Network Compression Framework) 8-bit quantization
with accuracy control in post-training mode (without the fine-tuning
pipeline). This notebook uses a fine-tuned
Wav2Vec2-Base-960h
PyTorch model trained on the LibriSpeech ASR
corpus. The tutorial is designed to be
extendable to custom models and datasets. It consists of the following
steps:
Download and prepare the Wav2Vec2 model and LibriSpeech dataset.
Define data loading and accuracy validation functionality.
Model quantization with accuracy control.
Compare Accuracy of original PyTorch model, OpenVINO FP16 and INT8 models.
Compare performance of the original and quantized models.
The advanced quantization flow allows to apply 8-bit quantization to the model with control of accuracy metric. This is achieved by keeping the most impactful operations within the model in the original precision. The flow is based on the Basic 8-bit quantization and has the following differences:
Besides the calibration dataset, a validation dataset is required to compute the accuracy metric. Both datasets can refer to the same data in the simplest case.
Validation function, used to compute accuracy metric is required. It can be a function that is already available in the source framework or a custom function.
Since accuracy validation is run several times during the quantization process, quantization with accuracy control can take more time than the Basic 8-bit quantization flow.
The resulted model can provide smaller performance improvement than the Basic 8-bit quantization flow because some of the operations are kept in the original precision.
Note
Currently, 8-bit quantization with accuracy control in NNCF is available only for models in OpenVINO representation.
The steps for the quantization with accuracy control are described below.
Table of contents:
# !pip install -q "openvino-dev>=2023.1.0" "nncf>=2.6.0"
!pip install -q "openvino==2023.1.0.dev20230811"
!pip install git+https://github.com/openvinotoolkit/nncf.git@develop
!pip install -q soundfile librosa transformers torch datasets torchmetrics
Collecting git+https://github.com/openvinotoolkit/nncf.git@develop
Cloning https://github.com/openvinotoolkit/nncf.git (to revision develop) to /tmp/pip-req-build-o2lphim0
Running command git clone --filter=blob:none --quiet https://github.com/openvinotoolkit/nncf.git /tmp/pip-req-build-o2lphim0
Filtering content: 1% (2/142)
Filtering content: 2% (3/142)
Filtering content: 3% (5/142)
Filtering content: 4% (6/142)
Filtering content: 5% (8/142), 10.29 MiB | 16.71 MiB/s
Filtering content: 6% (9/142), 10.29 MiB | 16.71 MiB/s
Filtering content: 7% (10/142), 10.29 MiB | 16.71 MiB/s
Filtering content: 7% (10/142), 12.61 MiB | 8.69 MiB/s
Filtering content: 8% (12/142), 12.61 MiB | 8.69 MiB/s
Filtering content: 9% (13/142), 12.61 MiB | 8.69 MiB/s
Filtering content: 10% (15/142), 14.35 MiB | 7.17 MiB/s
Filtering content: 11% (16/142), 14.35 MiB | 7.17 MiB/s
Filtering content: 12% (18/142), 14.35 MiB | 7.17 MiB/s
Filtering content: 13% (19/142), 17.07 MiB | 6.80 MiB/s
Filtering content: 14% (20/142), 17.07 MiB | 6.80 MiB/s
Filtering content: 15% (22/142), 17.07 MiB | 6.80 MiB/s
Filtering content: 16% (23/142), 17.07 MiB | 6.80 MiB/s
Filtering content: 17% (25/142), 19.78 MiB | 6.42 MiB/s
Filtering content: 18% (26/142), 19.78 MiB | 6.42 MiB/s
Filtering content: 19% (27/142), 19.78 MiB | 6.42 MiB/s
Filtering content: 20% (29/142), 19.78 MiB | 6.42 MiB/s
Filtering content: 21% (30/142), 19.78 MiB | 6.42 MiB/s
Filtering content: 22% (32/142), 22.80 MiB | 6.19 MiB/s
Filtering content: 23% (33/142), 22.80 MiB | 6.19 MiB/s
Filtering content: 24% (35/142), 22.80 MiB | 6.19 MiB/s
Filtering content: 25% (36/142), 22.80 MiB | 6.19 MiB/s
Filtering content: 26% (37/142), 22.80 MiB | 6.19 MiB/s
Filtering content: 26% (37/142), 25.18 MiB | 5.93 MiB/s
Filtering content: 27% (39/142), 25.18 MiB | 5.93 MiB/s
Filtering content: 28% (40/142), 25.18 MiB | 5.93 MiB/s
Filtering content: 29% (42/142), 25.18 MiB | 5.93 MiB/s
Filtering content: 30% (43/142), 25.18 MiB | 5.93 MiB/s
Filtering content: 31% (45/142), 25.18 MiB | 5.93 MiB/s
Filtering content: 32% (46/142), 27.34 MiB | 5.71 MiB/s
Filtering content: 33% (47/142), 27.34 MiB | 5.71 MiB/s
Filtering content: 34% (49/142), 27.34 MiB | 5.71 MiB/s
Filtering content: 35% (50/142), 27.34 MiB | 5.71 MiB/s
Filtering content: 36% (52/142), 27.34 MiB | 5.71 MiB/s
Filtering content: 37% (53/142), 27.34 MiB | 5.71 MiB/s
Filtering content: 38% (54/142), 27.34 MiB | 5.71 MiB/s
Filtering content: 39% (56/142), 27.34 MiB | 5.71 MiB/s
Filtering content: 40% (57/142), 29.35 MiB | 5.54 MiB/s
Filtering content: 41% (59/142), 29.35 MiB | 5.54 MiB/s
Filtering content: 42% (60/142), 29.35 MiB | 5.54 MiB/s
Filtering content: 43% (62/142), 29.35 MiB | 5.54 MiB/s
Filtering content: 44% (63/142), 29.35 MiB | 5.54 MiB/s
Filtering content: 45% (64/142), 29.35 MiB | 5.54 MiB/s
Filtering content: 46% (66/142), 29.35 MiB | 5.54 MiB/s
Filtering content: 47% (67/142), 29.35 MiB | 5.54 MiB/s
Filtering content: 48% (69/142), 29.35 MiB | 5.54 MiB/s
Filtering content: 49% (70/142), 29.35 MiB | 5.54 MiB/s
Filtering content: 50% (71/142), 29.35 MiB | 5.54 MiB/s
Filtering content: 51% (73/142), 29.35 MiB | 5.54 MiB/s
Filtering content: 52% (74/142), 29.35 MiB | 5.54 MiB/s
Filtering content: 53% (76/142), 29.35 MiB | 5.54 MiB/s
Filtering content: 54% (77/142), 29.35 MiB | 5.54 MiB/s
Filtering content: 55% (79/142), 29.35 MiB | 5.54 MiB/s
Filtering content: 56% (80/142), 29.35 MiB | 5.54 MiB/s
Filtering content: 57% (81/142), 29.35 MiB | 5.54 MiB/s
Filtering content: 58% (83/142), 29.35 MiB | 5.54 MiB/s
Filtering content: 59% (84/142), 29.35 MiB | 5.54 MiB/s
Filtering content: 60% (86/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 61% (87/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 62% (89/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 63% (90/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 64% (91/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 65% (93/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 66% (94/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 67% (96/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 68% (97/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 69% (98/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 70% (100/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 71% (101/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 72% (103/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 73% (104/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 74% (106/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 75% (107/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 76% (108/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 77% (110/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 78% (111/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 79% (113/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 80% (114/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 81% (116/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 82% (117/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 83% (118/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 84% (120/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 85% (121/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 86% (123/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 87% (124/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 88% (125/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 89% (127/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 90% (128/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 91% (130/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 92% (131/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 93% (133/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 94% (134/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 95% (135/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 96% (137/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 97% (138/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 98% (140/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 99% (141/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 100% (142/142), 31.63 MiB | 4.19 MiB/s
Filtering content: 100% (142/142), 32.00 MiB | 3.57 MiB/s, done.
Resolved https://github.com/openvinotoolkit/nncf.git to commit 90a1e860c93b553fa9684113e02d41d622235c55
Preparing metadata (setup.py) ... - done
Collecting pymoo@ git+https://github.com/anyoptimization/pymoo.git@695cb26923903f872c7256a9013609769f3cc2bd (from nncf==2.5.0.dev0+90a1e860)
Using cached pymoo-0.6.0.1-py3-none-any.whl
Requirement already satisfied: jsonschema>=3.2.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from nncf==2.5.0.dev0+90a1e860) (4.19.0)
Requirement already satisfied: jstyleson>=0.0.2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from nncf==2.5.0.dev0+90a1e860) (0.0.2)
Requirement already satisfied: natsort>=7.1.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from nncf==2.5.0.dev0+90a1e860) (8.4.0)
Requirement already satisfied: networkx<=2.8.2,>=2.6 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from nncf==2.5.0.dev0+90a1e860) (2.8.2)
Requirement already satisfied: ninja<1.11,>=1.10.0.post2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from nncf==2.5.0.dev0+90a1e860) (1.10.2.4)
Requirement already satisfied: numpy<1.25,>=1.19.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from nncf==2.5.0.dev0+90a1e860) (1.23.5)
Requirement already satisfied: openvino-telemetry>=2023.1.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from nncf==2.5.0.dev0+90a1e860) (2023.1.1)
Requirement already satisfied: packaging>=20.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from nncf==2.5.0.dev0+90a1e860) (23.1)
Requirement already satisfied: pandas<2.1,>=1.1.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from nncf==2.5.0.dev0+90a1e860) (2.0.3)
Requirement already satisfied: psutil in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from nncf==2.5.0.dev0+90a1e860) (5.9.5)
Requirement already satisfied: pydot>=1.4.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from nncf==2.5.0.dev0+90a1e860) (1.4.2)
Requirement already satisfied: pyparsing<3.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from nncf==2.5.0.dev0+90a1e860) (2.4.7)
Requirement already satisfied: scikit-learn>=0.24.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from nncf==2.5.0.dev0+90a1e860) (1.3.0)
Requirement already satisfied: scipy<1.11,>=1.3.2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from nncf==2.5.0.dev0+90a1e860) (1.10.1)
Requirement already satisfied: texttable>=1.6.3 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from nncf==2.5.0.dev0+90a1e860) (1.6.7)
Requirement already satisfied: tqdm>=4.54.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from nncf==2.5.0.dev0+90a1e860) (4.66.1)
Requirement already satisfied: attrs>=22.2.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from jsonschema>=3.2.0->nncf==2.5.0.dev0+90a1e860) (23.1.0)
Requirement already satisfied: importlib-resources>=1.4.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from jsonschema>=3.2.0->nncf==2.5.0.dev0+90a1e860) (6.0.1)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from jsonschema>=3.2.0->nncf==2.5.0.dev0+90a1e860) (2023.7.1)
Requirement already satisfied: pkgutil-resolve-name>=1.3.10 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from jsonschema>=3.2.0->nncf==2.5.0.dev0+90a1e860) (1.3.10)
Requirement already satisfied: referencing>=0.28.4 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from jsonschema>=3.2.0->nncf==2.5.0.dev0+90a1e860) (0.30.2)
Requirement already satisfied: rpds-py>=0.7.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from jsonschema>=3.2.0->nncf==2.5.0.dev0+90a1e860) (0.10.2)
Requirement already satisfied: python-dateutil>=2.8.2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from pandas<2.1,>=1.1.5->nncf==2.5.0.dev0+90a1e860) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from pandas<2.1,>=1.1.5->nncf==2.5.0.dev0+90a1e860) (2023.3.post1)
Requirement already satisfied: tzdata>=2022.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from pandas<2.1,>=1.1.5->nncf==2.5.0.dev0+90a1e860) (2023.3)
Requirement already satisfied: joblib>=1.1.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from scikit-learn>=0.24.0->nncf==2.5.0.dev0+90a1e860) (1.3.2)
Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from scikit-learn>=0.24.0->nncf==2.5.0.dev0+90a1e860) (3.2.0)
Requirement already satisfied: matplotlib>=3 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from pymoo@ git+https://github.com/anyoptimization/pymoo.git@695cb26923903f872c7256a9013609769f3cc2bd->nncf==2.5.0.dev0+90a1e860) (3.5.2)
Requirement already satisfied: autograd>=1.4 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from pymoo@ git+https://github.com/anyoptimization/pymoo.git@695cb26923903f872c7256a9013609769f3cc2bd->nncf==2.5.0.dev0+90a1e860) (1.6.2)
Collecting cma==3.2.2 (from pymoo@ git+https://github.com/anyoptimization/pymoo.git@695cb26923903f872c7256a9013609769f3cc2bd->nncf==2.5.0.dev0+90a1e860)
Using cached cma-3.2.2-py2.py3-none-any.whl (249 kB)
Collecting alive-progress (from pymoo@ git+https://github.com/anyoptimization/pymoo.git@695cb26923903f872c7256a9013609769f3cc2bd->nncf==2.5.0.dev0+90a1e860)
Obtaining dependency information for alive-progress from https://files.pythonhosted.org/packages/e3/02/5d7f9158d69b36fbe9eb0df8fb435008ec881e41bc7d839239004207d807/alive_progress-3.1.4-py3-none-any.whl.metadata
Using cached alive_progress-3.1.4-py3-none-any.whl.metadata (68 kB)
Requirement already satisfied: dill in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from pymoo@ git+https://github.com/anyoptimization/pymoo.git@695cb26923903f872c7256a9013609769f3cc2bd->nncf==2.5.0.dev0+90a1e860) (0.3.7)
Collecting Deprecated (from pymoo@ git+https://github.com/anyoptimization/pymoo.git@695cb26923903f872c7256a9013609769f3cc2bd->nncf==2.5.0.dev0+90a1e860)
Obtaining dependency information for Deprecated from https://files.pythonhosted.org/packages/20/8d/778b7d51b981a96554f29136cd59ca7880bf58094338085bcf2a979a0e6a/Deprecated-1.2.14-py2.py3-none-any.whl.metadata
Using cached Deprecated-1.2.14-py2.py3-none-any.whl.metadata (5.4 kB)
Requirement already satisfied: future>=0.15.2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from autograd>=1.4->pymoo@ git+https://github.com/anyoptimization/pymoo.git@695cb26923903f872c7256a9013609769f3cc2bd->nncf==2.5.0.dev0+90a1e860) (0.18.3)
Requirement already satisfied: zipp>=3.1.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from importlib-resources>=1.4.0->jsonschema>=3.2.0->nncf==2.5.0.dev0+90a1e860) (3.16.2)
Requirement already satisfied: cycler>=0.10 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from matplotlib>=3->pymoo@ git+https://github.com/anyoptimization/pymoo.git@695cb26923903f872c7256a9013609769f3cc2bd->nncf==2.5.0.dev0+90a1e860) (0.11.0)
Requirement already satisfied: fonttools>=4.22.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from matplotlib>=3->pymoo@ git+https://github.com/anyoptimization/pymoo.git@695cb26923903f872c7256a9013609769f3cc2bd->nncf==2.5.0.dev0+90a1e860) (4.42.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from matplotlib>=3->pymoo@ git+https://github.com/anyoptimization/pymoo.git@695cb26923903f872c7256a9013609769f3cc2bd->nncf==2.5.0.dev0+90a1e860) (1.4.5)
Requirement already satisfied: pillow>=6.2.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from matplotlib>=3->pymoo@ git+https://github.com/anyoptimization/pymoo.git@695cb26923903f872c7256a9013609769f3cc2bd->nncf==2.5.0.dev0+90a1e860) (10.0.0)
Requirement already satisfied: six>=1.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from python-dateutil>=2.8.2->pandas<2.1,>=1.1.5->nncf==2.5.0.dev0+90a1e860) (1.16.0)
Collecting about-time==4.2.1 (from alive-progress->pymoo@ git+https://github.com/anyoptimization/pymoo.git@695cb26923903f872c7256a9013609769f3cc2bd->nncf==2.5.0.dev0+90a1e860)
Using cached about_time-4.2.1-py3-none-any.whl (13 kB)
Collecting grapheme==0.6.0 (from alive-progress->pymoo@ git+https://github.com/anyoptimization/pymoo.git@695cb26923903f872c7256a9013609769f3cc2bd->nncf==2.5.0.dev0+90a1e860)
Using cached grapheme-0.6.0-py3-none-any.whl
Requirement already satisfied: wrapt<2,>=1.10 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from Deprecated->pymoo@ git+https://github.com/anyoptimization/pymoo.git@695cb26923903f872c7256a9013609769f3cc2bd->nncf==2.5.0.dev0+90a1e860) (1.14.1)
Using cached alive_progress-3.1.4-py3-none-any.whl (75 kB)
Using cached Deprecated-1.2.14-py2.py3-none-any.whl (9.6 kB)
Building wheels for collected packages: nncf
Building wheel for nncf (setup.py) ... - | / done
Created wheel for nncf: filename=nncf-2.5.0.dev0+90a1e860-py3-none-any.whl size=1139358 sha256=35a2f1daf4360a3b65a6a2996cca9f15d165f6c25994f64d8ccf10960e7a55bc
Stored in directory: /tmp/pip-ephem-wheel-cache-mdg9hjsd/wheels/6d/17/88/a292ae87701bc65e2e1c63261d22d7fb0e15aa8448ee693d5f
Successfully built nncf
Installing collected packages: grapheme, Deprecated, cma, about-time, alive-progress, pymoo, nncf
Attempting uninstall: cma
Found existing installation: cma 2.7.0
Uninstalling cma-2.7.0:
Successfully uninstalled cma-2.7.0
Attempting uninstall: pymoo
Found existing installation: pymoo 0.5.0
Uninstalling pymoo-0.5.0:
Successfully uninstalled pymoo-0.5.0
Attempting uninstall: nncf
Found existing installation: nncf 2.5.0
Uninstalling nncf-2.5.0:
Successfully uninstalled nncf-2.5.0
Successfully installed Deprecated-1.2.14 about-time-4.2.1 alive-progress-3.1.4 cma-3.2.2 grapheme-0.6.0 nncf-2.5.0.dev0+90a1e860 pymoo-0.6.0.1
Imports¶
import numpy as np
import torch
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
2023-09-08 23:07:39.211214: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0. 2023-09-08 23:07:39.246066: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-09-08 23:07:39.789011: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Prepare the Model¶
For instantiating PyTorch model class,
we should use Wav2Vec2ForCTC.from_pretrained method with providing
model ID for downloading from HuggingFace hub. Model weights and
configuration files will be downloaded automatically in first time
usage. Keep in mind that downloading the files can take several minutes
and depends on your internet connection.
Additionally, we can create processor class which is responsible for model specific pre- and post-processing steps.
BATCH_SIZE = 1
MAX_SEQ_LENGTH = 30480
torch_model = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-base-960h", ctc_loss_reduction="mean")
processor = Wav2Vec2Processor.from_pretrained("facebook/wav2vec2-base-960h")
Some weights of Wav2Vec2ForCTC were not initialized from the model checkpoint at facebook/wav2vec2-base-960h and are newly initialized: ['wav2vec2.masked_spec_embed']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Convert it to the OpenVINO Intermediate Representation (OpenVINO IR)
import openvino as ov
default_input = torch.zeros([1, MAX_SEQ_LENGTH], dtype=torch.float)
ov_model = ov.convert_model(torch_model, example_input=default_input)
WARNING:tensorflow:Please fix your imports. Module tensorflow.python.training.tracking.base has been moved to tensorflow.python.trackable.base. The old module will be deleted in version 2.11.
[ WARNING ] Please fix your imports. Module %s has been moved to %s. The old module will be deleted in version %s.
INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
WARNING:nncf:NNCF provides best results with torch==2.0.1, while current torch version is 1.13.1+cpu. If you encounter issues, consider switching to torch==2.0.1
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:595: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:634: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
Prepare LibriSpeech Dataset¶
For demonstration purposes, we will use short dummy version of
LibriSpeech dataset - patrickvonplaten/librispeech_asr_dummy to
speed up model evaluation. Model accuracy can be different from reported
in the paper. For reproducing original accuracy, use librispeech_asr
dataset.
from datasets import load_dataset
dataset = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation")
test_sample = dataset[0]["audio"]
# define preprocessing function for converting audio to input values for model
def map_to_input(batch):
preprocessed_signal = processor(batch["audio"]["array"], return_tensors="pt", padding="longest", sampling_rate=batch['audio']['sampling_rate'])
input_values = preprocessed_signal.input_values
batch['input_values'] = input_values
return batch
# apply preprocessing function to dataset and remove audio column, to save memory as we do not need it anymore
dataset = dataset.map(map_to_input, batched=False, remove_columns=["audio"])
Prepare calibration dataset¶
import nncf
def transform_fn(data_item):
"""
Extract the model's input from the data item.
The data item here is the data item that is returned from the data source per iteration.
This function should be passed when the data item cannot be used as model's input.
"""
return np.array(data_item["input_values"])
calibration_dataset = nncf.Dataset(dataset, transform_fn)
Prepare validation function¶
Define the validation function.
from torchmetrics import WordErrorRate
from tqdm.notebook import tqdm
def validation_fn(model, dataset):
"""
Calculate and returns a metric for the model.
"""
wer = WordErrorRate()
for sample in tqdm(dataset):
# run infer function on sample
output = model.output(0)
logits = model(np.array(sample['input_values']))[output]
predicted_ids = np.argmax(logits, axis=-1)
transcription = processor.batch_decode(torch.from_numpy(predicted_ids))
# update metric on sample result
wer.update(transcription, [sample['text']])
result = wer.compute()
return 1 - result
Run quantization with accuracy control¶
You should provide
the calibration dataset and the validation dataset. It can be the same
dataset. - parameter max_drop defines the accuracy drop threshold.
The quantization process stops when the degradation of accuracy metric
on the validation dataset is less than the max_drop. The default
value is 0.01. NNCF will stop the quantization and report an error if
the max_drop value can’t be reached. - drop_type defines how the
accuracy drop will be calculated: ABSOLUTE (used by default) or
RELATIVE. - ranking_subset_size - size of a subset that is used to
rank layers by their contribution to the accuracy drop. Default value is
300, and the more samples it has the better ranking, potentially. Here
we use the value 25 to speed up the execution.
Execution can take tens of minutes and requires up to 10 GB of free memory
from nncf.quantization.advanced_parameters import AdvancedAccuracyRestorerParameters
from nncf.parameters import ModelType
quantized_model = nncf.quantize_with_accuracy_control(
ov_model,
calibration_dataset=calibration_dataset,
validation_dataset=calibration_dataset,
validation_fn=validation_fn,
max_drop=0.01,
drop_type=nncf.DropType.ABSOLUTE,
model_type=ModelType.TRANSFORMER,
advanced_accuracy_restorer_parameters=AdvancedAccuracyRestorerParameters(
ranking_subset_size=25
),
)
Statistics collection: 24%|██▍ | 73/300 [00:13<00:42, 5.37it/s]
Applying Smooth Quant: 100%|██████████| 50/50 [00:00<00:00, 58.74it/s]
INFO:nncf:36 ignored nodes was found by name in the NNCFGraph
Statistics collection: 24%|██▍ | 73/300 [00:23<01:12, 3.12it/s]
Applying Fast Bias correction: 100%|██████████| 74/74 [00:25<00:00, 2.91it/s]
INFO:nncf:Validation of initial model was started
INFO:nncf:Elapsed Time: 00:00:00
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-499/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/torchmetrics/utilities/prints.py:62: FutureWarning: Importing WordErrorRate from torchmetrics was deprecated and will be removed in 2.0. Import WordErrorRate from torchmetrics.text instead. _future_warning(
0it [00:00, ?it/s]
0it [00:00, ?it/s]
INFO:nncf:Elapsed Time: 00:00:13
INFO:nncf:Metric of initial model: 0.9469565153121948
INFO:nncf:Collecting values for each data item using the initial model
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
INFO:nncf:Elapsed Time: 00:00:10
INFO:nncf:Validation of quantized model was started
INFO:nncf:Elapsed Time: 00:00:20
0it [00:00, ?it/s]
INFO:nncf:Elapsed Time: 00:00:13
INFO:nncf:Metric of quantized model: 0.49826085567474365
INFO:nncf:Collecting values for each data item using the quantized model
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
INFO:nncf:Elapsed Time: 00:00:07
INFO:nncf:Accuracy drop: 0.44869565963745117 (DropType.ABSOLUTE)
INFO:nncf:Accuracy drop: 0.44869565963745117 (DropType.ABSOLUTE)
INFO:nncf:Total number of quantized operations in the model: 94
INFO:nncf:Number of parallel processes to rank quantized operations: 11
INFO:nncf:ORIGINAL metric is used to rank quantizers
INFO:nncf:Calculating ranking score for groups of quantizers
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
INFO:nncf:Elapsed Time: 00:04:58
INFO:nncf:Changing the scope of quantizer nodes was started
INFO:nncf:Reverted 1 operations to the floating-point precision:
__module.wav2vec2.feature_extractor.conv_layers.2.conv/aten::_convolution/Convolution_11
0it [00:00, ?it/s]
INFO:nncf:Accuracy drop with the new quantization scope is 0.06000000238418579 (DropType.ABSOLUTE)
INFO:nncf:Reverted 1 operations to the floating-point precision:
__module.wav2vec2.feature_extractor.conv_layers.1.conv/aten::_convolution/Convolution_10
0it [00:00, ?it/s]
INFO:nncf:Algorithm completed: achieved required accuracy drop 0.007826089859008789 (DropType.ABSOLUTE)
INFO:nncf:2 out of 94 were reverted back to the floating-point precision:
__module.wav2vec2.feature_extractor.conv_layers.2.conv/aten::_convolution/Convolution_11
__module.wav2vec2.feature_extractor.conv_layers.1.conv/aten::_convolution/Convolution_10
Model Usage Example¶
import IPython.display as ipd
ipd.Audio(test_sample["array"], rate=16000)
core = ov.Core()
compiled_quantized_model = core.compile_model(model=quantized_model, device_name='CPU')
input_data = np.expand_dims(test_sample["array"], axis=0)
Next, make a prediction.
predictions = compiled_quantized_model([input_data])[0]
predicted_ids = np.argmax(predictions, axis=-1)
transcription = processor.batch_decode(torch.from_numpy(predicted_ids))
transcription
['A MAN SAID TO THE UNIVERSE SIR I EXIST']
Compare Accuracy of the Original and Quantized Models¶
Define dataloader for test dataset.
Define functions to get inference for PyTorch and OpenVINO models.
Define functions to compute Word Error Rate.
# inference function for pytorch
def torch_infer(model, sample):
logits = model(torch.Tensor(sample['input_values'])).logits
# take argmax and decode
predicted_ids = torch.argmax(logits, dim=-1)
transcription = processor.batch_decode(predicted_ids)
return transcription
# inference function for openvino
def ov_infer(model, sample):
output = model.output(0)
logits = model(np.array(sample['input_values']))[output]
predicted_ids = np.argmax(logits, axis=-1)
transcription = processor.batch_decode(torch.from_numpy(predicted_ids))
return transcription
def compute_wer(dataset, model, infer_fn):
wer = WordErrorRate()
for sample in tqdm(dataset):
# run infer function on sample
transcription = infer_fn(model, sample)
# update metric on sample result
wer.update(transcription, [sample['text']])
# finalize metric calculation
result = wer.compute()
return result
Now, compute WER for the original PyTorch model and quantized model.
pt_result = compute_wer(dataset, torch_model, torch_infer)
quantized_result = compute_wer(dataset, compiled_quantized_model, ov_infer)
print(f'[PyTorch] Word Error Rate: {pt_result:.4f}')
print(f'[Quantized OpenVino] Word Error Rate: {quantized_result:.4f}')
0%| | 0/73 [00:00<?, ?it/s]
0%| | 0/73 [00:00<?, ?it/s]
[PyTorch] Word Error Rate: 0.0530
[Quantized OpenVino] Word Error Rate: 0.0609