🤗 Hugging Face Model Hub with OpenVINO™

This tutorial is also available as a Jupyter notebook that can be cloned directly from GitHub. See the installation guide for instructions to run this tutorial locally on Windows, Linux or macOS. To run without installing anything, click the “launch binder” or “Open in Colab” button.

The Hugging Face (HF) Model Hub is a central repository for pre-trained deep learning models. It allows exploration and provides access to thousands of models for a wide range of tasks, including text classification, question answering, and image classification. Hugging Face provides Python packages that serve as APIs and tools to easily download and fine tune state-of-the-art pretrained models, namely transformers and diffusers packages.

Throughout this notebook we will learn: 1. How to load a HF pipeline using the transformers package and then convert it to OpenVINO. 2. How to load the same pipeline using Optimum Intel package.

Contents:

• Converting a Model from the HF Transformers Package

• Installing Requirements

• Imports

• Initializing a Model Using the HF Transformers Package

• Original Model inference

• Converting the Model to OpenVINO IR format

• Converted Model Inference

• Converting a Model Using the Optimum Intel Package

• Installing Requirements

• Import Optimum

• Initialize and Convert the Model Automatically

Converting a Model from the HF Transformers Package

Hugging Face transformers package provides API for initializing a model and loading a set of pre-trained weights using the model text handle. Discovering a desired model name is straightforward with HF website’s Models page, one can choose a model solving a particular machine learning problem and even sort the models by popularity and novelty.

Installing Requirements

%pip install -q --extra-index-url https://download.pytorch.org/whl/cpu transformers[torch]
%pip install -q ipywidgets
%pip install -q "openvino>=2023.1.0"

Note: you may need to restart the kernel to use updated packages.
Note: you may need to restart the kernel to use updated packages.
Note: you may need to restart the kernel to use updated packages.

Imports

from pathlib import Path

import numpy as np
import torch

from transformers import AutoModelForSequenceClassification
from transformers import AutoTokenizer

Initializing a Model Using the HF Transformers Package

We will use roberta text sentiment classification model in our example, it is a transformer-based encoder model pretrained in a special way, please refer to the model card to learn more.

Following the instructions on the model page, we use AutoModelForSequenceClassification to initialize the model and perform inference with it. To find more information on HF pipelines and model initialization please refer to HF tutorials.

MODEL = "cardiffnlp/twitter-roberta-base-sentiment-latest"

tokenizer = AutoTokenizer.from_pretrained(MODEL, return_dict=True)

# The torchscript=True flag is used to ensure the model outputs are tuples
# instead of ModelOutput (which causes JIT errors).
model = AutoModelForSequenceClassification.from_pretrained(MODEL, torchscript=True)

Some weights of the model checkpoint at cardiffnlp/twitter-roberta-base-sentiment-latest were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']
- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

Original Model inference

Let’s do a classification of a simple prompt below.

text = "HF models run perfectly with OpenVINO!"

encoded_input = tokenizer(text, return_tensors='pt')
output = model.forward(**encoded_input)
scores = output[0][0]
scores = torch.softmax(scores, dim=0).detach().numpy()

def print_prediction(scores):
    for i, descending_index in enumerate(scores.argsort()[::-1]):
        label = model.config.id2label[descending_index]
        score = np.round(float(scores[descending_index]), 4)
        print(f"{i+1}) {label} {score}")

print_prediction(scores)

1) positive 0.9485
2) neutral 0.0484
3) negative 0.0031

Converting the Model to OpenVINO IR format

We use the OpenVINO Model conversion API to convert the model (this one is implemented in PyTorch) to OpenVINO Intermediate Representation (IR).

Note how we reuse our real encoded_input, passing it to the ov.convert_model function. It will be used for model tracing.

import openvino as ov

save_model_path = Path('./models/model.xml')

if not save_model_path.exists():
    ov_model = ov.convert_model(model, example_input=dict(encoded_input))
    ov.save_model(ov_model, save_model_path)

Converted Model Inference

First, we pick a device to do the model inference

import ipywidgets as widgets

core = ov.Core()

device = widgets.Dropdown(
    options=core.available_devices + ["AUTO"],
    value='AUTO',
    description='Device:',
    disabled=False,
)

device

Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')

OpenVINO model IR must be compiled for a specific device prior to the model inference.

compiled_model = core.compile_model(save_model_path, device.value)

# Compiled model call is performed using the same parameters as for the original model
scores_ov = compiled_model(encoded_input.data)[0]

scores_ov = torch.softmax(torch.tensor(scores_ov[0]), dim=0).detach().numpy()

print_prediction(scores_ov)

1) positive 0.9483
2) neutral 0.0485
3) negative 0.0031

Note the prediction of the converted model match exactly the one of the original model.

This is a rather simple example as the pipeline includes just one encoder model. Contemporary state of the art pipelines often consist of several model, feel free to explore other OpenVINO tutorials: 1. Stable Diffusion v2 2. Zero-shot Image Classification with OpenAI CLIP 3. Controllable Music Generation with MusicGen

The workflow for the diffusers package is exactly the same. The first example in the list above relies on the diffusers.

Converting a Model Using the Optimum Intel Package

🤗 Optimum Intel is the interface between the 🤗 Transformers and Diffusers libraries and the different tools and libraries provided by Intel to accelerate end-to-end pipelines on Intel architectures.

Among other use cases, Optimum Intel provides a simple interface to optimize your Transformers and Diffusers models, convert them to the OpenVINO Intermediate Representation (IR) format and run inference using OpenVINO Runtime.

Install Requirements for Optimum

%pip install -q "optimum==1.13.0"
%pip install -q "optimum-intel"@git+https://github.com/huggingface/optimum-intel.git
%pip install -q onnx

huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using tokenizers before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

Note: you may need to restart the kernel to use updated packages.

huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using tokenizers before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

Note: you may need to restart the kernel to use updated packages.

huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using tokenizers before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

Note: you may need to restart the kernel to use updated packages.

Import Optimum

Documentation for Optimum Intel states: >You can now easily perform inference with OpenVINO Runtime on a variety of Intel processors (see the full list of supported devices). For that, just replace the AutoModelForXxx class with the corresponding OVModelForXxx class.

You can find Optimum Intel documentation on the Hugging Face website.

from optimum.intel.openvino import OVModelForSequenceClassification

INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino

huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using tokenizers before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using tokenizers before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
2023-10-30 23:06:03.589130: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0.
2023-10-30 23:06:03.624230: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-10-30 23:06:04.183799: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-534/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
  warnings.warn(

Initialize and Convert the Model Automatically

To load a Transformers model and convert it to the OpenVINO format on-the-fly, you can set export=True when loading your model.

model = OVModelForSequenceClassification.from_pretrained(MODEL, export=True, device=device.value)

# The save_pretrained() method saves the model weights to avoid conversion on the next load.
model.save_pretrained('./models')

Framework not specified. Using pt to export to ONNX.
Some weights of the model checkpoint at cardiffnlp/twitter-roberta-base-sentiment-latest were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']
- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Using the export variant default. Available variants are:
    - default: The default ONNX variant.
Using framework PyTorch: 2.1.0+cpu
Overriding 1 configuration item(s)
    - use_cache -> False

WARNING:tensorflow:Please fix your imports. Module tensorflow.python.training.tracking.base has been moved to tensorflow.python.trackable.base. The old module will be deleted in version 2.11.

Compiling the model to AUTO ...
Set CACHE_DIR to /tmp/tmpx5aqydhf/model_cache

Moreover, some models in the Hugging Face Models Hub are already converted and ready to run! You can filter those models out by library name, just type OpenVINO, or follow this link.

The Optimum Model Inference

Model inference is exactly the same as for the original model!

output = model.forward(**encoded_input)
scores = output[0][0]
scores = torch.softmax(scores, dim=0).detach().numpy()

print_prediction(scores)

1) positive 0.9485
2) neutral 0.0484
3) negative 0.0031

You can find more examples of using Optimum Intel here: 1. Accelerate Inference of Sparse Transformer Models 2. Grammatical Error Correction with OpenVINO 3. Stable Diffusion v2.1 using Optimum-Intel OpenVINO 4. Image generation with Stable Diffusion XL 5. Instruction following using Databricks Dolly 2.0 6. Create LLM-powered Chatbot using OpenVINO