OpenVINO Tokenizers: Incorporate Text Processing Into OpenVINO Pipelines

This Jupyter notebook can be launched on-line, opening an interactive environment in a browser window. You can also make a local installation. Choose one of the following options:

OpenVINO Tokenizers is an OpenVINO extension and a Python library designed to streamline tokenizer conversion for seamless integration into your projects. It supports Python and C++ environments and is compatible with all major platforms: Linux, Windows, and MacOS.

Table of contents:

• Tokenization Basics

• Acquiring OpenVINO Tokenizers

  • Convert Tokenizer from HuggingFace Hub with CLI Tool

  • Convert Tokenizer from HuggingFace Hub with Python API

• Text Generation Pipeline with OpenVINO Tokenizers

• Merge Tokenizer into a Model

• Conclusion

• Links

Tokenization Basics

One does not simply put text into a neural network, only numbers. The process of transforming text into a sequence of numbers is called tokenization. It usually contains several steps that transform the original string, splitting it into parts - tokens - with an associated number in a dictionary. You can check the interactive GPT-4 tokenizer to gain an intuitive understanding of the principles of tokenizer work.

There are two important points in the tokenizer-model relation: 1. Every neural network with text input is paired with a tokenizer and cannot be used without it. 2. To reproduce the model’s accuracy on a specific task, it is essential to utilize the same tokenizer employed during the model training.

That is why almost all model repositories on HuggingFace Hub also contain tokenizer files (tokenizer.json, vocab.txt, merges.txt, etc.).

The process of transforming a sequence of numbers into a string is called detokenization. Detokenizer can share the token dictionary with a tokenizer, like any LLM chat model, or operate with an entirely distinct dictionary. For instance, translation models dealing with different source and target languages often necessitate separate dictionaries.

Some tasks only need a tokenizer, like text classification, named entity recognition, question answering, and feature extraction. On the other hand, for tasks such as text generation, chat, translation, and abstractive summarization, both a tokenizer and a detokenizer are required.

Acquiring OpenVINO Tokenizers

OpenVINO Tokenizers Python library allows you to convert HuggingFace tokenizers into OpenVINO models. To install all required dependencies use pip install openvino-tokenizers[transformers].

%pip install -Uq pip
%pip uninstall -y openvino openvino-nightly openvino-dev
%pip install --pre -Uq openvino-tokenizers[transformers] --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly
%pip install "torch>=2.1" --extra-index-url https://download.pytorch.org/whl/cpu

Note: you may need to restart the kernel to use updated packages.
Found existing installation: openvino 2024.1.0
Uninstalling openvino-2024.1.0:
  Successfully uninstalled openvino-2024.1.0
WARNING: Skipping openvino-nightly as it is not installed.
Found existing installation: openvino-dev 2024.1.0
Uninstalling openvino-dev-2024.1.0:
  Successfully uninstalled openvino-dev-2024.1.0
Note: you may need to restart the kernel to use updated packages.
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
mobileclip 0.1.0 requires torch==1.13.1, but you have torch 2.3.0+cpu which is incompatible.
mobileclip 0.1.0 requires torchvision==0.14.1, but you have torchvision 0.18.0+cpu which is incompatible.
Note: you may need to restart the kernel to use updated packages.
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cpu
Requirement already satisfied: torch>=2.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (2.3.0+cpu)
Requirement already satisfied: filelock in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (3.14.0)
Requirement already satisfied: typing-extensions>=4.8.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (4.11.0)
Requirement already satisfied: sympy in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (1.12)
Requirement already satisfied: networkx in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (3.1)
Requirement already satisfied: jinja2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (3.1.4)
Requirement already satisfied: fsspec in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch>=2.1) (2024.3.1)
Requirement already satisfied: MarkupSafe>=2.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from jinja2->torch>=2.1) (2.1.5)
Requirement already satisfied: mpmath>=0.19 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from sympy->torch>=2.1) (1.3.0)
Note: you may need to restart the kernel to use updated packages.

from pathlib import Path


tokenizer_dir = Path("tokenizer/")
model_id = "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T"

Convert Tokenizer from HuggingFace Hub with CLI Tool

The first way is to use the CLI utility, bundled with OpenVINO Tokenizers. Use --with-detokenizer flag to add the detokenizer model to the output. By setting --clean-up-tokenization-spaces=False we ensure that the detokenizer correctly decodes a code-generation model output. --trust-remote-code flag works the same way as passing trust_remote_code=True to AutoTokenizer.from_pretrained constructor.

!convert_tokenizer $model_id --with-detokenizer -o $tokenizer_dir

 Loading Huggingface Tokenizer...
 /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: resume_download is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use force_download=True.
   warnings.warn(
 Converting Huggingface Tokenizer to OpenVINO...
 Saved OpenVINO Tokenizer: tokenizer/openvino_tokenizer.xml, tokenizer/openvino_tokenizer.bin
 Saved OpenVINO Detokenizer: tokenizer/openvino_detokenizer.xml, tokenizer/openvino_detokenizer.bin


⚠️ If you have any problems with the command above on MacOS, try to
install tbb.

The result is two OpenVINO models: openvino_tokenizer and openvino_detokenizer. Both can be interacted with using read_model, compile_model and save_model, similar to any other OpenVINO model.

Convert Tokenizer from HuggingFace Hub with Python API

The other method is to pass HuggingFace hf_tokenizer object to convert_tokenizer function:

from transformers import AutoTokenizer
from openvino_tokenizers import convert_tokenizer


hf_tokenizer = AutoTokenizer.from_pretrained(model_id)
ov_tokenizer, ov_detokenizer = convert_tokenizer(hf_tokenizer, with_detokenizer=True)
ov_tokenizer, ov_detokenizer

/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: resume_download is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use force_download=True.
  warnings.warn(

(<Model: 'tokenizer'
 inputs[
 <ConstOutput: names[string_input] shape[?] type: string>
 ]
 outputs[
 <ConstOutput: names[input_ids] shape[?,?] type: i64>,
 <ConstOutput: names[attention_mask] shape[?,?] type: i64>
 ]>,
 <Model: 'detokenizer'
 inputs[
 <ConstOutput: names[Parameter_21] shape[?,?] type: i64>
 ]
 outputs[
 <ConstOutput: names[string_output] shape[?] type: string>
 ]>)

That way you get OpenVINO model objects. Use save_model function from OpenVINO to reuse converted tokenizers later:

from openvino import save_model


save_model(ov_tokenizer, tokenizer_dir / "openvino_tokenizer.xml")
save_model(ov_detokenizer, tokenizer_dir / "openvino_detokenizer.xml")

To use the tokenizer, compile the converted model and input a list of strings. It’s essential to be aware that not all original tokenizers support multiple strings (also called batches) as input. This limitation arises from the requirement for all resulting number sequences to maintain the same length. To address this, a padding token must be specified, which will be appended to shorter tokenized strings. In cases where no padding token is determined in the original tokenizer, OpenVINO Tokenizers defaults to using \(0\) for padding. Presently, only right-side padding is supported, typically used for classification tasks, but not suitable for text generation.

from openvino import compile_model


tokenizer, detokenizer = compile_model(ov_tokenizer), compile_model(ov_detokenizer)
test_strings = ["Test", "strings"]

token_ids = tokenizer(test_strings)["input_ids"]
print(f"Token ids: {token_ids}")

detokenized_text = detokenizer(token_ids)["string_output"]
print(f"Detokenized text: {detokenized_text}")

Token ids: [[   1 4321]
 [   1 6031]]
Detokenized text: ['<s> Test' '<s> strings']

We can compare the result of converted (de)tokenizer with the original one:

hf_token_ids = hf_tokenizer(test_strings).input_ids
print(f"Token ids: {hf_token_ids}")

hf_detokenized_text = hf_tokenizer.batch_decode(hf_token_ids)
print(f"Detokenized text: {hf_detokenized_text}")

Token ids: [[1, 4321], [1, 6031]]

2024-05-07 00:39:47.665228: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0.
2024-05-07 00:39:47.699909: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-05-07 00:39:48.279101: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT

Detokenized text: ['<s> Test', '<s> strings']

Text Generation Pipeline with OpenVINO Tokenizers

Let’s build a text generation pipeline with OpenVINO Tokenizers and minimal dependencies. To obtain an OpenVINO model we will use the Optimum library. The latest version allows you to get a so-called stateful model.

The original TinyLlama-1.1B-intermediate-step-1431k-3T model is 4.4Gb. To reduce network and disk usage we will load a converted model which has also been compressed to int8. The original conversion command is commented.

model_dir = Path(Path(model_id).name)

if not model_dir.exists():
    # converting the original model
    # %pip install -U "git+https://github.com/huggingface/optimum-intel.git" "nncf>=2.8.0" onnx
    # %optimum-cli export openvino -m $model_id --task text-generation-with-past $model_dir

    # load already converted model
    from huggingface_hub import hf_hub_download

    hf_hub_download(
        "chgk13/TinyLlama-1.1B-intermediate-step-1431k-3T",
        filename="openvino_model.xml",
        local_dir=model_dir,
    )
    hf_hub_download(
        "chgk13/TinyLlama-1.1B-intermediate-step-1431k-3T",
        filename="openvino_model.bin",
        local_dir=model_dir,
    )

openvino_model.xml:   0%|          | 0.00/2.93M [00:00<?, ?B/s]

openvino_model.bin:   0%|          | 0.00/1.10G [00:00<?, ?B/s]

import numpy as np
from tqdm.notebook import trange
from pathlib import Path
from openvino_tokenizers import add_greedy_decoding
from openvino_tokenizers.constants import EOS_TOKEN_ID_NAME
from openvino import Core


core = Core()

# add the greedy decoding subgraph on top of LLM to get the most probable token as an output
ov_model = add_greedy_decoding(core.read_model(model_dir / "openvino_model.xml"))
compiled_model = core.compile_model(ov_model)
infer_request = compiled_model.create_infer_request()

The infer_request object provides control over the model’s state - a Key-Value cache that speeds up inference by reducing computations Multiple inference requests can be created, and each request maintains a distinct and separate state..

text_input = ["Quick brown fox jumped"]

model_input = {name.any_name: output for name, output in tokenizer(text_input).items()}

if "position_ids" in (input.any_name for input in infer_request.model_inputs):
    model_input["position_ids"] = np.arange(model_input["input_ids"].shape[1], dtype=np.int64)[np.newaxis, :]

# no beam search, set idx to 0
model_input["beam_idx"] = np.array([0], dtype=np.int32)
# end of sentence token is that model signifies the end of text generation
# read EOS token ID from rt_info of tokenizer/detokenizer ov.Model object
eos_token = ov_tokenizer.get_rt_info(EOS_TOKEN_ID_NAME).value

tokens_result = np.array([[]], dtype=np.int64)

# reset KV cache inside the model before inference
infer_request.reset_state()
max_infer = 10

for _ in trange(max_infer):
    infer_request.start_async(model_input)
    infer_request.wait()

    # get a prediction for the last token on the first inference
    output_token = infer_request.get_output_tensor().data[:, -1:]
    tokens_result = np.hstack((tokens_result, output_token))
    if output_token[0, 0] == eos_token:
        break

    # prepare input for new inference
    model_input["input_ids"] = output_token
    model_input["attention_mask"] = np.hstack((model_input["attention_mask"].data, [[1]]))
    model_input["position_ids"] = np.hstack(
        (
            model_input["position_ids"].data,
            [[model_input["position_ids"].data.shape[-1]]],
        )
    )

text_result = detokenizer(tokens_result)["string_output"]
print(f"Prompt:\n{text_input[0]}")
print(f"Generated:\n{text_result[0]}")

0%|          | 0/10 [00:00<?, ?it/s]

Prompt:
Quick brown fox jumped
Generated:
over the fence.

Merge Tokenizer into a Model

Packages like tensorflow-text offer the convenience of integrating text processing directly into the model, streamlining both distribution and usage. Similarly, with OpenVINO Tokenizers, you can create models that combine a converted tokenizer and a model. It’s important to note that not all scenarios benefit from this merge. In cases where a tokenizer is used once and a model is inferred multiple times, as seen in the earlier text generation example, maintaining a separate (de)tokenizer and model is advisable to prevent unnecessary tokenization-detokenization cycles during inference. Conversely, if both a tokenizer and a model are used once in each pipeline inference, merging simplifies the workflow and aids in avoiding the creation of intermediate objects:

The OpenVINO Python API allows you to avoid this by using the share_inputs option during inference, but it requires additional input from a developer every time the model is inferred. Combining the models and tokenizers simplifies memory management.

model_id = "mrm8488/bert-tiny-finetuned-sms-spam-detection"
model_dir = Path(Path(model_id).name)

if not model_dir.exists():
    %pip install -qU git+https://github.com/huggingface/optimum-intel.git onnx
    !optimum-cli export openvino --model $model_id --task text-classification $model_dir
    !convert_tokenizer $model_id -o $model_dir

huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using tokenizers before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

Note: you may need to restart the kernel to use updated packages.

huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using tokenizers before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

2024-05-07 00:40:23.074270: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  torch.utils._pytree._register_pytree_node(
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: resume_download is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use force_download=True.
  warnings.warn(
Framework not specified. Using pt to export the model.
Using framework PyTorch: 2.3.0+cpu
Overriding 1 configuration item(s)
    - use_cache -> False
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/transformers/modeling_utils.py:4371: FutureWarning: _is_quantized_training_enabled is going to be deprecated in transformers 4.39.0. Please use model.hf_quantizer.is_trainable instead
  warnings.warn(
Detokenizer is not supported, convert tokenizer only.

huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using tokenizers before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

Loading Huggingface Tokenizer...
/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: resume_download is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use force_download=True.
  warnings.warn(
Converting Huggingface Tokenizer to OpenVINO...
Saved OpenVINO Tokenizer: bert-tiny-finetuned-sms-spam-detection/openvino_tokenizer.xml, bert-tiny-finetuned-sms-spam-detection/openvino_tokenizer.bin

from openvino import Core, save_model
from openvino_tokenizers import connect_models


core = Core()
text_input = ["Free money!!!"]

ov_tokenizer = core.read_model(model_dir / "openvino_tokenizer.xml")
ov_model = core.read_model(model_dir / "openvino_model.xml")
combined_model = connect_models(ov_tokenizer, ov_model)
save_model(combined_model, model_dir / "combined_openvino_model.xml")

compiled_combined_model = core.compile_model(combined_model)
openvino_output = compiled_combined_model(text_input)

print(f"Logits: {openvino_output['logits']}")

Logits: [[ 1.2007061 -1.4698029]]

Conclusion

The OpenVINO Tokenizers integrate text processing operations into the OpenVINO ecosystem. Enabling the conversion of HuggingFace tokenizers into OpenVINO models, the library allows efficient deployment of deep learning pipelines across varied environments. The feature of combining tokenizers and models not only simplifies memory management but also helps to streamline model usage and deployment.

Links

• Installation instructions for different environments

• Supported Tokenizer Types

• OpenVINO.GenAI repository with the C++ example of OpenVINO Tokenizers usage

• HuggingFace Tokenizers Comparison Table