openvino_genai.Tokenizer#

class openvino_genai.Tokenizer#

Bases: pybind11_object

The class is used to encode prompts and decode resulting tokens

Chat template is initialized from sources in the following order overriding the previous value: 1. chat_template entry from tokenizer_config.json 2. chat_template entry from processor_config.json 3. chat_template entry from chat_template.json 4. chat_template entry from rt_info section of openvino.Model 5. If the template is known to be not supported by GenAI, it’s

replaced with a simplified supported version.

__init__(*args, **kwargs)#

Overloaded function.

__init__(self: openvino_genai.py_openvino_genai.Tokenizer, tokenizer_path: os.PathLike | str | bytes, properties: collections.abc.Mapping[str, object] = {}, **kwargs) -> None
__init__(self: openvino_genai.py_openvino_genai.Tokenizer, tokenizer_model: str, tokenizer_weights: openvino._pyopenvino.Tensor, detokenizer_model: str, detokenizer_weights: openvino._pyopenvino.Tensor, **kwargs) -> None

Methods

`__delattr__`(name, /)	Implement delattr(self, name).
`__dir__`()	Default dir() implementation.
`__eq__`(value, /)	Return self==value.
`__format__`(format_spec, /)	Default object formatter.
`__ge__`(value, /)	Return self>=value.
`__getattribute__`(name, /)	Return getattr(self, name).
`__getstate__`()	Helper for pickle.
`__gt__`(value, /)	Return self>value.
`__hash__`()	Return hash(self).
`__init__`(args, *kwargs)	Overloaded function.
`__init_subclass__`	This method is called when a class is subclassed.
`__le__`(value, /)	Return self<=value.
`__lt__`(value, /)	Return self<value.
`__ne__`(value, /)	Return self!=value.
`__new__`(**kwargs)
`__reduce__`()	Helper for pickle.
`__reduce_ex__`(protocol, /)	Helper for pickle.
`__repr__`()	Return repr(self).
`__setattr__`(name, value, /)	Implement setattr(self, name, value).
`__sizeof__`()	Size of object in memory, in bytes.
`__str__`()	Return str(self).
`__subclasshook__`	Abstract classes can override this to customize issubclass().
`_pybind11_conduit_v1_`
`apply_chat_template`(self, history, ...[, ...])	Embeds input prompts with special tags for a chat scenario.
`decode`(args, *kwargs)	Overloaded function.
`encode`(args, *kwargs)	Overloaded function.
`get_bos_token`(self)
`get_bos_token_id`(self)
`get_eos_token`(self)
`get_eos_token_id`(self)
`get_pad_token`(self)
`get_pad_token_id`(self)
`get_vocab`(self)	Returns the vocabulary as a Python dictionary with bytes keys and integer values.
`get_vocab_vector`(self)	Returns the vocabulary as list of strings, where position of a string represents token ID.
`set_chat_template`(self, chat_template)	Override a chat_template read from tokenizer_config.json.
`supports_paired_input`(self)	Returns true if the tokenizer supports paired input, false otherwise.

Attributes

`__annotations__`
`chat_template`

__annotations__ = {}#

__class__#: alias of pybind11_type

__delattr__(name, /)#: Implement delattr(self, name).

__dir__()#: Default dir() implementation.

__eq__(value, /)#: Return self==value.

__format__(format_spec, /)#

Default object formatter.

Return str(self) if format_spec is empty. Raise TypeError otherwise.

__ge__(value, /)#: Return self>=value.

__getattribute__(name, /)#: Return getattr(self, name).

__getstate__()#: Helper for pickle.

__gt__(value, /)#: Return self>value.

__hash__()#: Return hash(self).

__init__(*args, **kwargs)#

Overloaded function.

__init__(self: openvino_genai.py_openvino_genai.Tokenizer, tokenizer_path: os.PathLike | str | bytes, properties: collections.abc.Mapping[str, object] = {}, **kwargs) -> None
__init__(self: openvino_genai.py_openvino_genai.Tokenizer, tokenizer_model: str, tokenizer_weights: openvino._pyopenvino.Tensor, detokenizer_model: str, detokenizer_weights: openvino._pyopenvino.Tensor, **kwargs) -> None

__init_subclass__()#

This method is called when a class is subclassed.

The default implementation does nothing. It may be overridden to extend subclasses.

__le__(value, /)#: Return self<=value.

__lt__(value, /)#: Return self<value.

__ne__(value, /)#: Return self!=value.

__new__(**kwargs)#

__reduce__()#: Helper for pickle.

__reduce_ex__(protocol, /)#: Helper for pickle.

__repr__()#: Return repr(self).

__setattr__(name, value, /)#: Implement setattr(self, name, value).

__sizeof__()#: Size of object in memory, in bytes.

__str__()#: Return str(self).

__subclasshook__()#

Abstract classes can override this to customize issubclass().

This is invoked early on by abc.ABCMeta.__subclasscheck__(). It should return True, False or NotImplemented. If it returns NotImplemented, the normal algorithm is used. Otherwise, it overrides the normal algorithm (and the outcome is cached).

_pybind11_conduit_v1_()#

apply_chat_template(self: openvino_genai.py_openvino_genai.Tokenizer, history: collections.abc.Sequence[collections.abc.Mapping[str, str]], add_generation_prompt: bool, chat_template: str = '') → str#: Embeds input prompts with special tags for a chat scenario.

property chat_template#

decode(*args, **kwargs)#

Overloaded function.

decode(self: openvino_genai.py_openvino_genai.Tokenizer, tokens: collections.abc.Sequence[typing.SupportsInt], skip_special_tokens: bool = True) -> str

Decode a sequence into a string prompt.

decode(self: openvino_genai.py_openvino_genai.Tokenizer, tokens: openvino._pyopenvino.Tensor, skip_special_tokens: bool = True) -> list[str]

Decode tensor into a list of string prompts.

decode(self: openvino_genai.py_openvino_genai.Tokenizer, tokens: collections.abc.Sequence[collections.abc.Sequence[typing.SupportsInt]], skip_special_tokens: bool = True) -> list[str]

Decode a batch of tokens into a list of string prompt.

encode(*args, **kwargs)#

Overloaded function.

encode(self: openvino_genai.py_openvino_genai.Tokenizer, prompts: collections.abc.Sequence[str], add_special_tokens: bool = True, pad_to_max_length: bool = False, max_length: typing.SupportsInt | None = None, padding_side: str | None = None) -> openvino_genai.py_openvino_genai.TokenizedInputs

Encodes a list of prompts into tokenized inputs. Args:

‘prompts’ - list of prompts to encode ‘add_special_tokens’ - whether to add special tokens like BOS, EOS, PAD. Default is True. ‘pad_to_max_length’ - whether to pad the sequence to the maximum length. Default is False. ‘max_length’ - maximum length of the sequence. If None (default), the value will be taken from the IR (where default value from original HF/GGUF model is stored). ‘padding_side’ - side to pad the sequence, can be ‘left’ or ‘right’. If None (default), the value will be taken from the IR (where default value from original HF/GGUF model is stored).

Returns:: TokenizedInputs object containing input_ids and attention_mask tensors.

encode(self: openvino_genai.py_openvino_genai.Tokenizer, prompt: str, add_special_tokens: bool = True, pad_to_max_length: bool = False, max_length: typing.SupportsInt | None = None, padding_side: str | None = None) -> openvino_genai.py_openvino_genai.TokenizedInputs

Encodes a single prompt into tokenized input. Args:

‘prompt’ - prompt to encode ‘add_special_tokens’ - whether to add special tokens like BOS, EOS, PAD. Default is True. ‘pad_to_max_length’ - whether to pad the sequence to the maximum length. Default is False. ‘max_length’ - maximum length of the sequence. If None (default), the value will be taken from the IR (where default value from original HF/GGUF model is stored). ‘padding_side’ - side to pad the sequence, can be ‘left’ or ‘right’. If None (default), the value will be taken from the IR (where default value from original HF/GGUF model is stored).

Returns:: TokenizedInputs object containing input_ids and attention_mask tensors.

encode(self: openvino_genai.py_openvino_genai.Tokenizer, prompts_1: collections.abc.Sequence[str], prompts_2: collections.abc.Sequence[str], add_special_tokens: bool = True, pad_to_max_length: bool = False, max_length: typing.SupportsInt | None = None, padding_side: str | None = None) -> openvino_genai.py_openvino_genai.TokenizedInputs

Encodes a list of prompts into tokenized inputs. The number of strings must be the same, or one of the inputs can contain one string. In the latter case, the single-string input will be broadcast into the shape of the other input, which is more efficient than repeating the string in pairs.) Args:

‘prompts_1’ - list of prompts to encode ‘prompts_2’ - list of prompts to encode ‘add_special_tokens’ - whether to add special tokens like BOS, EOS, PAD. Default is True. ‘pad_to_max_length’ - whether to pad the sequence to the maximum length. Default is False. ‘max_length’ - maximum length of the sequence. If None (default), the value will be taken from the IR (where default value from original HF/GGUF model is stored). ‘padding_side’ - side to pad the sequence, can be ‘left’ or ‘right’. If None (default), the value will be taken from the IR (where default value from original HF/GGUF model is stored).

Returns:: TokenizedInputs object containing input_ids and attention_mask tensors.

encode(self: openvino_genai.py_openvino_genai.Tokenizer, prompts: list, add_special_tokens: bool = True, pad_to_max_length: bool = False, max_length: typing.SupportsInt | None = None, padding_side: str | None = None) -> openvino_genai.py_openvino_genai.TokenizedInputs

Encodes a list of paired prompts into tokenized inputs. Input format is same as for HF paired input [[prompt_1, prompt_2], …]. Args:

‘prompts’ - list of prompts to encoden ‘add_special_tokens’ - whether to add special tokens like BOS, EOS, PAD. Default is True. ‘pad_to_max_length’ - whether to pad the sequence to the maximum length. Default is False. ‘max_length’ - maximum length of the sequence. If None (default), the value will be taken from the IR (where default value from original HF/GGUF model is stored). ‘padding_side’ - side to pad the sequence, can be ‘left’ or ‘right’. If None (default), the value will be taken from the IR (where default value from original HF/GGUF model is stored).

Returns:: TokenizedInputs object containing input_ids and attention_mask tensors.

get_bos_token(self: openvino_genai.py_openvino_genai.Tokenizer) → str#

get_bos_token_id(self: openvino_genai.py_openvino_genai.Tokenizer) → int#

get_eos_token(self: openvino_genai.py_openvino_genai.Tokenizer) → str#

get_eos_token_id(self: openvino_genai.py_openvino_genai.Tokenizer) → int#

get_pad_token(self: openvino_genai.py_openvino_genai.Tokenizer) → str#

get_pad_token_id(self: openvino_genai.py_openvino_genai.Tokenizer) → int#

get_vocab(self: openvino_genai.py_openvino_genai.Tokenizer) → dict#

Returns the vocabulary as a Python dictionary with bytes keys and integer values.: Bytes are used for keys because not all vocabulary entries might be valid UTF-8 strings.

get_vocab_vector(self: openvino_genai.py_openvino_genai.Tokenizer) → list[str]#: Returns the vocabulary as list of strings, where position of a string represents token ID.

set_chat_template(self: openvino_genai.py_openvino_genai.Tokenizer, chat_template: str) → None#: Override a chat_template read from tokenizer_config.json.

supports_paired_input(self: openvino_genai.py_openvino_genai.Tokenizer) → bool#: Returns true if the tokenizer supports paired input, false otherwise.