openvino_genai.PerfMetrics

class openvino_genai.PerfMetrics

Bases: pybind11_object

Holds performance metrics for each generate call.

PerfMetrics holds fields with mean and standard deviations for the following metrics: - Time To the First Token (TTFT), ms - Time per Output Token (TPOT), ms/token - Generate total duration, ms - Tokenization duration, ms - Detokenization duration, ms - Throughput, tokens/s

Additional fields include: - Load time, ms - Number of generated tokens - Number of tokens in the input prompt

Preferable way to access values is via get functions. Getters calculate mean and std values from raw_metrics and return pairs. If mean and std were already calculated, getters return cached values.

Parameters:

• get_load_time (float) – Returns the load time in milliseconds.

• get_num_generated_tokens (int) – Returns the number of generated tokens.

• get_num_input_tokens (int) – Returns the number of tokens in the input prompt.

• get_ttft (MeanStdPair) – Returns the mean and standard deviation of TTFT in milliseconds.

• get_tpot (MeanStdPair) – Returns the mean and standard deviation of TPOT in milliseconds.

• get_throughput (MeanStdPair) – Returns the mean and standard deviation of throughput in tokens per second.

• get_generate_duration (MeanStdPair) – Returns the mean and standard deviation of generate durations in milliseconds.

• get_tokenization_duration (MeanStdPair) – Returns the mean and standard deviation of tokenization durations in milliseconds.

• get_detokenization_duration (MeanStdPair) – Returns the mean and standard deviation of detokenization durations in milliseconds.

• raw_metrics (RawPerfMetrics) – A structure of RawPerfMetrics type that holds raw metrics.

__init__(self: openvino_genai.py_openvino_genai.PerfMetrics) → None

Methods

__add__(self, metrics)
__delattr__(name, /)	Implement delattr(self, name).
__dir__()	Default dir() implementation.
__eq__(value, /)	Return self==value.
__format__(format_spec, /)	Default object formatter.
__ge__(value, /)	Return self>=value.
__getattribute__(name, /)	Return getattr(self, name).
__gt__(value, /)	Return self>value.
__hash__()	Return hash(self).
__iadd__(self, right)
__init__(self)
__init_subclass__	This method is called when a class is subclassed.
__le__(value, /)	Return self<=value.
__lt__(value, /)	Return self<value.
__ne__(value, /)	Return self!=value.
__new__(**kwargs)
__reduce__()	Helper for pickle.
__reduce_ex__(protocol, /)	Helper for pickle.
__repr__()	Return repr(self).
__setattr__(name, value, /)	Implement setattr(self, name, value).
__sizeof__()	Size of object in memory, in bytes.
__str__()	Return str(self).
__subclasshook__	Abstract classes can override this to customize issubclass().
get_detokenization_duration(self)
get_generate_duration(self)
get_inference_duration(self)
get_ipot(self)
get_load_time(self)
get_num_generated_tokens(self)
get_num_input_tokens(self)
get_throughput(self)
get_tokenization_duration(self)
get_tpot(self)
get_ttft(self)

Attributes

raw_metrics

__add__(self: openvino_genai.py_openvino_genai.PerfMetrics, metrics: openvino_genai.py_openvino_genai.PerfMetrics) → openvino_genai.py_openvino_genai.PerfMetrics

__class__

alias of pybind11_type

__delattr__(name, /)

Implement delattr(self, name).

__dir__()

Default dir() implementation.

__eq__(value, /)

Return self==value.

__format__(format_spec, /)

Default object formatter.

__ge__(value, /)

Return self>=value.

__getattribute__(name, /)

Return getattr(self, name).

__gt__(value, /)

Return self>value.

__hash__()

Return hash(self).

__iadd__(self: openvino_genai.py_openvino_genai.PerfMetrics, right: openvino_genai.py_openvino_genai.PerfMetrics) → openvino_genai.py_openvino_genai.PerfMetrics

__init__(self: openvino_genai.py_openvino_genai.PerfMetrics) → None

__init_subclass__()

This method is called when a class is subclassed.

The default implementation does nothing. It may be overridden to extend subclasses.

__le__(value, /)

Return self<=value.

__lt__(value, /)

Return self<value.

__ne__(value, /)

Return self!=value.

__new__(**kwargs)

__reduce__()

Helper for pickle.

__reduce_ex__(protocol, /)

Helper for pickle.

__repr__()

Return repr(self).

__setattr__(name, value, /)

Implement setattr(self, name, value).

__sizeof__()

Size of object in memory, in bytes.

__str__()

Return str(self).

__subclasshook__()

Abstract classes can override this to customize issubclass().

This is invoked early on by abc.ABCMeta.__subclasscheck__(). It should return True, False or NotImplemented. If it returns NotImplemented, the normal algorithm is used. Otherwise, it overrides the normal algorithm (and the outcome is cached).

get_detokenization_duration(self: openvino_genai.py_openvino_genai.PerfMetrics) → openvino_genai.py_openvino_genai.MeanStdPair

get_generate_duration(self: openvino_genai.py_openvino_genai.PerfMetrics) → openvino_genai.py_openvino_genai.MeanStdPair

get_inference_duration(self: openvino_genai.py_openvino_genai.PerfMetrics) → openvino_genai.py_openvino_genai.MeanStdPair

get_ipot(self: openvino_genai.py_openvino_genai.PerfMetrics) → openvino_genai.py_openvino_genai.MeanStdPair

get_load_time(self: openvino_genai.py_openvino_genai.PerfMetrics) → float

get_num_generated_tokens(self: openvino_genai.py_openvino_genai.PerfMetrics) → int

get_num_input_tokens(self: openvino_genai.py_openvino_genai.PerfMetrics) → int

get_throughput(self: openvino_genai.py_openvino_genai.PerfMetrics) → openvino_genai.py_openvino_genai.MeanStdPair

get_tokenization_duration(self: openvino_genai.py_openvino_genai.PerfMetrics) → openvino_genai.py_openvino_genai.MeanStdPair

get_tpot(self: openvino_genai.py_openvino_genai.PerfMetrics) → openvino_genai.py_openvino_genai.MeanStdPair

get_ttft(self: openvino_genai.py_openvino_genai.PerfMetrics) → openvino_genai.py_openvino_genai.MeanStdPair

property raw_metrics