openvino.runtime.CompiledModel#

class openvino.runtime.CompiledModel(other: CompiledModel, weights: bytes | None = None)#

Bases: CompiledModel

CompiledModel class.

CompiledModel represents Model that is compiled for a specific device by applying multiple optimization transformations, then mapping to compute kernels.

__init__(self: openvino._pyopenvino.CompiledModel, other: openvino._pyopenvino.CompiledModel) → None#

Methods

__call__([inputs, share_inputs, ...])

Callable infer wrapper for CompiledModel.

__delattr__(name, /)

Implement delattr(self, name).

__dir__()

Default dir() implementation.

__eq__(value, /)

Return self==value.

__format__(format_spec, /)

Default object formatter.

__ge__(value, /)

Return self>=value.

__getattribute__(name, /)

Return getattr(self, name).

__gt__(value, /)

Return self>value.

__hash__()

Return hash(self).

__init__(self, other)

__init_subclass__

This method is called when a class is subclassed.

__le__(value, /)

Return self<=value.

__lt__(value, /)

Return self<value.

__ne__(value, /)

Return self!=value.

__new__(**kwargs)

__reduce__()

Helper for pickle.

__reduce_ex__(protocol, /)

Helper for pickle.

__repr__(self)

__setattr__(name, value, /)

Implement setattr(self, name, value).

__sizeof__()

Size of object in memory, in bytes.

__str__()

Return str(self).

__subclasshook__

Abstract classes can override this to customize issubclass().

create_infer_request()

Creates an inference request object used to infer the compiled model.

export_model(*args, **kwargs)

Overloaded function.

get_property(self, property)

Gets properties for current compiled model.

get_runtime_model(self)

Gets runtime model information from a device.

infer_new_request([inputs])

Infers specified input(s) in synchronous mode.

input(*args, **kwargs)

Overloaded function.

output(*args, **kwargs)

Overloaded function.

query_state()

Gets state control interface for the underlaying infer request.

release_memory(self)

Release intermediate memory.

reset_state()

Resets all internal variable states of the underlaying infer request.

set_property(*args, **kwargs)

Overloaded function.

Attributes

inputs

Gets all inputs of a compiled model.

outputs

Gets all outputs of a compiled model.
__call__(inputs: Any | None = None, share_inputs: bool = True, share_outputs: bool = False, *, decode_strings: bool = True) → OVDict#

Callable infer wrapper for CompiledModel.

Infers specified input(s) in synchronous mode.

Blocks all methods of CompiledModel while request is running.

Method creates new temporary InferRequest and run inference on it. It is advised to use a dedicated InferRequest class for performance, optimizing workflows, and creating advanced pipelines.

This method stores created InferRequest inside CompiledModel object, which can be later reused in consecutive calls.

The allowed types of keys in the inputs dictionary are:

  1. int

  2. str

  3. openvino.runtime.ConstOutput

The allowed types of values in the inputs are:

  1. numpy.ndarray and all the types that are castable to it, e.g. torch.Tensor

  2. openvino.runtime.Tensor

Can be called with only one openvino.runtime.Tensor or numpy.ndarray, it will work only with one-input models. When model has more inputs, function throws error.

Parameters:

  • inputs (Any, optional) – Data to be set on input tensors.

  • share_inputs (bool, optional) –

    Enables share_inputs mode. Controls memory usage on inference’s inputs.

    If set to False inputs the data dispatcher will safely copy data to existing Tensors (including up- or down-casting according to data type, resizing of the input Tensor). Keeps Tensor inputs “as-is”.

    If set to True the data dispatcher tries to provide “zero-copy” Tensors for every input in form of: * numpy.ndarray and all the types that are castable to it, e.g. torch.Tensor Data that is going to be copied: * numpy.ndarray which are not C contiguous and/or not writable (WRITEABLE flag is set to False) * inputs which data types are mismatched from Infer Request’s inputs * inputs that should be in BF16 data type * scalar inputs (i.e. np.float_/str/bytes/int/float) * lists of simple data types (i.e. str/bytes/int/float) Keeps Tensor inputs “as-is”.

    Note: Use with extra care, shared data can be modified during runtime! Note: Using share_inputs may result in extra memory overhead.

    Default value: True

  • share_outputs (bool, optional) –

    Enables share_outputs mode. Controls memory usage on inference’s outputs.

    If set to False outputs will safely copy data to numpy arrays.

    If set to True the data will be returned in form of views of output Tensors. This mode still returns the data in format of numpy arrays but lifetime of the data is connected to OpenVINO objects.

    Note: Use with extra care, shared data can be modified or lost during runtime! Note: String/textual data will always be copied!

    Default value: False

  • decode_strings (bool, optional, keyword-only) –

    Controls decoding outputs of textual based data.

    If set to True string outputs will be returned as numpy arrays of U kind.

    If set to False string outputs will be returned as numpy arrays of S kind.

    Default value: True

Returns:

Dictionary of results from output tensors with port/int/str as keys.

Return type:

OVDict

__class__#

alias of pybind11_type

__delattr__(name, /)#

Implement delattr(self, name).

__dir__()#

Default dir() implementation.

__eq__(value, /)#

Return self==value.

__format__(format_spec, /)#

Default object formatter.

__ge__(value, /)#

Return self>=value.

__getattribute__(name, /)#

Return getattr(self, name).

__gt__(value, /)#

Return self>value.

__hash__()#

Return hash(self).

__init__(self: openvino._pyopenvino.CompiledModel, other: openvino._pyopenvino.CompiledModel) → None#
__init_subclass__()#

This method is called when a class is subclassed.

The default implementation does nothing. It may be overridden to extend subclasses.

__le__(value, /)#

Return self<=value.

__lt__(value, /)#

Return self<value.

__ne__(value, /)#

Return self!=value.

__new__(**kwargs)#
__reduce__()#

Helper for pickle.

__reduce_ex__(protocol, /)#

Helper for pickle.

__repr__(self: openvino._pyopenvino.CompiledModel) → str#
__setattr__(name, value, /)#

Implement setattr(self, name, value).

__sizeof__()#

Size of object in memory, in bytes.

__str__()#

Return str(self).

__subclasshook__()#

Abstract classes can override this to customize issubclass().

This is invoked early on by abc.ABCMeta.__subclasscheck__(). It should return True, False or NotImplemented. If it returns NotImplemented, the normal algorithm is used. Otherwise, it overrides the normal algorithm (and the outcome is cached).

create_infer_request() → InferRequest#

Creates an inference request object used to infer the compiled model.

The created request has allocated input and output tensors.

Returns:

New InferRequest object.

Return type:

openvino.runtime.InferRequest

export_model(*args, **kwargs)#

Overloaded function.

  1. export_model(self: openvino._pyopenvino.CompiledModel) -> bytes

    Exports the compiled model to bytes/output stream.

    GIL is released while running this function.

    return:

    Bytes object that contains this compiled model.

    rtype:

    bytes

    user_stream = compiled.export_model()

with open('./my_model', 'wb') as f:
    f.write(user_stream)

# ...

new_compiled = core.import_model(user_stream, "CPU")

  2. export_model(self: openvino._pyopenvino.CompiledModel, model_stream: object) -> None

    Exports the compiled model to bytes/output stream.

    Advanced version of export_model. It utilizes, streams from the standard Python library io.

    Function performs flushing of the stream, writes to it, and then rewinds the stream to the beginning (using seek(0)).

    GIL is released while running this function.

    param model_stream:

    A stream object to which the model will be serialized.

    type model_stream:

    io.BytesIO

    rtype:

    None

    user_stream = io.BytesIO()
compiled.export_model(user_stream)

with open('./my_model', 'wb') as f:
    f.write(user_stream.getvalue()) # or read() if seek(0) was applied before

# ...

new_compiled = core.import_model(user_stream, "CPU")
get_property(self: openvino._pyopenvino.CompiledModel, property: str) → object#

Gets properties for current compiled model.

Parameters:

name (str) – Property name.

Return type:

Any

get_runtime_model(self: openvino._pyopenvino.CompiledModel) → openvino._pyopenvino.Model#

Gets runtime model information from a device.

This object (returned model) represents the internal device-specific model which is optimized for the particular accelerator. It contains device-specific nodes, runtime information, and can be used only to understand how the source model is optimized and which kernels, element types, and layouts are selected.

Returns:

Model, containing Executable Graph information.

Return type:

openvino.runtime.Model

infer_new_request(inputs: Any | None = None) → OVDict#

Infers specified input(s) in synchronous mode.

Blocks all methods of CompiledModel while request is running.

Method creates new temporary InferRequest and run inference on it. It is advised to use a dedicated InferRequest class for performance, optimizing workflows, and creating advanced pipelines.

The allowed types of keys in the inputs dictionary are:

  1. int

  2. str

  3. openvino.runtime.ConstOutput

The allowed types of values in the inputs are:

  1. numpy.ndarray and all the types that are castable to it, e.g. torch.Tensor

  2. openvino.runtime.Tensor

Can be called with only one openvino.runtime.Tensor or numpy.ndarray, it will work only with one-input models. When model has more inputs, function throws error.

Parameters:

inputs (Any, optional) – Data to be set on input tensors.

Returns:

Dictionary of results from output tensors with port/int/str keys.

Return type:

OVDict

input(*args, **kwargs)#

Overloaded function.

  1. input(self: openvino._pyopenvino.CompiledModel) -> openvino._pyopenvino.ConstOutput

    Gets a single input of a compiled model. If a model has more than one input, this method throws an exception.

    return:

    A compiled model input.

    rtype:

    openvino.runtime.ConstOutput

  2. input(self: openvino._pyopenvino.CompiledModel, index: int) -> openvino._pyopenvino.ConstOutput

    Gets input of a compiled model identified by an index. If the input with given index is not found, this method throws an exception.

    param index:

    An input index.

    type index:

    int

    return:

    A compiled model input.

    rtype:

    openvino.runtime.ConstOutput

  3. input(self: openvino._pyopenvino.CompiledModel, tensor_name: str) -> openvino._pyopenvino.ConstOutput

    Gets input of a compiled model identified by a tensor_name. If the input with given tensor name is not found, this method throws an exception.

    param tensor_name:

    An input tensor name.

    type tensor_name:

    str

    return:

    A compiled model input.

    rtype:

    openvino.runtime.ConstOutput

property inputs#

Gets all inputs of a compiled model.

Returns:

Inputs of a compiled model.

Return type:

List[openvino.runtime.ConstOutput]

output(*args, **kwargs)#

Overloaded function.

  1. output(self: openvino._pyopenvino.CompiledModel) -> openvino._pyopenvino.ConstOutput

    Gets a single output of a compiled model. If the model has more than one output, this method throws an exception.

    return:

    A compiled model output.

    rtype:

    openvino.runtime.ConstOutput

  2. output(self: openvino._pyopenvino.CompiledModel, index: int) -> openvino._pyopenvino.ConstOutput

    Gets output of a compiled model identified by an index. If the output with given index is not found, this method throws an exception.

    param index:

    An output index.

    type index:

    int

    return:

    A compiled model output.

    rtype:

    openvino.runtime.ConstOutput

  3. output(self: openvino._pyopenvino.CompiledModel, tensor_name: str) -> openvino._pyopenvino.ConstOutput

    Gets output of a compiled model identified by a tensor_name. If the output with given tensor name is not found, this method throws an exception.

    param tensor_name:

    An output tensor name.

    type tensor_name:

    str

    return:

    A compiled model output.

    rtype:

    openvino.runtime.ConstOutput

property outputs#

Gets all outputs of a compiled model.

Returns:

Outputs of a compiled model.

Return type:

List[openvino.runtime.ConstOutput]

query_state() → None#

Gets state control interface for the underlaying infer request.

Returns:

List of VariableState objects.

Return type:

List[openvino.runtime.VariableState]

release_memory(self: openvino._pyopenvino.CompiledModel) → None#

Release intermediate memory.

This method forces the Compiled model to release memory allocated for intermediate structures, e.g. caches, tensors, temporal buffers etc., when possible

reset_state() → None#

Resets all internal variable states of the underlaying infer request.

Resets all internal variable states to a value specified as default for the corresponding ReadValue node.

set_property(*args, **kwargs)#

Overloaded function.

  1. set_property(self: openvino._pyopenvino.CompiledModel, properties: dict[str, object]) -> None

    Sets properties for current compiled model.

    param properties:

    Dict of pairs: (property name, property value)

    type properties:

    dict

    rtype:

    None

  2. set_property(self: openvino._pyopenvino.CompiledModel, property: tuple[str, object]) -> None

    Sets properties for current compiled model.

    param property:

    Tuple of (property name, matching property value).

    type property:

    tuple