Working with Open Model Zoo Models

This tutorial is also available as a Jupyter notebook that can be cloned directly from GitHub. See the installation guide for instructions to run this tutorial locally on Windows, Linux or macOS. To run without installing anything, click the launch binder button.

This tutorial shows how to download a model from Open Model Zoo, convert it to OpenVINO™ IR format, show information about the model, and benchmark the model.

OpenVINO and Open Model Zoo Tools

OpenVINO and Open Model Zoo tools are listed in the table below.

Tool	Command	Description
Model Downloader	omz_download er	Download models from Open Model Zoo.
Model Converter	omz_converte r	Convert Open Model Zoo models to OpenVINO’s IR format.
Info Dumper	omz_info_dum per	Print information about Open Model Zoo models.
Benchmark Tool	benchmark_ap p	Benchmark model performance by computing inference time.

Preparation

Model Name

Set model_name to the name of the Open Model Zoo model to use in this notebook. Refer to the list of public and Intel pre-trained models for a full list of models that can be used. Set model_name to the model you want to use.

# model_name = "resnet-50-pytorch"
model_name = "mobilenet-v2-pytorch"

Imports

import json
import sys
from pathlib import Path

from IPython.display import Markdown, display
from openvino.runtime import Core

sys.path.append("../utils")
from notebook_utils import DeviceNotFoundAlert, NotebookAlert

Settings and Configuration

Set the file and directory paths. By default, this notebook downloads models from Open Model Zoo to the open_model_zoo_models directory in your $HOME directory. On Windows, the $HOME directory is usually c:\users\username, on Linux /home/username. To change the folder, change base_model_dir in the cell below.

The following settings can be changed:

• base_model_dir: Models will be downloaded into the intel and public folders in this directory.

• omz_cache_dir: Cache folder for Open Model Zoo. Specifying a cache directory is not required for Model Downloader and Model Converter, but it speeds up subsequent downloads.

• precision: If specified, only models with this precision will be downloaded and converted.

base_model_dir = Path("model")
omz_cache_dir = Path("cache")
precision = "FP16"

# Check if an iGPU is available on this system to use with Benchmark App.
ie = Core()
gpu_available = "GPU" in ie.available_devices

print(
    f"base_model_dir: {base_model_dir}, omz_cache_dir: {omz_cache_dir}, gpu_availble: {gpu_available}"
)

base_model_dir: model, omz_cache_dir: cache, gpu_availble: False

Download a Model from Open Model Zoo

import shutil

if Path("open_model_zoo").exists():
    shutil.rmtree("open_model_zoo")

!git clone https://github.com/openvinotoolkit/open_model_zoo.git
%cd open_model_zoo
!git checkout aa02473c20c1b62763de5385229ffde476e8119c
%cd tools/model_tools

!pip install .
%cd ../../../

Cloning into 'open_model_zoo'...
remote: Enumerating objects: 103165, done.
remote: Counting objects: 100% (889/889), done.
remote: Compressing objects: 100% (525/525), done.
remote: Total 103165 (delta 291), reused 825 (delta 268), pack-reused 102276
Receiving objects: 100% (103165/103165), 303.37 MiB | 3.87 MiB/s, done.
Resolving deltas: 100% (70208/70208), done.
/opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/notebooks/104-model-tools/open_model_zoo
Note: switching to 'aa02473c20c1b62763de5385229ffde476e8119c'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at aa02473c2 Merge pull request #3621 from eaidova/ea/fix_imports_order
/opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/notebooks/104-model-tools/open_model_zoo/tools/model_tools
Processing /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/notebooks/104-model-tools/open_model_zoo/tools/model_tools
  Installing build dependencies ... -  | done
  Getting requirements to build wheel ... - done
  Preparing metadata (pyproject.toml) ... - done
Requirement already satisfied: openvino-telemetry>=2022.1.0 in /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from omz-tools==1.0.2) (2022.3.0)
Requirement already satisfied: pyyaml>=5.4.1 in /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from omz-tools==1.0.2) (6.0)
Requirement already satisfied: requests>=2.25.1 in /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from omz-tools==1.0.2) (2.31.0)
Requirement already satisfied: charset-normalizer<4,>=2 in /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests>=2.25.1->omz-tools==1.0.2) (3.1.0)
Requirement already satisfied: idna<4,>=2.5 in /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests>=2.25.1->omz-tools==1.0.2) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests>=2.25.1->omz-tools==1.0.2) (1.26.16)
Requirement already satisfied: certifi>=2017.4.17 in /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests>=2.25.1->omz-tools==1.0.2) (2023.5.7)
Building wheels for collected packages: omz-tools
  Building wheel for omz-tools (pyproject.toml) ... -  | / -  done
  Created wheel for omz-tools: filename=omz_tools-1.0.2-py3-none-any.whl size=3364798 sha256=3180da2c121c85b9de8ca19bcdd1b0d774d203a28f153a893b993431c328c30b
  Stored in directory: /tmp/pip-ephem-wheel-cache-tlfkktz2/wheels/fa/a1/e1/1a73fbbacdfff3d1371ea61c64d2882179d286cc1c8521f629
Successfully built omz-tools
Installing collected packages: omz-tools
Successfully installed omz-tools-1.0.2
/opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/notebooks/104-model-tools

Specify, display and run the Model Downloader command to download the model.

## Uncomment the next line to show help in omz_downloader which explains the command-line options.

# !omz_downloader --help

download_command = (
    f"omz_downloader --name {model_name} --output_dir {base_model_dir} --cache_dir {omz_cache_dir}"
)
display(Markdown(f"Download command: `{download_command}`"))
display(Markdown(f"Downloading {model_name}..."))
! $download_command

Download command: omz_downloader --name mobilenet-v2-pytorch --output_dir model --cache_dir cache

Downloading mobilenet-v2-pytorch…

################|| Downloading mobilenet-v2-pytorch ||################

========== Downloading model/public/mobilenet-v2-pytorch/mobilenet_v2-b0353104.pth

Convert a Model to OpenVINO IR format

Specify, display and run the Model Converter command to convert the model to OpenVINO IR format. Model conversion may take a while. The output of the Model Converter command will be displayed. When the conversion is successful, the last lines of the output will include: [ SUCCESS ] Generated IR version 11 model. For downloaded models that are already in OpenVINO IR format, conversion will be skipped.

## Uncomment the next line to show Help in omz_converter which explains the command-line options.

# !omz_converter --help

convert_command = f"omz_converter --name {model_name} --precisions {precision} --download_dir {base_model_dir} --output_dir {base_model_dir}"
display(Markdown(f"Convert command: `{convert_command}`"))
display(Markdown(f"Converting {model_name}..."))

! $convert_command

Convert command: omz_converter --name mobilenet-v2-pytorch --precisions FP16 --download_dir model --output_dir model

Converting mobilenet-v2-pytorch…

========== Converting mobilenet-v2-pytorch to ONNX
Conversion to ONNX command: /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/.venv/bin/python -- /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/internal_scripts/pytorch_to_onnx.py --model-name=mobilenet_v2 --weights=model/public/mobilenet-v2-pytorch/mobilenet_v2-b0353104.pth --import-module=torchvision.models --input-shape=1,3,224,224 --output-file=model/public/mobilenet-v2-pytorch/mobilenet-v2.onnx --input-names=data --output-names=prob

Traceback (most recent call last):
  File "/opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/internal_scripts/pytorch_to_onnx.py", line 187, in <module>
    main()
  File "/opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/internal_scripts/pytorch_to_onnx.py", line 179, in main
    model = load_model(args.model_name, args.weights,
  File "/opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/internal_scripts/pytorch_to_onnx.py", line 126, in load_model
    with prepend_to_path(model_paths):
  File "/opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/internal_scripts/pytorch_to_onnx.py", line 49, in __enter__
    sys.path = self._preprended_paths + sys.path
TypeError: unsupported operand type(s) for +: 'NoneType' and 'list'

FAILED:
mobilenet-v2-pytorch

Get Model Information

The Info Dumper prints the following information for Open Model Zoo models:

• Model name

• Description

• Framework that was used to train the model

• License URL

• Precisions supported by the model

• Subdirectory: the location of the downloaded model

• Task type

This information can be shown by running omz_info_dumper --name model_name in a terminal. The information can also be parsed and used in scripts.

In the next cell, run Info Dumper and use json to load the information in a dictionary.

model_info_output = %sx omz_info_dumper --name $model_name
model_info = json.loads(model_info_output.get_nlstr())

if len(model_info) > 1:
    NotebookAlert(
        f"There are multiple IR files for the {model_name} model. The first model in the "
        "omz_info_dumper output will be used for benchmarking. Change "
        "`selected_model_info` in the cell below to select a different model from the list.",
        "warning",
    )

model_info

[{'name': 'mobilenet-v2-pytorch',
  'composite_model_name': None,
  'description': 'MobileNet V2 is image classification model pre-trained on ImageNet dataset. This is a PyTorch* implementation of MobileNetV2 architecture as described in the paper "Inverted Residuals and Linear Bottlenecks: Mobile Networks for Classification, Detection and Segmentation" <https://arxiv.org/abs/1801.04381>.nThe model input is a blob that consists of a single image of "1, 3, 224, 224" in "RGB" order.nThe model output is typical object classifier for the 1000 different classifications matching with those in the ImageNet database.',
  'framework': 'pytorch',
  'license_url': 'https://raw.githubusercontent.com/pytorch/vision/master/LICENSE',
  'accuracy_config': '/opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/models/public/mobilenet-v2-pytorch/accuracy-check.yml',
  'model_config': '/opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/models/public/mobilenet-v2-pytorch/model.yml',
  'precisions': ['FP16', 'FP32'],
  'quantization_output_precisions': ['FP16-INT8', 'FP32-INT8'],
  'subdirectory': 'public/mobilenet-v2-pytorch',
  'task_type': 'classification',
  'input_info': [{'name': 'data',
    'shape': [1, 3, 224, 224],
    'layout': 'NCHW'}],
  'model_stages': []}]

Having information of the model in a JSON file enables extraction of the path to the model directory, and building the path to the OpenVINO IR file.

selected_model_info = model_info[0]
model_path = (
    base_model_dir
    / Path(selected_model_info["subdirectory"])
    / Path(f"{precision}/{selected_model_info['name']}.xml")
)
print(model_path, "exists:", model_path.exists())

model/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml exists: False

Run Benchmark Tool

By default, Benchmark Tool runs inference for 60 seconds in asynchronous mode on CPU. It returns inference speed as latency (milliseconds per image) and throughput values (frames per second).

## Uncomment the next line to show Help in benchmark_app which explains the command-line options.
# !benchmark_app --help

benchmark_command = f"benchmark_app -m {model_path} -t 15"
display(Markdown(f"Benchmark command: `{benchmark_command}`"))
display(Markdown(f"Benchmarking {model_name} on CPU with async inference for 15 seconds..."))

! $benchmark_command

Benchmark command: benchmark_app -m model/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml -t 15

Benchmarking mobilenet-v2-pytorch on CPU with async inference for 15 seconds…

[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2022.3.0-9052-9752fafe8eb-releases/2022/3
[ INFO ]
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2022.3.0-9052-9752fafe8eb-releases/2022/3
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ ERROR ] Model file /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/notebooks/104-model-tools/model/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml cannot be opened!
Traceback (most recent call last):
  File "/opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 368, in main
    model = benchmark.read_model(args.path_to_model)
  File "/opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/benchmark.py", line 69, in read_model
    return self.core.read_model(model_filename, weights_filename)
RuntimeError: Model file /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/notebooks/104-model-tools/model/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml cannot be opened!

Benchmark with Different Settings

The benchmark_app tool displays logging information that is not always necessary. A more compact result is achieved when the output is parsed with json.

The following cells show some examples of benchmark_app with different parameters. Below are some useful parameters:

• -d A device to use for inference. For example: CPU, GPU, MULTI. Default: CPU.

• -t Time expressed in number of seconds to run inference. Default: 60.

• -api Use asynchronous (async) or synchronous (sync) inference. Default: async.

• -b Batch size. Default: 1.

Run ! benchmark_app --help to get an overview of all possible command-line parameters.

In the next cell, define the benchmark_model() function that calls benchmark_app. This makes it easy to try different combinations. In the cell below that, you display available devices on the system.

Note: In this notebook, benchmark_app runs for 15 seconds to give a quick indication of performance. For more accurate performance, it is recommended to run inference for at least one minute by setting the t parameter to 60 or higher, and run benchmark_app in a terminal/command prompt after closing other applications. Copy the benchmark command and paste it in a command prompt where you have activated the openvino_env environment.

def benchmark_model(model_xml, device="CPU", seconds=60, api="async", batch=1):
    ie = Core()
    model_path = Path(model_xml)
    if ("GPU" in device) and ("GPU" not in ie.available_devices):
        DeviceNotFoundAlert("GPU")
    else:
        benchmark_command = f"benchmark_app -m {model_path} -d {device} -t {seconds} -api {api} -b {batch}"
        display(Markdown(f"**Benchmark {model_path.name} with {device} for {seconds} seconds with {api} inference**"))
        display(Markdown(f"Benchmark command: `{benchmark_command}`"))

        benchmark_output = %sx $benchmark_command
        print("command ended")
        benchmark_result = [line for line in benchmark_output
                            if not (line.startswith(r"[") or line.startswith("      ") or line == "")]
        print("\n".join(benchmark_result))

ie = Core()

# Show devices available for OpenVINO Runtime
for device in ie.available_devices:
    device_name = ie.get_property(device, "FULL_DEVICE_NAME")
    print(f"{device}: {device_name}")

CPU: Intel(R) Core(TM) i9-10920X CPU @ 3.50GHz

benchmark_model(model_path, device="CPU", seconds=15, api="async")

Benchmark mobilenet-v2-pytorch.xml with CPU for 15 seconds with async inference

Benchmark command: benchmark_app -m model/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml -d CPU -t 15 -api async -b 1

command ended
Traceback (most recent call last):
  File "/opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 368, in main
    model = benchmark.read_model(args.path_to_model)
  File "/opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/benchmark.py", line 69, in read_model
    return self.core.read_model(model_filename, weights_filename)
RuntimeError: Model file /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/notebooks/104-model-tools/model/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml cannot be opened!

benchmark_model(model_path, device="AUTO", seconds=15, api="async")

Benchmark mobilenet-v2-pytorch.xml with AUTO for 15 seconds with async inference

Benchmark command: benchmark_app -m model/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml -d AUTO -t 15 -api async -b 1

command ended
Traceback (most recent call last):
  File "/opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/main.py", line 368, in main
    model = benchmark.read_model(args.path_to_model)
  File "/opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/tools/benchmark/benchmark.py", line 69, in read_model
    return self.core.read_model(model_filename, weights_filename)
RuntimeError: Model file /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-416/.workspace/scm/ov-notebook/notebooks/104-model-tools/model/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml cannot be opened!

benchmark_model(model_path, device="GPU", seconds=15, api="async")

Running this cell requires a GPU device, which is not available on this system. The following device is available: CPU

benchmark_model(model_path, device="MULTI:CPU,GPU", seconds=15, api="async")

Running this cell requires a GPU device, which is not available on this system. The following device is available: CPU