Working with Open Model Zoo Models#

This Jupyter notebook can be launched on-line, opening an interactive environment in a browser window. You can also make a local installation. Choose one of the following options:

BinderGoogle ColabGithub

This tutorial shows how to download a model from Open Model Zoo, convert it to OpenVINO™ IR format, show information about the model, and benchmark the model.

Table of contents:#

OpenVINO and Open Model Zoo Tools#

OpenVINO and Open Model Zoo tools are listed in the table below.

Tool

Command

Description

Model Downloader

omz_downlo ader

Download models from Open Model Zoo.

Model Converter

omz_conver ter

Convert Open Model Zoo models to OpenVINO’s IR format.

Info Dumper

omz_info_d umper

Print information about Open Model Zoo models.

Benchmark Tool

benchmark_ app

Benchmark model performance by computing inference time.

# Install openvino package
%pip install -q "openvino-dev>=2024.0.0" torch torchvision --extra-index-url https://download.pytorch.org/whl/cpu
Note: you may need to restart the kernel to use updated packages.

Preparation#

Model Name#

Set model_name to the name of the Open Model Zoo model to use in this notebook. Refer to the list of public and Intel pre-trained models for a full list of models that can be used. Set model_name to the model you want to use.

# model_name = "resnet-50-pytorch"
model_name = "mobilenet-v2-pytorch"

Imports#

import json
from pathlib import Path

import openvino as ov
from IPython.display import Markdown, display

# Fetch `notebook_utils` module
import requests

r = requests.get(
    url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py",
)

open("notebook_utils.py", "w").write(r.text)
from notebook_utils import DeviceNotFoundAlert, NotebookAlert

Settings and Configuration#

Set the file and directory paths. By default, this notebook downloads models from Open Model Zoo to the open_model_zoo_models directory in your $HOME directory. On Windows, the $HOME directory is usually c:\users\username, on Linux /home/username. To change the folder, change base_model_dir in the cell below.

The following settings can be changed:

  • base_model_dir: Models will be downloaded into the intel and public folders in this directory.

  • omz_cache_dir: Cache folder for Open Model Zoo. Specifying a cache directory is not required for Model Downloader and Model Converter, but it speeds up subsequent downloads.

  • precision: If specified, only models with this precision will be downloaded and converted.

base_model_dir = Path("model")
omz_cache_dir = Path("cache")
precision = "FP16"

# Check if an GPU is available on this system to use with Benchmark App.
core = ov.Core()
gpu_available = "GPU" in core.available_devices

print(f"base_model_dir: {base_model_dir}, omz_cache_dir: {omz_cache_dir}, gpu_availble: {gpu_available}")
base_model_dir: model, omz_cache_dir: cache, gpu_availble: False

Download a Model from Open Model Zoo#

Specify, display and run the Model Downloader command to download the model.

## Uncomment the next line to show help in omz_downloader which explains the command-line options.

# !omz_downloader --help
download_command = f"omz_downloader --name {model_name} --output_dir {base_model_dir} --cache_dir {omz_cache_dir}"
display(Markdown(f"Download command: `{download_command}`"))
display(Markdown(f"Downloading {model_name}..."))
! $download_command

Download command: omz_downloader --name mobilenet-v2-pytorch --output_dir model --cache_dir cache

Downloading mobilenet-v2-pytorch…

################|| Downloading mobilenet-v2-pytorch ||################

========== Downloading model/public/mobilenet-v2-pytorch/mobilenet_v2-b0353104.pth

Convert a Model to OpenVINO IR format#

Specify, display and run the Model Converter command to convert the model to OpenVINO IR format. Model conversion may take a while. The output of the Model Converter command will be displayed. When the conversion is successful, the last lines of the output will include: [ SUCCESS ] Generated IR version 11 model. For downloaded models that are already in OpenVINO IR format, conversion will be skipped.

## Uncomment the next line to show Help in omz_converter which explains the command-line options.

# !omz_converter --help
convert_command = f"omz_converter --name {model_name} --precisions {precision} --download_dir {base_model_dir} --output_dir {base_model_dir}"
display(Markdown(f"Convert command: `{convert_command}`"))
display(Markdown(f"Converting {model_name}..."))

! $convert_command

Convert command: omz_converter --name mobilenet-v2-pytorch --precisions FP16 --download_dir model --output_dir model

Converting mobilenet-v2-pytorch…

========== Converting mobilenet-v2-pytorch to ONNX
Conversion to ONNX command: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-697/.workspace/scm/ov-notebook/.venv/bin/python -- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-697/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/omz_tools/internal_scripts/pytorch_to_onnx.py --model-name=mobilenet_v2 --weights=model/public/mobilenet-v2-pytorch/mobilenet_v2-b0353104.pth --import-module=torchvision.models --input-shape=1,3,224,224 --output-file=model/public/mobilenet-v2-pytorch/mobilenet-v2.onnx --input-names=data --output-names=prob

ONNX check passed successfully.

========== Converting mobilenet-v2-pytorch to IR (FP16)
Conversion command: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-697/.workspace/scm/ov-notebook/.venv/bin/python -- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-697/.workspace/scm/ov-notebook/.venv/bin/mo --framework=onnx --output_dir=model/public/mobilenet-v2-pytorch/FP16 --model_name=mobilenet-v2-pytorch --input=data '--mean_values=data[123.675,116.28,103.53]' '--scale_values=data[58.624,57.12,57.375]' --reverse_input_channels --output=prob --input_model=model/public/mobilenet-v2-pytorch/mobilenet-v2.onnx '--layout=data(NCHW)' '--input_shape=[1, 3, 224, 224]' --compress_to_fp16=True

[ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False.
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
[ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API.
Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-697/.workspace/scm/ov-notebook/notebooks/model-tools/model/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-697/.workspace/scm/ov-notebook/notebooks/model-tools/model/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.bin

Get Model Information#

The Info Dumper prints the following information for Open Model Zoo models:

  • Model name

  • Description

  • Framework that was used to train the model

  • License URL

  • Precisions supported by the model

  • Subdirectory: the location of the downloaded model

  • Task type

This information can be shown by running omz_info_dumper --name model_name in a terminal. The information can also be parsed and used in scripts.

In the next cell, run Info Dumper and use json to load the information in a dictionary.

model_info_output = %sx omz_info_dumper --name $model_name
model_info = json.loads(model_info_output.get_nlstr())

if len(model_info) > 1:
    NotebookAlert(
        f"There are multiple IR files for the {model_name} model. The first model in the "
        "omz_info_dumper output will be used for benchmarking. Change "
        "`selected_model_info` in the cell below to select a different model from the list.",
        "warning",
    )

model_info
[{'name': 'mobilenet-v2-pytorch',
  'composite_model_name': None,
  'description': 'MobileNet V2 is image classification model pre-trained on ImageNet dataset. This is a PyTorch* implementation of MobileNetV2 architecture as described in the paper "Inverted Residuals and Linear Bottlenecks: Mobile Networks for Classification, Detection and Segmentation" <https://arxiv.org/abs/1801.04381>.nThe model input is a blob that consists of a single image of "1, 3, 224, 224" in "RGB" order.nThe model output is typical object classifier for the 1000 different classifications matching with those in the ImageNet database.',
  'framework': 'pytorch',
  'license_url': 'https://raw.githubusercontent.com/pytorch/vision/master/LICENSE',
  'accuracy_config': '/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-697/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/omz_tools/models/public/mobilenet-v2-pytorch/accuracy-check.yml',
  'model_config': '/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-697/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/omz_tools/models/public/mobilenet-v2-pytorch/model.yml',
  'precisions': ['FP16', 'FP32'],
  'subdirectory': 'public/mobilenet-v2-pytorch',
  'task_type': 'classification',
  'input_info': [{'name': 'data',
    'shape': [1, 3, 224, 224],
    'layout': 'NCHW'}],
  'model_stages': []}]

Having information of the model in a JSON file enables extraction of the path to the model directory, and building the path to the OpenVINO IR file.

selected_model_info = model_info[0]
model_path = base_model_dir / Path(selected_model_info["subdirectory"]) / Path(f"{precision}/{selected_model_info['name']}.xml")
print(model_path, "exists:", model_path.exists())
model/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml exists: True

Run Benchmark Tool#

By default, Benchmark Tool runs inference for 60 seconds in asynchronous mode on CPU. It returns inference speed as latency (milliseconds per image) and throughput values (frames per second).

## Uncomment the next line to show Help in benchmark_app which explains the command-line options.
# !benchmark_app --help
benchmark_command = f"benchmark_app -m {model_path} -t 15"
display(Markdown(f"Benchmark command: `{benchmark_command}`"))
display(Markdown(f"Benchmarking {model_name} on CPU with async inference for 15 seconds..."))

! $benchmark_command

Benchmark command: benchmark_app -m model/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml -t 15

Benchmarking mobilenet-v2-pytorch on CPU with async inference for 15 seconds…

[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 27.44 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ]     data (node: data) : f32 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Model outputs:
[ INFO ]     prob (node: prob) : f32 / [...] / [1,1000]
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ]     data (node: data) : u8 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Model outputs:
[ INFO ]     prob (node: prob) : f32 / [...] / [1,1000]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 162.51 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ]   NETWORK_NAME: main_graph
[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 6
[ INFO ]   NUM_STREAMS: 6
[ INFO ]   AFFINITY: Affinity.CORE
[ INFO ]   INFERENCE_NUM_THREADS: 24
[ INFO ]   PERF_COUNT: NO
[ INFO ]   INFERENCE_PRECISION_HINT: <Type: 'float32'>
[ INFO ]   PERFORMANCE_HINT: THROUGHPUT
[ INFO ]   EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE
[ INFO ]   PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ]   ENABLE_CPU_PINNING: True
[ INFO ]   SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
[ INFO ]   MODEL_DISTRIBUTION_POLICY: set()
[ INFO ]   ENABLE_HYPER_THREADING: True
[ INFO ]   EXECUTION_DEVICES: ['CPU']
[ INFO ]   CPU_DENORMALS_OPTIMIZATION: False
[ INFO ]   LOG_LEVEL: Level.NO
[ INFO ]   CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0
[ INFO ]   DYNAMIC_QUANTIZATION_GROUP_SIZE: 0
[ INFO ]   KV_CACHE_PRECISION: <Type: 'float16'>
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'data'!. This input will be filled with random values!
[ INFO ] Fill input 'data' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 6 inference requests, limits: 15000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 5.52 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count:            20316 iterations
[ INFO ] Duration:         15008.24 ms
[ INFO ] Latency:
[ INFO ]    Median:        4.30 ms
[ INFO ]    Average:       4.30 ms
[ INFO ]    Min:           2.64 ms
[ INFO ]    Max:           12.94 ms
[ INFO ] Throughput:   1353.66 FPS

Benchmark with Different Settings#

The benchmark_app tool displays logging information that is not always necessary. A more compact result is achieved when the output is parsed with json.

The following cells show some examples of benchmark_app with different parameters. Below are some useful parameters:

  • -d A device to use for inference. For example: CPU, GPU, MULTI. Default: CPU.

  • -t Time expressed in number of seconds to run inference. Default: 60.

  • -api Use asynchronous (async) or synchronous (sync) inference. Default: async.

  • -b Batch size. Default: 1.

Run ! benchmark_app --help to get an overview of all possible command-line parameters.

In the next cell, define the benchmark_model() function that calls benchmark_app. This makes it easy to try different combinations. In the cell below that, you display available devices on the system.

Note: In this notebook, benchmark_app runs for 15 seconds to give a quick indication of performance. For more accurate performance, it is recommended to run inference for at least one minute by setting the t parameter to 60 or higher, and run benchmark_app in a terminal/command prompt after closing other applications. Copy the benchmark command and paste it in a command prompt where you have activated the openvino_env environment.

def benchmark_model(model_xml, device="CPU", seconds=60, api="async", batch=1):
    core = ov.Core()
    model_path = Path(model_xml)
    if ("GPU" in device) and ("GPU" not in core.available_devices):
        DeviceNotFoundAlert("GPU")
    else:
        benchmark_command = f"benchmark_app -m {model_path} -d {device} -t {seconds} -api {api} -b {batch}"
        display(Markdown(f"**Benchmark {model_path.name} with {device} for {seconds} seconds with {api} inference**"))
        display(Markdown(f"Benchmark command: `{benchmark_command}`"))

        benchmark_output = %sx $benchmark_command
        print("command ended")
        benchmark_result = [line for line in benchmark_output if not (line.startswith(r"[") or line.startswith("      ") or line == "")]
        print("\n".join(benchmark_result))
core = ov.Core()

# Show devices available for OpenVINO Runtime
for device in core.available_devices:
    device_name = core.get_property(device, "FULL_DEVICE_NAME")
    print(f"{device}: {device_name}")
CPU: Intel(R) Core(TM) i9-10920X CPU @ 3.50GHz

You can select inference device using device widget

import ipywidgets as widgets

device = widgets.Dropdown(
    options=core.available_devices + ["AUTO"],
    value="CPU",
    description="Device:",
    disabled=False,
)

device
Dropdown(description='Device:', options=('CPU', 'AUTO'), value='CPU')
benchmark_model(model_path, device=device.value, seconds=15, api="async")

Benchmark mobilenet-v2-pytorch.xml with CPU for 15 seconds with async inference

Benchmark command: benchmark_app -m model/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml -d CPU -t 15 -api async -b 1

command ended