Working with Open Model Zoo Models

This tutorial is also available as a Jupyter notebook that can be cloned directly from GitHub. See the installation guide for instructions to run this tutorial locally on Windows, Linux or macOS. To run without installing anything, click the launch binder button.

Binder Github

This tutorial shows how to download a model from the Open Model Zoo, convert it to OpenVINO’s IR format, show information about the model, and benchmark the model.

OpenVINO and Open Model Zoo Tools

The OpenVINO and Open Model Zoo tools are listed in the table below.

Tool

Command

Description

Model Downloader

omz_download er

Download models from Open Model Zoo

Model Converter

omz_converte r

Convert Open Model Zoo models to OpenVINO’s IR format

Info Dumper

omz_info_dum per

Print information about Open Model Zoo models

Benchmark Tool

benchmark_ap p

Benchmark model performance by computing inference time

Preparation

Model Name

Set model_name to the name of the Open Model Zoo model to use in this notebook. Refer to the list of public and Intel pre-trained models for a full list of models that can be used. Set the model_name to the model you want to use.

# model_name = "resnet-50-pytorch"
model_name = "mobilenet-v2-pytorch"

Imports

import json
import sys
from pathlib import Path

from IPython.display import Markdown, display
from openvino.inference_engine import IECore

sys.path.append("../utils")
from notebook_utils import DeviceNotFoundAlert, NotebookAlert

Settings and Configuration

Set the file and directory paths. By default, this demo notebook downloads models from Open Model Zoo to a directory open_model_zoo_models in your $HOME directory. On Windows, the $HOME directory is usually c:\users\username, on Linux /home/username. If you want to change the folder, change base_model_dir in the cell below.

You can change the following settings:

  • base_model_dir: Models will be downloaded into the intel and public folders in this directory.

  • omz_cache_dir: Cache folder for Open Model Zoo. Specifying a cache directory is not required for Model Downloader and Model Converter, but it speeds up subsequent downloads.

  • precision: If specified, only models with this precision will be downloaded and converted.

base_model_dir = Path("~/open_model_zoo_models").expanduser()
omz_cache_dir = Path("~/open_model_zoo_cache").expanduser()
precision = "FP16"

# Check if an iGPU is available on this system to use with Benchmark App
ie = IECore()
gpu_available = "GPU" in ie.available_devices

print(
    f"base_model_dir: {base_model_dir}, omz_cache_dir: {omz_cache_dir}, gpu_availble: {gpu_available}"
)
base_model_dir: /home/runner/open_model_zoo_models, omz_cache_dir: /home/runner/open_model_zoo_cache, gpu_availble: False

Download Model from Open Model Zoo

Specify, display and run the Model Downloader command to download the model

## Uncomment the next line to show omz_downloader's help which explains the command line options

# !omz_downloader --help
download_command = (
    f"omz_downloader --name {model_name} --output_dir {base_model_dir} --cache_dir {omz_cache_dir}"
)
display(Markdown(f"Download command: `{download_command}`"))
display(Markdown(f"Downloading {model_name}..."))
! $download_command

Download command: omz_downloader --name mobilenet-v2-pytorch --output_dir /home/runner/open_model_zoo_models --cache_dir /home/runner/open_model_zoo_cache

Downloading mobilenet-v2-pytorch…

################|| Downloading mobilenet-v2-pytorch ||################

========== Downloading /home/runner/open_model_zoo_models/public/mobilenet-v2-pytorch/mobilenet_v2-b0353104.pth

Convert Model to OpenVINO IR format

Specify, display and run the Model Converter command to convert the model to IR format. Model Conversion may take a while. The output of the Model Converter command will be displayed. Conversion succeeded if the last lines of the output include [ SUCCESS ] Generated IR version 10 model. For downloaded models that are already in IR format, conversion will be skipped.

## Uncomment the next line to show omz_converter's help which explains the command line options

# !omz_converter --help
convert_command = f"omz_converter --name {model_name} --precisions {precision} --download_dir {base_model_dir} --output_dir {base_model_dir}"
display(Markdown(f"Convert command: `{convert_command}`"))
display(Markdown(f"Converting {model_name}..."))

! $convert_command

Convert command: omz_converter --name mobilenet-v2-pytorch --precisions FP16 --download_dir /home/runner/open_model_zoo_models --output_dir /home/runner/open_model_zoo_models

Converting mobilenet-v2-pytorch…

========== Converting mobilenet-v2-pytorch to ONNX
Conversion to ONNX command: /opt/hostedtoolcache/Python/3.8.12/x64/bin/python -- /opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/open_model_zoo/model_tools/internal_scripts/pytorch_to_onnx.py --model-name=mobilenet_v2 --weights=/home/runner/open_model_zoo_models/public/mobilenet-v2-pytorch/mobilenet_v2-b0353104.pth --import-module=torchvision.models --input-shape=1,3,224,224 --output-file=/home/runner/open_model_zoo_models/public/mobilenet-v2-pytorch/mobilenet-v2.onnx --input-names=data --output-names=prob

ONNX check passed successfully.

========== Converting mobilenet-v2-pytorch to IR (FP16)
Conversion command: /opt/hostedtoolcache/Python/3.8.12/x64/bin/python -m mo --framework=onnx --data_type=FP16 --output_dir=/home/runner/open_model_zoo_models/public/mobilenet-v2-pytorch/FP16 --model_name=mobilenet-v2-pytorch --input=data '--mean_values=data[123.675,116.28,103.53]' '--scale_values=data[58.624,57.12,57.375]' --reverse_input_channels --output=prob --input_model=/home/runner/open_model_zoo_models/public/mobilenet-v2-pytorch/mobilenet-v2.onnx

Model Optimizer arguments:
Common parameters:
    - Path to the Input Model:  /home/runner/open_model_zoo_models/public/mobilenet-v2-pytorch/mobilenet-v2.onnx
    - Path for generated IR:    /home/runner/open_model_zoo_models/public/mobilenet-v2-pytorch/FP16
    - IR output name:   mobilenet-v2-pytorch
    - Log level:    ERROR
    - Batch:    Not specified, inherited from the model
    - Input layers:     data
    - Output layers:    prob
    - Input shapes:     Not specified, inherited from the model
    - Mean values:  data[123.675,116.28,103.53]
    - Scale values:     data[58.624,57.12,57.375]
    - Scale factor:     Not specified
    - Precision of IR:  FP16
    - Enable fusing:    True
    - Enable grouped convolutions fusing:   True
    - Move mean values to preprocess section:   None
    - Reverse input channels:   True
ONNX specific parameters:
    - Inference Engine found in:    /opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/openvino
Inference Engine version:   2021.4.2-3976-0943ed67223-refs/pull/539/head
Model Optimizer version:    2021.4.2-3976-0943ed67223-refs/pull/539/head
[ SUCCESS ] Generated IR version 10 model.
[ SUCCESS ] XML file: /home/runner/open_model_zoo_models/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml
[ SUCCESS ] BIN file: /home/runner/open_model_zoo_models/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.bin
[ SUCCESS ] Total execution time: 15.59 seconds.
[ SUCCESS ] Memory consumed: 123 MB.

Get Model Information

The Info Dumper prints the following information for Open Model Zoo models:

  • Model name

  • Description

  • Framework that was used to train the model

  • License URL

  • Precisions supported by the model

  • Subdirectory: the location of the downloaded model

  • Task type

This information can be shown by running omz_info_dumper --name model_name in a terminal. The information can also be parsed and used in scripts.

In the next cell, we run Info Dumper and use json to load the information in a dictionary.

model_info_output = %sx omz_info_dumper --name $model_name
model_info = json.loads(model_info_output.get_nlstr())

if len(model_info) > 1:
    NotebookAlert(
        f"There are multiple IR files for the {model_name} model. The first model in the "
        "omz_info_dumper output will be used for benchmarking. Change "
        "`selected_model_info` in the cell below to select a different model from the list.",
        "warning",
    )

model_info
[{'name': 'mobilenet-v2-pytorch',
  'composite_model_name': None,
  'description': 'MobileNet V2 is image classification model pre-trained on ImageNet dataset. This is a PyTorch* implementation of MobileNetV2 architecture as described in the paper "Inverted Residuals and Linear Bottlenecks: Mobile Networks for Classification, Detection and Segmentation" <https://arxiv.org/abs/1801.04381>.nThe model input is a blob that consists of a single image of "1, 3, 224, 224" in "RGB" order.nThe model output is typical object classifier for the 1000 different classifications matching with those in the ImageNet database.',
  'framework': 'pytorch',
  'license_url': 'https://raw.githubusercontent.com/pytorch/vision/master/LICENSE',
  'precisions': ['FP16', 'FP32'],
  'quantization_output_precisions': ['FP16-INT8', 'FP32-INT8'],
  'subdirectory': 'public/mobilenet-v2-pytorch',
  'task_type': 'classification'}]

Having the model information in a JSON file allows us to extract the path to the model directory, and build the path to the IR file.

selected_model_info = model_info[0]
model_path = (
    base_model_dir
    / Path(selected_model_info["subdirectory"])
    / Path(f"{precision}/{selected_model_info['name']}.xml")
)
print(model_path, "exists:", model_path.exists())
/home/runner/open_model_zoo_models/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml exists: True

Run Benchmark Tool

By default, Benchmark Tool runs inference for 60 seconds in asynchronous mode on CPU. It returns inference speed as latency (milliseconds per image) and throughput (frames per second) values.

## Uncomment the next line to show benchmark_app's help which explains the command line options

# !benchmark_app --help
benchmark_command = f"benchmark_app -m {model_path} -t 15"
display(Markdown(f"Benchmark command: `{benchmark_command}`"))
display(Markdown(f"Benchmarking {model_name} on CPU with async inference for 15 seconds..."))

! $benchmark_command

Benchmark command: benchmark_app -m /home/runner/open_model_zoo_models/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml -t 15

Benchmarking mobilenet-v2-pytorch on CPU with async inference for 15 seconds…

[Step 1/11] Parsing and validating input arguments
[ WARNING ]  -nstreams default value is determined automatically for a device. Although the automatic selection usually provides a reasonable performance, but it still may be non-optimal for some cases, for more information look at README.
[Step 2/11] Loading Inference Engine
[ INFO ] InferenceEngine:
         API version............. 2021.4.2-3976-0943ed67223-refs/pull/539/head
[ INFO ] Device info
         CPU
         MKLDNNPlugin............ version 2.1
         Build................... 2021.4.2-3976-0943ed67223-refs/pull/539/head

[Step 3/11] Setting device configuration
[ WARNING ] -nstreams default value is determined automatically for CPU device. Although the automatic selection usually provides a reasonable performance,but it still may be non-optimal for some cases, for more information look at README.
[Step 4/11] Reading network files
[ INFO ] Read network took 18.24 ms
[Step 5/11] Resizing network to match image sizes and given batch
[ INFO ] Network batch size: 1
[Step 6/11] Configuring input of the model
[ INFO ] Network input 'data' precision U8, dimensions (NCHW): 1 3 224 224
[ INFO ] Network output 'prob' precision FP32, dimensions (NC): 1 1000
[Step 7/11] Loading the model to the device
[ INFO ] Load network took 151.33 ms
[Step 8/11] Setting optimal runtime parameters
[Step 9/11] Creating infer requests and filling input blobs with images
[ WARNING ] No input files were given: all inputs will be filled with random values!
[ INFO ] Infer Request 0 filling
[ INFO ] Fill input 'data' with random values (image is expected)
[Step 10/11] Measuring performance (Start inference asynchronously, 1 inference requests using 1 streams for CPU, limits: 15000 ms duration)
[ INFO ] First inference took 10.89 ms
[Step 11/11] Dumping statistics report
Count:      2100 iterations
Duration:   15009.94 ms
Latency:    6.80 ms
Throughput: 139.91 FPS

Benchmark with Different Settings

benchmark_app displays logging information that is not always necessary. We parse the output with json and show a more compact result

The following cells show some examples of benchmark_app with different parameters. Some useful parameters are:

  • -d Device to use for inference. For example: CPU, GPU, MULTI. Default: CPU

  • -t Time in number of seconds to run inference. Default: 60

  • -api Use asynchronous (async) or synchronous (sync) inference. Default: async

  • -b Batch size. Default: 1

Run ! benchmark_app --help to get an overview of all possible command line parameters.

In the next cell, we define a benchmark_model() function that calls benchmark_app. This makes it easy to try different combinations. In the cell below that, we display the available devices on the system.

NOTE: In this notebook we run benchmark_app for 15 seconds to give a quick indication of performance. For more accurate performance, we recommended running inference for at least one minute by setting the t parameter to 60 or higher, and running benchmark_app in a terminal/command prompt after closing other applications. You can copy the benchmark command and paste it in a command prompt where you have activated the openvino_env environment.

def benchmark_model(model_xml, device="CPU", seconds=60, api="async", batch=1):
    ie = IECore()
    model_path = Path(model_xml)
    if ("GPU" in device) and ("GPU" not in ie.available_devices):
        DeviceNotFoundAlert("GPU")
    else:
        benchmark_command = f"benchmark_app -m {model_path} -d {device} -t {seconds} -api {api} -b {batch}"
        display(Markdown(f"**Benchmark {model_path.name} with {device} for {seconds} seconds with {api} inference**"))
        display(Markdown(f"Benchmark command: `{benchmark_command}`"))

        benchmark_output = %sx $benchmark_command
        benchmark_result = [line for line in benchmark_output
                            if not (line.startswith(r"[") or line.startswith("  ") or line == "")]
        print("\n".join(benchmark_result))
ie = IECore()

# Show devices available for OpenVINO Inference Engine
for device in ie.available_devices:
    device_name = ie.get_metric(device, "FULL_DEVICE_NAME")
    print(f"{device}: {device_name}")
CPU: Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
benchmark_model(model_path, device="CPU", seconds=15, api="async")

Benchmark mobilenet-v2-pytorch.xml with CPU for 15 seconds with async inference

Benchmark command: benchmark_app -m /home/runner/open_model_zoo_models/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml -d CPU -t 15 -api async -b 1

Count:      2145 iterations
Duration:   15007.41 ms
Latency:    6.69 ms
Throughput: 142.93 FPS
benchmark_model(model_path, device="AUTO", seconds=15, api="async")

Benchmark mobilenet-v2-pytorch.xml with AUTO for 15 seconds with async inference

Benchmark command: benchmark_app -m /home/runner/open_model_zoo_models/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml -d AUTO -t 15 -api async -b 1

Count:      2103 iterations
Duration:   15008.34 ms
Latency:    6.75 ms
Throughput: 140.12 FPS
benchmark_model(model_path, device="GPU", seconds=15, api="async")
Running this cell requires a GPU device, which is not available on this system. The following device is available: CPU
benchmark_model(model_path, device="MULTI:CPU,GPU", seconds=15, api="async")
Running this cell requires a GPU device, which is not available on this system. The following device is available: CPU