Working with Open Model Zoo Models#
This Jupyter notebook can be launched on-line, opening an interactive environment in a browser window. You can also make a local installation. Choose one of the following options:
This tutorial shows how to download a model from Open Model Zoo, convert it to OpenVINO™ IR format, show information about the model, and benchmark the model.
Table of contents:#
OpenVINO and Open Model Zoo Tools#
OpenVINO and Open Model Zoo tools are listed in the table below.
Tool |
Command |
Description |
---|---|---|
Model Downloader |
|
Download models from Open Model Zoo. |
Model Converter |
|
Convert Open Model Zoo models to OpenVINO’s IR format. |
Info Dumper |
|
Print information about Open Model Zoo models. |
Benchmark Tool |
|
Benchmark model performance by computing inference time. |
# Install openvino package
%pip install -q "openvino-dev>=2024.0.0" torch torchvision --extra-index-url https://download.pytorch.org/whl/cpu
Note: you may need to restart the kernel to use updated packages.
Preparation#
Model Name#
Set model_name
to the name of the Open Model Zoo model to use in
this notebook. Refer to the list of
public
and
Intel
pre-trained models for a full list of models that can be used. Set
model_name
to the model you want to use.
# model_name = "resnet-50-pytorch"
model_name = "mobilenet-v2-pytorch"
Imports#
import json
from pathlib import Path
import openvino as ov
from IPython.display import Markdown, display
# Fetch `notebook_utils` module
import requests
r = requests.get(
url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py",
)
open("notebook_utils.py", "w").write(r.text)
from notebook_utils import DeviceNotFoundAlert, NotebookAlert
Settings and Configuration#
Set the file and directory paths. By default, this notebook downloads
models from Open Model Zoo to the open_model_zoo_models
directory in
your $HOME
directory. On Windows, the $HOME directory is usually
c:\users\username
, on Linux /home/username
. To change the
folder, change base_model_dir
in the cell below.
The following settings can be changed:
base_model_dir
: Models will be downloaded into theintel
andpublic
folders in this directory.omz_cache_dir
: Cache folder for Open Model Zoo. Specifying a cache directory is not required for Model Downloader and Model Converter, but it speeds up subsequent downloads.precision
: If specified, only models with this precision will be downloaded and converted.
base_model_dir = Path("model")
omz_cache_dir = Path("cache")
precision = "FP16"
# Check if an GPU is available on this system to use with Benchmark App.
core = ov.Core()
gpu_available = "GPU" in core.available_devices
print(f"base_model_dir: {base_model_dir}, omz_cache_dir: {omz_cache_dir}, gpu_availble: {gpu_available}")
base_model_dir: model, omz_cache_dir: cache, gpu_availble: False
Download a Model from Open Model Zoo#
Specify, display and run the Model Downloader command to download the model.
## Uncomment the next line to show help in omz_downloader which explains the command-line options.
# !omz_downloader --help
download_command = f"omz_downloader --name {model_name} --output_dir {base_model_dir} --cache_dir {omz_cache_dir}"
display(Markdown(f"Download command: `{download_command}`"))
display(Markdown(f"Downloading {model_name}..."))
! $download_command
Download command:
omz_downloader --name mobilenet-v2-pytorch --output_dir model --cache_dir cache
Downloading mobilenet-v2-pytorch…
################|| Downloading mobilenet-v2-pytorch ||################
========== Downloading model/public/mobilenet-v2-pytorch/mobilenet_v2-b0353104.pth
Convert a Model to OpenVINO IR format#
Specify, display and run the Model Converter command to convert the
model to OpenVINO IR format. Model conversion may take a while. The
output of the Model Converter command will be displayed. When the
conversion is successful, the last lines of the output will include:
[ SUCCESS ] Generated IR version 11 model.
For downloaded models
that are already in OpenVINO IR format, conversion will be skipped.
## Uncomment the next line to show Help in omz_converter which explains the command-line options.
# !omz_converter --help
convert_command = f"omz_converter --name {model_name} --precisions {precision} --download_dir {base_model_dir} --output_dir {base_model_dir}"
display(Markdown(f"Convert command: `{convert_command}`"))
display(Markdown(f"Converting {model_name}..."))
! $convert_command
Convert command:
omz_converter --name mobilenet-v2-pytorch --precisions FP16 --download_dir model --output_dir model
Converting mobilenet-v2-pytorch…
========== Converting mobilenet-v2-pytorch to ONNX
Conversion to ONNX command: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/bin/python -- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/omz_tools/internal_scripts/pytorch_to_onnx.py --model-name=mobilenet_v2 --weights=model/public/mobilenet-v2-pytorch/mobilenet_v2-b0353104.pth --import-module=torchvision.models --input-shape=1,3,224,224 --output-file=model/public/mobilenet-v2-pytorch/mobilenet-v2.onnx --input-names=data --output-names=prob
ONNX check passed successfully.
========== Converting mobilenet-v2-pytorch to IR (FP16)
Conversion command: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/bin/python -- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/bin/mo --framework=onnx --output_dir=model/public/mobilenet-v2-pytorch/FP16 --model_name=mobilenet-v2-pytorch --input=data '--mean_values=data[123.675,116.28,103.53]' '--scale_values=data[58.624,57.12,57.375]' --reverse_input_channels --output=prob --input_model=model/public/mobilenet-v2-pytorch/mobilenet-v2.onnx '--layout=data(NCHW)' '--input_shape=[1, 3, 224, 224]' --compress_to_fp16=True
[ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False.
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
[ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. Please use OpenVINO Model Converter (OVC). OVC represents a lightweight alternative of MO and provides simplified model conversion API.
Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/model-tools/model/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/notebooks/model-tools/model/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.bin
Get Model Information#
The Info Dumper prints the following information for Open Model Zoo models:
Model name
Description
Framework that was used to train the model
License URL
Precisions supported by the model
Subdirectory: the location of the downloaded model
Task type
This information can be shown by running
omz_info_dumper --name model_name
in a terminal. The information can
also be parsed and used in scripts.
In the next cell, run Info Dumper and use json
to load the
information in a dictionary.
model_info_output = %sx omz_info_dumper --name $model_name
model_info = json.loads(model_info_output.get_nlstr())
if len(model_info) > 1:
NotebookAlert(
f"There are multiple IR files for the {model_name} model. The first model in the "
"omz_info_dumper output will be used for benchmarking. Change "
"`selected_model_info` in the cell below to select a different model from the list.",
"warning",
)
model_info
[{'name': 'mobilenet-v2-pytorch', 'composite_model_name': None, 'description': 'MobileNet V2 is image classification model pre-trained on ImageNet dataset. This is a PyTorch* implementation of MobileNetV2 architecture as described in the paper "Inverted Residuals and Linear Bottlenecks: Mobile Networks for Classification, Detection and Segmentation" <https://arxiv.org/abs/1801.04381>.nThe model input is a blob that consists of a single image of "1, 3, 224, 224" in "RGB" order.nThe model output is typical object classifier for the 1000 different classifications matching with those in the ImageNet database.', 'framework': 'pytorch', 'license_url': 'https://raw.githubusercontent.com/pytorch/vision/master/LICENSE', 'accuracy_config': '/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/omz_tools/models/public/mobilenet-v2-pytorch/accuracy-check.yml', 'model_config': '/opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-674/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/omz_tools/models/public/mobilenet-v2-pytorch/model.yml', 'precisions': ['FP16', 'FP32'], 'subdirectory': 'public/mobilenet-v2-pytorch', 'task_type': 'classification', 'input_info': [{'name': 'data', 'shape': [1, 3, 224, 224], 'layout': 'NCHW'}], 'model_stages': []}]
Having information of the model in a JSON file enables extraction of the path to the model directory, and building the path to the OpenVINO IR file.
selected_model_info = model_info[0]
model_path = base_model_dir / Path(selected_model_info["subdirectory"]) / Path(f"{precision}/{selected_model_info['name']}.xml")
print(model_path, "exists:", model_path.exists())
model/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml exists: True
Run Benchmark Tool#
By default, Benchmark Tool runs inference for 60 seconds in asynchronous mode on CPU. It returns inference speed as latency (milliseconds per image) and throughput values (frames per second).
## Uncomment the next line to show Help in benchmark_app which explains the command-line options.
# !benchmark_app --help
benchmark_command = f"benchmark_app -m {model_path} -t 15"
display(Markdown(f"Benchmark command: `{benchmark_command}`"))
display(Markdown(f"Benchmarking {model_name} on CPU with async inference for 15 seconds..."))
! $benchmark_command
Benchmark command:
benchmark_app -m model/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml -t 15
Benchmarking mobilenet-v2-pytorch on CPU with async inference for 15 seconds…
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 27.49 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] data (node: data) : f32 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Model outputs:
[ INFO ] prob (node: prob) : f32 / [...] / [1,1000]
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ] data (node: data) : u8 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Model outputs:
[ INFO ] prob (node: prob) : f32 / [...] / [1,1000]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 145.46 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ] NETWORK_NAME: main_graph
[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 6
[ INFO ] NUM_STREAMS: 6
[ INFO ] AFFINITY: Affinity.CORE
[ INFO ] INFERENCE_NUM_THREADS: 24
[ INFO ] PERF_COUNT: NO
[ INFO ] INFERENCE_PRECISION_HINT: <Type: 'float32'>
[ INFO ] PERFORMANCE_HINT: THROUGHPUT
[ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE
[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ] ENABLE_CPU_PINNING: True
[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
[ INFO ] MODEL_DISTRIBUTION_POLICY: set()
[ INFO ] ENABLE_HYPER_THREADING: True
[ INFO ] EXECUTION_DEVICES: ['CPU']
[ INFO ] CPU_DENORMALS_OPTIMIZATION: False
[ INFO ] LOG_LEVEL: Level.NO
[ INFO ] CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0
[ INFO ] DYNAMIC_QUANTIZATION_GROUP_SIZE: 0
[ INFO ] KV_CACHE_PRECISION: <Type: 'float16'>
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'data'!. This input will be filled with random values!
[ INFO ] Fill input 'data' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 6 inference requests, limits: 15000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 5.83 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count: 20130 iterations
[ INFO ] Duration: 15004.99 ms
[ INFO ] Latency:
[ INFO ] Median: 4.34 ms
[ INFO ] Average: 4.35 ms
[ INFO ] Min: 3.10 ms
[ INFO ] Max: 12.27 ms
[ INFO ] Throughput: 1341.55 FPS
Benchmark with Different Settings#
The benchmark_app
tool displays logging information that is not
always necessary. A more compact result is achieved when the output is
parsed with json
.
The following cells show some examples of benchmark_app
with
different parameters. Below are some useful parameters:
-d
A device to use for inference. For example: CPU, GPU, MULTI. Default: CPU.-t
Time expressed in number of seconds to run inference. Default: 60.-api
Use asynchronous (async) or synchronous (sync) inference. Default: async.-b
Batch size. Default: 1.
Run ! benchmark_app --help
to get an overview of all possible
command-line parameters.
In the next cell, define the benchmark_model()
function that calls
benchmark_app
. This makes it easy to try different combinations. In
the cell below that, you display available devices on the system.
Note: In this notebook,
benchmark_app
runs for 15 seconds to give a quick indication of performance. For more accurate performance, it is recommended to run inference for at least one minute by setting thet
parameter to 60 or higher, and runbenchmark_app
in a terminal/command prompt after closing other applications. Copy the benchmark command and paste it in a command prompt where you have activated theopenvino_env
environment.
def benchmark_model(model_xml, device="CPU", seconds=60, api="async", batch=1):
core = ov.Core()
model_path = Path(model_xml)
if ("GPU" in device) and ("GPU" not in core.available_devices):
DeviceNotFoundAlert("GPU")
else:
benchmark_command = f"benchmark_app -m {model_path} -d {device} -t {seconds} -api {api} -b {batch}"
display(Markdown(f"**Benchmark {model_path.name} with {device} for {seconds} seconds with {api} inference**"))
display(Markdown(f"Benchmark command: `{benchmark_command}`"))
benchmark_output = %sx $benchmark_command
print("command ended")
benchmark_result = [line for line in benchmark_output if not (line.startswith(r"[") or line.startswith(" ") or line == "")]
print("\n".join(benchmark_result))
core = ov.Core()
# Show devices available for OpenVINO Runtime
for device in core.available_devices:
device_name = core.get_property(device, "FULL_DEVICE_NAME")
print(f"{device}: {device_name}")
CPU: Intel(R) Core(TM) i9-10920X CPU @ 3.50GHz
You can select inference device using device widget
import ipywidgets as widgets
device = widgets.Dropdown(
options=core.available_devices + ["AUTO"],
value="CPU",
description="Device:",
disabled=False,
)
device
Dropdown(description='Device:', options=('CPU', 'AUTO'), value='CPU')
benchmark_model(model_path, device=device.value, seconds=15, api="async")
Benchmark mobilenet-v2-pytorch.xml with CPU for 15 seconds with async inference
Benchmark command:
benchmark_app -m model/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml -d CPU -t 15 -api async -b 1
command ended