Deblur Photos with DeblurGAN-v2 and OpenVINO

This tutorial is also available as a Jupyter notebook that can be cloned directly from GitHub. See the installation guide for instructions to run this tutorial locally on Windows, Linux or macOS. To run without installing anything, click the launch binder button.

Binder Github

This tutorial demonstrates Single Image Motion Deblurring with DeblurGAN-v2 in OpenVINO, by first converting the VITA-Group/DeblurGANv2 model to OpenVINO’s Intermediate Representation (IR) format. Model information can be found here.

What is deblurring?

Deblurring is the task of removing motion blurs that usually occur in photos shot with hand-held cameras when there are moving objects in the scene. Blurs not only reduce the human perception about the quality of the image, but also complicate computer vision analyses.

Kupyn, O., Martyniuk, T., Wu, J., & Wang, Z. (2019). Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 8878-8887).



import json
import shutil
import sys
from pathlib import Path

import cv2
import matplotlib.pyplot as plt
import numpy as np
from PIL import Image
from IPython.display import Markdown, display
from openvino.runtime import Core

from notebook_utils import load_image


# Device to use for inference. For example, "CPU", or "GPU"

# Directory to save the model in
model_dir = Path("model")

# Output directory
output_dir = Path("output")

# The name of the Open Model Zoo model
model_name = "deblurgan-v2"
model_xml_path = model_dir / Path(f"{model_name}.xml")

precision = "FP16"
base_model_dir = Path("~/open_model_zoo_models").expanduser()
omz_cache_dir = Path("~/open_model_zoo_cache").expanduser()

Download DeblurGAN-v2 Model from Open Model Zoo

download_command = (
    f"omz_downloader --name {model_name} --output_dir"
    f" {base_model_dir} --cache_dir {omz_cache_dir}"
display(Markdown(f"Download command: `{download_command}`"))
display(Markdown(f"Downloading {model_name}..."))
! $download_command

Download command: omz_downloader --name deblurgan-v2 --output_dir /opt/home/k8sworker/open_model_zoo_models --cache_dir /opt/home/k8sworker/open_model_zoo_cache

Downloading deblurgan-v2…

################|| Downloading deblurgan-v2 ||################

========== Retrieving /opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/models/ from the cache

========== Retrieving /opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/models/ from the cache

========== Retrieving /opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/models/ from the cache

========== Retrieving /opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/models/ from the cache

========== Retrieving /opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/ckpt/fpn_mobilenet.h5 from the cache

========== Replacing text in /opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/models/
========== Replacing text in /opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/models/
========== Replacing text in /opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/models/

Convert DeblurGAN-v2 Model to OpenVINO IR format

Model Conversion may take a while. Conversion succeeded if the last lines of the output include [ SUCCESS ] Generated IR version 11 model.

convert_command = (
    f"omz_converter --name {model_name} --precisions {precision} "
    f"--download_dir {base_model_dir} --output_dir {base_model_dir}"
display(Markdown(f"Convert command: `{convert_command}`"))
display(Markdown(f"Converting {model_name}..."))

! $convert_command

Convert command: omz_converter --name deblurgan-v2 --precisions FP16 --download_dir /opt/home/k8sworker/open_model_zoo_models --output_dir /opt/home/k8sworker/open_model_zoo_models

Converting deblurgan-v2…

========== Converting deblurgan-v2 to ONNX
Conversion to ONNX command: /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-188/.workspace/scm/ov-notebook/.venv/bin/python -- /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-188/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/internal_scripts/ --model-path=/opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-188/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/models/public/deblurgan-v2 --model-path=/opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2 --model-name=DeblurV2 --import-module=model --input-shape=1,3,736,1312 --output-file=/opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/deblurgan-v2.onnx '--model-param=weights=r"/opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/ckpt/fpn_mobilenet.h5"' '--model-param=model_name="fpn_mobilenet"' --input-names=blur_image --output-names=deblur_image

ONNX check passed successfully.

========== Converting deblurgan-v2 to IR (FP16)
Conversion command: /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-188/.workspace/scm/ov-notebook/.venv/bin/python -- /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-188/.workspace/scm/ov-notebook/.venv/bin/mo --framework=onnx --data_type=FP16 --output_dir=/opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/FP16 --model_name=deblurgan-v2 --input=blur_image --model_name=deblurgan-v2 --reverse_input_channels '--mean_values=blur_image[127.5]' '--scale_values=blur_image[127.5]' --output=deblur_image --input_model=/opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/deblurgan-v2.onnx '--layout=blur_image(NCHW)' '--input_shape=[1, 3, 736, 1312]'

Model Optimizer arguments:
Common parameters:
    - Path to the Input Model:  /opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/deblurgan-v2.onnx
    - Path for generated IR:    /opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/FP16
    - IR output name:   deblurgan-v2
    - Log level:    ERROR
    - Batch:    Not specified, inherited from the model
    - Input layers:     blur_image
    - Output layers:    deblur_image
    - Input shapes:     [1, 3, 736, 1312]
    - Source layout:    Not specified
    - Target layout:    Not specified
    - Layout:   blur_image(NCHW)
    - Mean values:  blur_image[127.5]
    - Scale values:     blur_image[127.5]
    - Scale factor:     Not specified
    - Precision of IR:  FP16
    - Enable fusing:    True
    - User transformations:     Not specified
    - Reverse input channels:   True
    - Enable IR generation for fixed input shape:   False
    - Use the transformations config file:  None
Advanced parameters:
    - Force the usage of legacy Frontend of Model Optimizer for model conversion into IR:   False
    - Force the usage of new Frontend of Model Optimizer for model conversion into IR:  False
OpenVINO runtime found in:  /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-188/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino
OpenVINO runtime version:   2022.1.0-7019-cdb9bec7210-releases/2022/1
Model Optimizer version:    2022.1.0-7019-cdb9bec7210-releases/2022/1
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/FP16/deblurgan-v2.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/FP16/deblurgan-v2.bin
[ SUCCESS ] Total execution time: 0.85 seconds.
[ SUCCESS ] Memory consumed: 129 MB.
It's been a while, check for a new version of Intel(R) Distribution of OpenVINO(TM) toolkit here or on the GitHub*
[ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11.
Find more information about API v2.0 and IR v11 at

Copy Model

Copy the DeblurGAN-v2 Model to the specified model_dir directory in the settings.

model_info = %sx omz_info_dumper --name $model_name
model_info_json = json.loads(model_info.get_nlstr())[0]
model_downloaded_dir = (
    base_model_dir / Path(model_info_json["subdirectory"]) / Path(precision)

for model_file in model_downloaded_dir.iterdir():
        shutil.copyfile(model_file, model_dir /
    except FileExistsError:

Load the Model

Load and compile the DeblurGAN-v2 model in the Inference Engine with ie.read_model and compile it for the specified device with ie.compile_model. Get input and output keys and the expected input shape for the model.

ie = Core()
model = ie.read_model(model=model_xml_path)
compiled_model = ie.compile_model(model=model, device_name=DEVICE)
model_input_layer = compiled_model.input(0)
model_output_layer = compiled_model.output(0)
<ConstOutput: names[blur_image] shape{1,3,736,1312} type: f32>
<ConstOutput: names[deblur_image] shape{1,3,736,1312} type: f32>

Deblur Image

Load, resize and reshape input image

The input image is read using the default load_image function from notebooks.utils, resized to meet the network’s expected input sizes, and reshaped to (N, C, H, W), where N=number of images in the batch, C=number of channels, H=height, and W=width.

# Image filename (local path or URL)
filename = ""
# Load input image
image = load_image(filename)

# In case immage is stored in RGBA format
image = np.asarray(Image.fromarray(image).convert('RGB'))

# N,C,H,W = batch size, number of channels, height, width
N, C, H, W = model_input_layer.shape

# Resize image to meet network expected input sizes
resized_image = cv2.resize(image, (W, H))

# Reshape to network input shape
input_image = np.expand_dims(resized_image.transpose(2, 0, 1), 0)
plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
<matplotlib.image.AxesImage at 0x7f6a6c833880>

Do inference on input image

Do the inference, convert the result to an image shape and resize it to the original image size.

# Inference
result = compiled_model([input_image])[model_output_layer]

# Convert result to an image shape
result_image = result[0].transpose((1, 2, 0))

# Resize to original image size
resized_result_image = cv2.resize(result_image, image.shape[:2][::-1])
plt.imshow(cv2.cvtColor(result_image, cv2.COLOR_BGR2RGB))
<matplotlib.image.AxesImage at 0x7f6a68048790>

Load groundtruth image

Load the groundtruth image for comparison.

# Load grountruth image
groundtruth_image = load_image("")

Display results

Images are in BGR format, so they will be converted to the RGB format in order to be properly displayed by matplotlib.

# Create subplot(r,c) by providing the no. of rows (r),
# number of columns (c) and figure size
f, ax = plt.subplots(1, 3, figsize=(20, 15))

# Use the created array and display the images horizontally.
ax[0].imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))

ax[1].imshow(cv2.cvtColor(resized_result_image, cv2.COLOR_BGR2RGB))

ax[2].imshow(cv2.cvtColor(groundtruth_image, cv2.COLOR_BGR2RGB))
<matplotlib.image.AxesImage at 0x7f6a507655e0>

Save the deblurred image

Save the output image of the DeblurGAN-v2 model in the output_dir directory.

savename = "deblurred.png"
    output_dir / Path(savename),
    cv2.cvtColor(resized_result_image, cv2.COLOR_BGR2RGB),