Deblur Photos with DeblurGAN-v2 and OpenVINO™¶

This tutorial is also available as a Jupyter notebook that can be cloned directly from GitHub. See the installation guide for instructions to run this tutorial locally on Windows, Linux or macOS. To run without installing anything, click the launch binder button.

This tutorial demonstrates Single Image Motion Deblurring with DeblurGAN-v2 in OpenVINO, by first converting the VITA-Group/DeblurGANv2 model to OpenVINO Intermediate Representation (OpenVINO IR) format. For more information about the model, see the documentation.

What is deblurring?¶

Deblurring is the task of removing motion blurs that usually occur in photos shot with hand-held cameras when there are moving objects in the scene. Blurs not only reduce the human perception about the quality of the image, but also complicate computer vision analyses.

For more information, refer to the following research paper:

Kupyn, O., Martyniuk, T., Wu, J., & Wang, Z. (2019). Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 8878-8887).

Preparations¶

Imports¶

import json
import shutil
import sys
from pathlib import Path

import cv2
import matplotlib.pyplot as plt
import numpy as np
from PIL import Image
from IPython.display import Markdown, display
from openvino.runtime import Core

sys.path.append("../utils")
from notebook_utils import load_image

Settings¶

# A device to use for inference. For example, "CPU", or "GPU".
DEVICE = "CPU"

# A directory where the model will be downloaded.
model_dir = Path("model")
model_dir.mkdir(exist_ok=True)

# The output directory.
output_dir = Path("output")
output_dir.mkdir(exist_ok=True)

# The name of the model from Open Model Zoo.
model_name = "deblurgan-v2"
model_xml_path = model_dir / Path(f"{model_name}.xml")

precision = "FP16"
base_model_dir = Path("~/open_model_zoo_models").expanduser()
omz_cache_dir = Path("~/open_model_zoo_cache").expanduser()

Download DeblurGAN-v2 Model from Open Model Zoo¶

download_command = (
    f"omz_downloader --name {model_name} --output_dir"
    f" {base_model_dir} --cache_dir {omz_cache_dir}"
)
display(Markdown(f"Download command: `{download_command}`"))
display(Markdown(f"Downloading {model_name}..."))
! $download_command

Download command: omz_downloader --name deblurgan-v2 --output_dir /opt/home/k8sworker/open_model_zoo_models --cache_dir /opt/home/k8sworker/open_model_zoo_cache

Downloading deblurgan-v2…

################|| Downloading deblurgan-v2 ||################

========== Retrieving /opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/models/__init__.py from the cache

========== Retrieving /opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/models/fpn_mobilenet.py from the cache

========== Retrieving /opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/models/mobilenet_v2.py from the cache

========== Retrieving /opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/models/networks.py from the cache

========== Retrieving /opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/ckpt/fpn_mobilenet.h5 from the cache

========== Replacing text in /opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/models/networks.py
========== Replacing text in /opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/models/fpn_mobilenet.py
========== Replacing text in /opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/models/fpn_mobilenet.py

Convert DeblurGAN-v2 Model to OpenVINO IR format¶

Model Conversion may take a while. When the conversion succeeds, the last lines of the output will include [ SUCCESS ] Generated IR version 11 model.

convert_command = (
    f"omz_converter --name {model_name} --precisions {precision} "
    f"--download_dir {base_model_dir} --output_dir {base_model_dir}"
)
display(Markdown(f"Convert command: `{convert_command}`"))
display(Markdown(f"Converting {model_name}..."))

! $convert_command

Convert command: omz_converter --name deblurgan-v2 --precisions FP16 --download_dir /opt/home/k8sworker/open_model_zoo_models --output_dir /opt/home/k8sworker/open_model_zoo_models

Converting deblurgan-v2…

========== Converting deblurgan-v2 to ONNX
Conversion to ONNX command: /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-275/.workspace/scm/ov-notebook/.venv/bin/python -- /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-275/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/internal_scripts/pytorch_to_onnx.py --model-path=/opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-275/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino/model_zoo/models/public/deblurgan-v2 --model-path=/opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2 --model-name=DeblurV2 --import-module=model --input-shape=1,3,736,1312 --output-file=/opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/deblurgan-v2.onnx '--model-param=weights=r"/opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/ckpt/fpn_mobilenet.h5"' '--model-param=model_name="fpn_mobilenet"' --input-names=blur_image --output-names=deblur_image

ONNX check passed successfully.

========== Converting deblurgan-v2 to IR (FP16)
Conversion command: /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-275/.workspace/scm/ov-notebook/.venv/bin/python -- /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-275/.workspace/scm/ov-notebook/.venv/bin/mo --framework=onnx --data_type=FP16 --output_dir=/opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/FP16 --model_name=deblurgan-v2 --input=blur_image --model_name=deblurgan-v2 --reverse_input_channels '--mean_values=blur_image[127.5]' '--scale_values=blur_image[127.5]' --output=deblur_image --input_model=/opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/deblurgan-v2.onnx '--layout=blur_image(NCHW)' '--input_shape=[1, 3, 736, 1312]'

Model Optimizer arguments:
Common parameters:
    - Path to the Input Model:  /opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/deblurgan-v2.onnx
    - Path for generated IR:    /opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/FP16
    - IR output name:   deblurgan-v2
    - Log level:    ERROR
    - Batch:    Not specified, inherited from the model
    - Input layers:     blur_image
    - Output layers:    deblur_image
    - Input shapes:     [1, 3, 736, 1312]
    - Source layout:    Not specified
    - Target layout:    Not specified
    - Layout:   blur_image(NCHW)
    - Mean values:  blur_image[127.5]
    - Scale values:     blur_image[127.5]
    - Scale factor:     Not specified
    - Precision of IR:  FP16
    - Enable fusing:    True
    - User transformations:     Not specified
    - Reverse input channels:   True
    - Enable IR generation for fixed input shape:   False
    - Use the transformations config file:  None
Advanced parameters:
    - Force the usage of legacy Frontend of Model Optimizer for model conversion into IR:   False
    - Force the usage of new Frontend of Model Optimizer for model conversion into IR:  False
OpenVINO runtime found in:  /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-275/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino
OpenVINO runtime version:   2022.2.0-7713-af16ea1d79a-releases/2022/2
Model Optimizer version:    2022.2.0-7713-af16ea1d79a-releases/2022/2
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/FP16/deblurgan-v2.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/open_model_zoo_models/public/deblurgan-v2/FP16/deblurgan-v2.bin
[ SUCCESS ] Total execution time: 0.83 seconds.
[ SUCCESS ] Memory consumed: 128 MB.
[ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11.
Find more information about API v2.0 and IR v11 at https://docs.openvino.ai

Copy the Model¶

Copy the DeblurGAN-v2 Model to the specified model_dir directory in the settings.

model_info = %sx omz_info_dumper --name $model_name

model_info_json = json.loads(model_info.get_nlstr())[0]

model_downloaded_dir = (
    base_model_dir / Path(model_info_json["subdirectory"]) / Path(precision)
)

for model_file in model_downloaded_dir.iterdir():
    try:
        shutil.copyfile(model_file, model_dir / model_file.name)
    except FileExistsError:
        pass

Load the Model¶

Load and compile the DeblurGAN-v2 model in the OpenVINO Runtime with ie.read_model and compile it for the specified device with ie.compile_model. Get input and output keys and the expected input shape for the model.

ie = Core()
model = ie.read_model(model=model_xml_path)
compiled_model = ie.compile_model(model=model, device_name=DEVICE)

model_input_layer = compiled_model.input(0)
model_output_layer = compiled_model.output(0)

model_input_layer

<ConstOutput: names[blur_image] shape{1,3,736,1312} type: f32>

model_output_layer

<ConstOutput: names[deblur_image] shape{1,3,736,1312} type: f32>

Deblur Image¶

Load, resize and reshape input image¶

The input image is read by using the default load_image function from notebooks.utils. Then, resized to meet the network expected input sizes, and reshaped to (N, C, H, W), where N is a number of images in the batch, C is a number of channels, H is the height, and W is the width.

# Image filename (local path or URL)
filename = "https://user-images.githubusercontent.com/41332813/166901955-5d813e4c-a895-4da2-a36c-a96b3f627ebb.png"

# Load the input image.
image = load_image(filename)

# Convert the image if it is stored in RGBA format.
image = np.asarray(Image.fromarray(image).convert('RGB'))

# N,C,H,W = batch size, number of channels, height, width.
N, C, H, W = model_input_layer.shape

# Resize the image to meet network expected input sizes.
resized_image = cv2.resize(image, (W, H))

# Reshape to network input shape.
input_image = np.expand_dims(resized_image.transpose(2, 0, 1), 0)

plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))

<matplotlib.image.AxesImage at 0x7f9fbc603bb0>

../_images/217-vision-deblur-with-output_24_1.png

Do Inference on the Input Image¶

Do the inference, convert the result to an image shape and resize it to the original image size.

# Inference.
result = compiled_model([input_image])[model_output_layer]

# Convert the result to an image shape.
result_image = result[0].transpose((1, 2, 0))

# Resize to the original image size.
resized_result_image = cv2.resize(result_image, image.shape[:2][::-1])

plt.imshow(cv2.cvtColor(result_image, cv2.COLOR_BGR2RGB))

<matplotlib.image.AxesImage at 0x7f9fbc56d340>

../_images/217-vision-deblur-with-output_27_1.png

Load the groundtruth Image¶

Load the groundtruth image for comparison.

# Load grountruth image.
groundtruth_image = load_image("https://user-images.githubusercontent.com/41332813/166900611-83be8ae5-23ed-4426-b535-fad960795687.png")

Display results¶

Images are in BGR format, so they will be converted to the RGB format in order to be properly displayed by matplotlib library.

# Create subplot(r,c) by providing the no. of rows (r),
# number of columns (c) and figure size.
f, ax = plt.subplots(1, 3, figsize=(20, 15))

# Use the created array and display the images horizontally.
ax[0].set_title("Blurred")
ax[0].imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))

ax[1].set_title("DeblurGAN-v2")
ax[1].imshow(cv2.cvtColor(resized_result_image, cv2.COLOR_BGR2RGB))

ax[2].set_title("Groundtruth")
ax[2].imshow(cv2.cvtColor(groundtruth_image, cv2.COLOR_BGR2RGB))

<matplotlib.image.AxesImage at 0x7f9fbc499550>

../_images/217-vision-deblur-with-output_31_1.png

Save the deblurred image¶

Save the output image of the DeblurGAN-v2 model in the output_dir directory.

savename = "deblurred.png"
plt.imsave(
    output_dir / Path(savename),
    cv2.cvtColor(resized_result_image, cv2.COLOR_BGR2RGB),
)

Prev Next