OpenVINO™ Explainable AI Toolkit (2/3): Deep Dive#

This Jupyter notebook can be launched on-line, opening an interactive environment in a browser window. You can also make a local installation. Choose one of the following options:

Google ColabGithub

Warning

Important note: This notebook requires python >= 3.10. Please make sure that your environment fulfill to this requirement before running it

This is the second notebook in series of exploring OpenVINO™ Explainable AI (XAI):

  1. OpenVINO™ Explainable AI Toolkit (1/3): Basic

  2. OpenVINO™ Explainable AI Toolkit (2/3): Deep Dive

  3. OpenVINO™ Explainable AI Toolkit (3/3): Saliency map interpretation

OpenVINO™ Explainable AI (XAI) provides a suite of XAI algorithms for visual explanation of OpenVINO™ Intermediate Representation (IR) models.

Using OpenVINO XAI, you can generate saliency maps that highlight regions of interest in input images from the model’s perspective. This helps users understand why complex AI models produce specific responses.

This notebook shows an example of how to use OpenVINO XAI, exploring its methods and functionality.

It displays a heatmap indicating areas of interest where a neural network (for classification or detection) focuses before making a decision.

Let’s imagine the case that our OpenVINO IR model is up and running on a inference pipeline. While watching the outputs, we may want to analyze the model’s behavior for debugging or understanding purposes.

By using the OpenVINO XAI Explainer, we can visualize why the model gives such responses, meaning on which areas it focused before predicting a particular label.

Table of contents:

Installation Instructions#

This is a self-contained example that relies solely on its own code.

We recommend running the notebook in a virtual environment. You only need a Jupyter server to start. For details, please refer to Installation Guide.

Prerequisites#

Install requirements#

%%capture

import platform

# Install openvino package
%pip install -q "openvino>=2024.2.0" opencv-python tqdm

%pip install -q --no-deps "openvino-xai>=1.0.0"

if platform.system() != "Windows":
    %pip install -q "matplotlib>=3.4"
else:
    %pip install -q "matplotlib>=3.4,<3.7"

Imports#

from pathlib import Path

import cv2
import matplotlib.pyplot as plt
import numpy as np

import openvino.runtime as ov
from openvino.runtime.utils.data_helpers.wrappers import OVDict
import openvino_xai as xai
from openvino_xai.explainer import ExplainMode
from openvino_xai.explainer.explanation import Explanation

# Fetch `notebook_utils` module
import requests

r = requests.get(
    url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py",
)

open("notebook_utils.py", "w").write(r.text)

from notebook_utils import download_file

Download IR model#

In this notebook for demonstration purposes we’ll use an already converted to IR model from OpenVINO storage.

base_artifacts_dir = Path("./artifacts").expanduser()

model_name = "v3-small_224_1.0_float"
model_xml_name = f"{model_name}.xml"
model_bin_name = f"{model_name}.bin"

model_xml_path = base_artifacts_dir / model_xml_name

base_url = "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/mobelinet-v3-tf/FP32/"

if not model_xml_path.exists():
    download_file(base_url + model_xml_name, model_xml_name, base_artifacts_dir)
    download_file(base_url + model_bin_name, model_bin_name, base_artifacts_dir)
else:
    print(f"{model_name} already downloaded to {base_artifacts_dir}")

v3-small_224_1.0_float already downloaded to artifacts

Load the Image#

# Download the image from the openvino_notebooks storage
image_filename = download_file(
    "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/coco.jpg",
    directory="data",
)

# The MobileNet model expects images in RGB format.
image = cv2.cvtColor(cv2.imread(filename=str(image_filename)), code=cv2.COLOR_BGR2RGB)
plt.imshow(image);

'data/coco.jpg' already exists.
../_images/explainable-ai-2-deep-dive-with-output_10_1.png

Preprocess image for MobileNet#

# Resize to MobileNetV3 input image shape.
preprocessed_image = cv2.resize(src=image, dsize=(224, 224))
# Add batch dimension
preprocessed_image = np.expand_dims(preprocessed_image, 0)

Basic usage: Auto mode explainer#

The easiest way to run the explainer is to do it in Auto mode. Under the hood of Auto mode, it will first try to run the White-Box mode. If this fails, it will then run the Black-Box mode. See more details about White-Box and Black-Box modes below.

Generating saliency maps involves model inference. The explainer will perform model inference, but to do so, it requires preprocess_fn and postprocess_fn.
At this stage, we can avoid passing preprocess_fn by preprocessing the data beforehand (e.g., resizing and adding a batch dimension as shown above). We also don’t pass postprocess_fn here for simplicity, since the White-Box mode doesn’t fail on the example model.

To learn more about pre- and post-process functions, refer to the Pre- and post-process functions section.

Create Explainer#

# Create ov.Model
model = ov.Core().read_model(model_xml_path)

# Create explainer object
explainer = xai.Explainer(
    model=model,
    task=xai.Task.CLASSIFICATION,
)

INFO:openvino_xai:Assigning preprocess_fn to identity function assumes that input images were already preprocessed by user before passing it to the model. Please define preprocessing function OR preprocess images beforehand.
INFO:openvino_xai:Target insertion layer is not provided - trying to find it in auto mode.
INFO:openvino_xai:Using ReciproCAM method (for CNNs).
INFO:openvino_xai:Explaining the model in white-box mode.

Do explanation#

The predicted label for this image is flat-coated_retriever with label index 206. So here and further we will check saliency maps for this index.

# You can choose classes to generate saliency maps for.
# In this notebook we will check maps for predicted class 206 - flat-coated retriever
retriever_class_index = 206

explanation = explainer(
    preprocessed_image,
    targets=retriever_class_index,
    overlay=True,  # False by default
)

Visualize saliency maps#

explanation: Explanation
# Dict[int: np.ndarray] where key - class id, value - processed saliency map e.g. 354x500x3
explanation.saliency_map

# Check saved saliency maps
print(f"Saliency maps were generated for the following classes: {explanation.targets}")
print(f"Saliency map size: {explanation.shape}")

# Show saliency maps for retriever class
retriever_sal_map = explanation.saliency_map[retriever_class_index]
plt.imshow(retriever_sal_map);

Saliency maps were generated for the following classes: [206]
Saliency map size: (224, 224, 3)
../_images/explainable-ai-2-deep-dive-with-output_21_1.png

Save saliency maps#

# Save saliency map
output = base_artifacts_dir / "explain_auto"
explanation.save(output)

# See saved saliency maps
image_sal_map = cv2.imread(f"{output}/target_{retriever_class_index}.jpg")
image_sal_map = cv2.cvtColor(image_sal_map, cv2.COLOR_BGR2RGB)
plt.imshow(image_sal_map);
../_images/explainable-ai-2-deep-dive-with-output_24_0.png

Return saliency maps for all classes#

explanation = explainer(preprocessed_image, targets=-1)

# Check saved saliency maps
print(f"Saliency maps were generated for the following classes: {explanation.targets}")
print(f"Saliency map size: {explanation.shape}")

Saliency maps were generated for the following classes: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418, 419, 420, 421, 422, 423, 424, 425, 426, 427, 428, 429, 430, 431, 432, 433, 434, 435, 436, 437, 438, 439, 440, 441, 442, 443, 444, 445, 446, 447, 448, 449, 450, 451, 452, 453, 454, 455, 456, 457, 458, 459, 460, 461, 462, 463, 464, 465, 466, 467, 468, 469, 470, 471, 472, 473, 474, 475, 476, 477, 478, 479, 480, 481, 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492, 493, 494, 495, 496, 497, 498, 499, 500, 501, 502, 503, 504, 505, 506, 507, 508, 509, 510, 511, 512, 513, 514, 515, 516, 517, 518, 519, 520, 521, 522, 523, 524, 525, 526, 527, 528, 529, 530, 531, 532, 533, 534, 535, 536, 537, 538, 539, 540, 541, 542, 543, 544, 545, 546, 547, 548, 549, 550, 551, 552, 553, 554, 555, 556, 557, 558, 559, 560, 561, 562, 563, 564, 565, 566, 567, 568, 569, 570, 571, 572, 573, 574, 575, 576, 577, 578, 579, 580, 581, 582, 583, 584, 585, 586, 587, 588, 589, 590, 591, 592, 593, 594, 595, 596, 597, 598, 599, 600, 601, 602, 603, 604, 605, 606, 607, 608, 609, 610, 611, 612, 613, 614, 615, 616, 617, 618, 619, 620, 621, 622, 623, 624, 625, 626, 627, 628, 629, 630, 631, 632, 633, 634, 635, 636, 637, 638, 639, 640, 641, 642, 643, 644, 645, 646, 647, 648, 649, 650, 651, 652, 653, 654, 655, 656, 657, 658, 659, 660, 661, 662, 663, 664, 665, 666, 667, 668, 669, 670, 671, 672, 673, 674, 675, 676, 677, 678, 679, 680, 681, 682, 683, 684, 685, 686, 687, 688, 689, 690, 691, 692, 693, 694, 695, 696, 697, 698, 699, 700, 701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716, 717, 718, 719, 720, 721, 722, 723, 724, 725, 726, 727, 728, 729, 730, 731, 732, 733, 734, 735, 736, 737, 738, 739, 740, 741, 742, 743, 744, 745, 746, 747, 748, 749, 750, 751, 752, 753, 754, 755, 756, 757, 758, 759, 760, 761, 762, 763, 764, 765, 766, 767, 768, 769, 770, 771, 772, 773, 774, 775, 776, 777, 778, 779, 780, 781, 782, 783, 784, 785, 786, 787, 788, 789, 790, 791, 792, 793, 794, 795, 796, 797, 798, 799, 800, 801, 802, 803, 804, 805, 806, 807, 808, 809, 810, 811, 812, 813, 814, 815, 816, 817, 818, 819, 820, 821, 822, 823, 824, 825, 826, 827, 828, 829, 830, 831, 832, 833, 834, 835, 836, 837, 838, 839, 840, 841, 842, 843, 844, 845, 846, 847, 848, 849, 850, 851, 852, 853, 854, 855, 856, 857, 858, 859, 860, 861, 862, 863, 864, 865, 866, 867, 868, 869, 870, 871, 872, 873, 874, 875, 876, 877, 878, 879, 880, 881, 882, 883, 884, 885, 886, 887, 888, 889, 890, 891, 892, 893, 894, 895, 896, 897, 898, 899, 900, 901, 902, 903, 904, 905, 906, 907, 908, 909, 910, 911, 912, 913, 914, 915, 916, 917, 918, 919, 920, 921, 922, 923, 924, 925, 926, 927, 928, 929, 930, 931, 932, 933, 934, 935, 936, 937, 938, 939, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 952, 953, 954, 955, 956, 957, 958, 959, 960, 961, 962, 963, 964, 965, 966, 967, 968, 969, 970, 971, 972, 973, 974, 975, 976, 977, 978, 979, 980, 981, 982, 983, 984, 985, 986, 987, 988, 989, 990, 991, 992, 993, 994, 995, 996, 997, 998, 999, 1000]
Saliency map size: (224, 224, 3)

Pre- and post-process functions#

The explainer can apply pre-processing internally during model inference, allowing you to provide a raw image as input to the explainer.

To enable this, define preprocess_fn and provide it to the explainer constructor. By default, preprocess_fn is an identity function that passes the input without any changes, assuming it is preprocessed beforehand.

In Auto mode, the explainer tries to run the White-Box mode first. If it fails, the corresponding exception will be raised, and the Black-Box mode will be enabled as a fallback.

The Black-Box mode requires access to the output logits (activated or not). Therefore, in such cases, postprocess_fn is required, which accepts the raw IR model output and returns logits (see below for a reference).

def preprocess_fn(x: np.ndarray) -> np.ndarray:
    # Implementing own pre-process function based on model's implementation
    x = cv2.resize(src=x, dsize=(224, 224))

    # Add batch dimension
    x = np.expand_dims(x, 0)
    return x


def postprocess_fn(x: OVDict):
    # Implementing own post-process function based on model's implementation
    # Return "logits" model output
    return x[0]

# Create explainer object
explainer = xai.Explainer(
    model=model,
    task=xai.Task.CLASSIFICATION,
    preprocess_fn=preprocess_fn,
    postprocess_fn=postprocess_fn,
)

explanation = explainer(image, targets=retriever_class_index)

INFO:openvino_xai:Target insertion layer is not provided - trying to find it in auto mode.
INFO:openvino_xai:Using ReciproCAM method (for CNNs).
INFO:openvino_xai:Explaining the model in white-box mode.

Visualization Parameters#

  • resize (True by default): If True, resize saliency map to the input image size.

  • colormap (True by default): If True, apply colormap to the grayscale saliency map.

  • overlay (False by default): If True, generate overlay of the saliency map over the input image.

  • original_input_image (None by default): Provide the original, unprocessed image to apply the overlay. This ensures the overlay is not applied to a preprocessed image, which may be resized or normalized and lose readability.

  • overlay_weight (0.5 by default): Weight of the saliency map when overlaying the input data with the saliency map.

# Create explainer object
explainer = xai.Explainer(model=model, task=xai.Task.CLASSIFICATION)

# Return overlayed image
explanation = explainer(
    preprocessed_image,
    targets=[retriever_class_index],  # target can be a single label index, label name or a list of indices/names
    overlay=True,  # False by default
    original_input_image=image,  # to apply overlay on the original image instead of preprocessed one that was used for the explainer
)

retriever_sal_map = explanation.saliency_map[retriever_class_index]
plt.imshow(retriever_sal_map)

# Save saliency map
output = base_artifacts_dir / "overlay"
explanation.save(output)

INFO:openvino_xai:Assigning preprocess_fn to identity function assumes that input images were already preprocessed by user before passing it to the model. Please define preprocessing function OR preprocess images beforehand.
INFO:openvino_xai:Target insertion layer is not provided - trying to find it in auto mode.
INFO:openvino_xai:Using ReciproCAM method (for CNNs).
INFO:openvino_xai:Explaining the model in white-box mode.
../_images/explainable-ai-2-deep-dive-with-output_32_1.png 
# Return low-resolution saliency map
explanation = explainer(
    preprocessed_image,
    targets=[retriever_class_index],  # target can be a single label index, label name or a list of indices/names
    overlay=False,  # False by default
)

retriever_sal_map = explanation.saliency_map[retriever_class_index]
plt.imshow(retriever_sal_map)

# Save saliency map
output = base_artifacts_dir / "colormap"
explanation.save(output)
../_images/explainable-ai-2-deep-dive-with-output_33_0.png 
# Return low-resolution gray-scale saliency map
explanation = explainer(
    preprocessed_image,
    targets=[retriever_class_index],  # target can be a single label index, label name or a list of indices/names
    resize=False,  # True by default
    colormap=False,  # True by default
)

retriever_sal_map = explanation.saliency_map[retriever_class_index]
plt.imshow(retriever_sal_map, cmap="gray")

# Save saliency map
output = base_artifacts_dir / "grayscale"
explanation.save(output)
../_images/explainable-ai-2-deep-dive-with-output_34_0.png

White-Box explainer#

ReciproCAM explain method#

The White-Box explainer treats the model as a white box and needs to make inner modifications. It adds extra XAI nodes after the backbone to estimate which activations are important for model prediction.

If a method is not specified, the XAI branch will be generated using the ReciproCAM method.

By default, the insertion of the XAI branch will be done automatically by searching for the correct node.

It works quickly and precisely, requiring only one model inference.

# Create explainer object
explainer = xai.Explainer(
    model=model,
    task=xai.Task.CLASSIFICATION,
    preprocess_fn=preprocess_fn,
    # defaults to ExplainMode.AUTO
    explain_mode=ExplainMode.WHITEBOX,
    # ReciproCAM is the default XAI method for CNNs
    explain_method=xai.Method.RECIPROCAM,
)

INFO:openvino_xai:Target insertion layer is not provided - trying to find it in auto mode.
INFO:openvino_xai:Using ReciproCAM method (for CNNs).
INFO:openvino_xai:Explaining the model in white-box mode.

Insert XAI branch#

It’s possible to update the model with an XAI branch using the insert_xai functional API.

insert_xai will return an OpenVINO model with the XAI branch inserted and an additional saliency_map output.

This helps to avoid OpenVINO XAI dependency in the inference environment.

Note: XAI branch introduce an additional computational overhead (usually less than a single model forward pass).

# insert XAI branch
model_xai: ov.Model
model_xai = xai.insert_xai(
    model,
    task=xai.Task.CLASSIFICATION,
    explain_method=xai.Method.RECIPROCAM,
    target_layer="MobilenetV3/Conv_1/Conv2D",  # MobileNet V3
    embed_scaling=True,
)

INFO:openvino_xai:Target insertion layer MobilenetV3/Conv_1/Conv2D is provided.
INFO:openvino_xai:Using ReciproCAM method (for CNNs).
INFO:openvino_xai:Insertion of the XAI branch into the model was successful.

Import ImageNet label names and add them to saliency maps#

If label_names are not provided to the explainer call, the saved saliency map will have the predicted class index, not the name. For example, image_name_target_206.jpg instead of image_name_target_retriever.jpg.

To conveniently view label names in saliency maps, we provide ImageNet label names information to the explanation call.

imagenet_filename = download_file(
    "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/datasets/imagenet/imagenet_2012.txt",
    directory="data",
)

imagenet_classes = imagenet_filename.read_text().splitlines()

'data/imagenet_2012.txt' already exists.

imagenet_labels = []
for label in imagenet_classes:
    class_label = " ".join(label.split(" ")[1:])
    first_class_label = class_label.split(",")[0].replace(" ", "_")
    imagenet_labels.append(first_class_label)

print(" ".join(imagenet_labels[:10]))

tench goldfish great_white_shark tiger_shark hammerhead electric_ray stingray cock hen ostrich

# The model description states that for this model, class 0 is a background.
# Therefore, a background must be added at the beginning of imagenet_classes.
imagenet_labels = ["background"] + imagenet_labels

# Create explainer object
explainer = xai.Explainer(
    model=model,
    task=xai.Task.CLASSIFICATION,
    preprocess_fn=preprocess_fn,
    explain_mode=ExplainMode.WHITEBOX,
)

# Adding ImageNet label names.
explanation = explainer(
    image,
    # Return saliency maps for 2 named labels
    targets=["flat-coated_retriever", "microwave"],  # Also label indices [206, 652] are possible as target
    label_names=imagenet_labels,
)

INFO:openvino_xai:Target insertion layer is not provided - trying to find it in auto mode.
INFO:openvino_xai:Using ReciproCAM method (for CNNs).
INFO:openvino_xai:Explaining the model in white-box mode.

# Save saliency map
output = base_artifacts_dir / "label_names"
explanation.save(output)

Below in base_artifacts_dir / "label_names" you can see saved saliency maps with label name on it:

# See saliency mas saved in `output` with predicted label in image name
for file_name in output.glob("*"):
    print(file_name)

artifacts/label_names/target_microwave.jpg
artifacts/label_names/target_flat-coated_retriever.jpg

Activation map explain method#

The Activation Map method shows a general attention map without respect to specific classes. It can be useful for understanding which areas the model identifies as important.

If the explanation method is set to Method.ACTIVATIONMAP, instead of saliency maps for each class, the activation map is returned as explanation.saliency_map["per_image_map"].

# Create explainer object
explainer = xai.Explainer(
    model=model,
    task=xai.Task.CLASSIFICATION,
    preprocess_fn=preprocess_fn,
    explain_mode=ExplainMode.WHITEBOX,
    explain_method=xai.Method.ACTIVATIONMAP,
)

explanation = explainer(image, targets=-1, overlay=True)
activation_map = explanation.saliency_map["per_image_map"]

plt.imshow(activation_map)
plt.show()

INFO:openvino_xai:Target insertion layer is not provided - trying to find it in auto mode.
INFO:openvino_xai:Using ActivationMap method (for CNNs).
INFO:openvino_xai:Explaining the model in white-box mode.
../_images/explainable-ai-2-deep-dive-with-output_57_1.png