OpenVINO™ Explainable AI Toolkit (2/3): Deep Dive#
This Jupyter notebook can be launched on-line, opening an interactive environment in a browser window. You can also make a local installation. Choose one of the following options:
Warning
Important note: This notebook requires python >= 3.10. Please make sure that your environment fulfill to this requirement before running it
This is the second notebook in series of exploring OpenVINO™ Explainable AI (XAI):
OpenVINO™ Explainable AI (XAI) provides a suite of XAI algorithms for visual explanation of OpenVINO™ Intermediate Representation (IR) models.
Using OpenVINO XAI, you can generate saliency maps that highlight regions of interest in input images from the model’s perspective. This helps users understand why complex AI models produce specific responses.
This notebook shows an example of how to use OpenVINO XAI, exploring its methods and functionality.
It displays a heatmap indicating areas of interest where a neural network (for classification or detection) focuses before making a decision.
Let’s imagine the case that our OpenVINO IR model is up and running on a inference pipeline. While watching the outputs, we may want to analyze the model’s behavior for debugging or understanding purposes.
By using the OpenVINO XAI Explainer
, we can visualize why the model
gives such responses, meaning on which areas it focused before
predicting a particular label.
Table of contents:
Installation Instructions#
This is a self-contained example that relies solely on its own code.
We recommend running the notebook in a virtual environment. You only need a Jupyter server to start. For details, please refer to Installation Guide.
Prerequisites#
Install requirements#
%%capture
import platform
# Install openvino package
%pip install -q "openvino>=2024.2.0" opencv-python tqdm
%pip install -q --no-deps "openvino-xai>=1.0.0"
if platform.system() != "Windows":
%pip install -q "matplotlib>=3.4"
else:
%pip install -q "matplotlib>=3.4,<3.7"
Imports#
from pathlib import Path
import cv2
import matplotlib.pyplot as plt
import numpy as np
import openvino.runtime as ov
from openvino.runtime.utils.data_helpers.wrappers import OVDict
import openvino_xai as xai
from openvino_xai.explainer import ExplainMode
from openvino_xai.explainer.explanation import Explanation
# Fetch `notebook_utils` module
import requests
r = requests.get(
url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py",
)
open("notebook_utils.py", "w").write(r.text)
from notebook_utils import download_file
Download IR model#
In this notebook for demonstration purposes we’ll use an already converted to IR model from OpenVINO storage.
base_artifacts_dir = Path("./artifacts").expanduser()
model_name = "v3-small_224_1.0_float"
model_xml_name = f"{model_name}.xml"
model_bin_name = f"{model_name}.bin"
model_xml_path = base_artifacts_dir / model_xml_name
base_url = "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/mobelinet-v3-tf/FP32/"
if not model_xml_path.exists():
download_file(base_url + model_xml_name, model_xml_name, base_artifacts_dir)
download_file(base_url + model_bin_name, model_bin_name, base_artifacts_dir)
else:
print(f"{model_name} already downloaded to {base_artifacts_dir}")
v3-small_224_1.0_float already downloaded to artifacts
Load the Image#
# Download the image from the openvino_notebooks storage
image_filename = download_file(
"https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/coco.jpg",
directory="data",
)
# The MobileNet model expects images in RGB format.
image = cv2.cvtColor(cv2.imread(filename=str(image_filename)), code=cv2.COLOR_BGR2RGB)
plt.imshow(image);
'data/coco.jpg' already exists.
Preprocess image for MobileNet#
# Resize to MobileNetV3 input image shape.
preprocessed_image = cv2.resize(src=image, dsize=(224, 224))
# Add batch dimension
preprocessed_image = np.expand_dims(preprocessed_image, 0)
Basic usage: Auto mode explainer#
The easiest way to run the explainer is to do it in Auto mode. Under the
hood of Auto mode, it will first try to run the White-Box
mode. If
this fails, it will then run the Black-Box
mode. See more details
about White-Box and
Black-Box modes below.
preprocess_fn
and postprocess_fn
.preprocess_fn
by preprocessing
the data beforehand (e.g., resizing and adding a batch dimension as
shown above). We also don’t pass postprocess_fn
here for
simplicity, since the White-Box mode doesn’t fail on the example
model.To learn more about pre- and post-process functions, refer to the Pre- and post-process functions section.
Create Explainer#
# Create ov.Model
model = ov.Core().read_model(model_xml_path)
# Create explainer object
explainer = xai.Explainer(
model=model,
task=xai.Task.CLASSIFICATION,
)
INFO:openvino_xai:Assigning preprocess_fn to identity function assumes that input images were already preprocessed by user before passing it to the model. Please define preprocessing function OR preprocess images beforehand.
INFO:openvino_xai:Target insertion layer is not provided - trying to find it in auto mode.
INFO:openvino_xai:Using ReciproCAM method (for CNNs).
INFO:openvino_xai:Explaining the model in white-box mode.
Do explanation#
The predicted label for this image is flat-coated_retriever
with
label index 206
. So here and further we will check saliency maps for
this index.
# You can choose classes to generate saliency maps for.
# In this notebook we will check maps for predicted class 206 - flat-coated retriever
retriever_class_index = 206
explanation = explainer(
preprocessed_image,
targets=retriever_class_index,
overlay=True, # False by default
)
Visualize saliency maps#
explanation: Explanation
# Dict[int: np.ndarray] where key - class id, value - processed saliency map e.g. 354x500x3
explanation.saliency_map
# Check saved saliency maps
print(f"Saliency maps were generated for the following classes: {explanation.targets}")
print(f"Saliency map size: {explanation.shape}")
# Show saliency maps for retriever class
retriever_sal_map = explanation.saliency_map[retriever_class_index]
plt.imshow(retriever_sal_map);
Saliency maps were generated for the following classes: [206]
Saliency map size: (224, 224, 3)
Save saliency maps#
# Save saliency map
output = base_artifacts_dir / "explain_auto"
explanation.save(output)
# See saved saliency maps
image_sal_map = cv2.imread(f"{output}/target_{retriever_class_index}.jpg")
image_sal_map = cv2.cvtColor(image_sal_map, cv2.COLOR_BGR2RGB)
plt.imshow(image_sal_map);
Return saliency maps for all classes#
explanation = explainer(preprocessed_image, targets=-1)
# Check saved saliency maps
print(f"Saliency maps were generated for the following classes: {explanation.targets}")
print(f"Saliency map size: {explanation.shape}")
Saliency maps were generated for the following classes: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418, 419, 420, 421, 422, 423, 424, 425, 426, 427, 428, 429, 430, 431, 432, 433, 434, 435, 436, 437, 438, 439, 440, 441, 442, 443, 444, 445, 446, 447, 448, 449, 450, 451, 452, 453, 454, 455, 456, 457, 458, 459, 460, 461, 462, 463, 464, 465, 466, 467, 468, 469, 470, 471, 472, 473, 474, 475, 476, 477, 478, 479, 480, 481, 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492, 493, 494, 495, 496, 497, 498, 499, 500, 501, 502, 503, 504, 505, 506, 507, 508, 509, 510, 511, 512, 513, 514, 515, 516, 517, 518, 519, 520, 521, 522, 523, 524, 525, 526, 527, 528, 529, 530, 531, 532, 533, 534, 535, 536, 537, 538, 539, 540, 541, 542, 543, 544, 545, 546, 547, 548, 549, 550, 551, 552, 553, 554, 555, 556, 557, 558, 559, 560, 561, 562, 563, 564, 565, 566, 567, 568, 569, 570, 571, 572, 573, 574, 575, 576, 577, 578, 579, 580, 581, 582, 583, 584, 585, 586, 587, 588, 589, 590, 591, 592, 593, 594, 595, 596, 597, 598, 599, 600, 601, 602, 603, 604, 605, 606, 607, 608, 609, 610, 611, 612, 613, 614, 615, 616, 617, 618, 619, 620, 621, 622, 623, 624, 625, 626, 627, 628, 629, 630, 631, 632, 633, 634, 635, 636, 637, 638, 639, 640, 641, 642, 643, 644, 645, 646, 647, 648, 649, 650, 651, 652, 653, 654, 655, 656, 657, 658, 659, 660, 661, 662, 663, 664, 665, 666, 667, 668, 669, 670, 671, 672, 673, 674, 675, 676, 677, 678, 679, 680, 681, 682, 683, 684, 685, 686, 687, 688, 689, 690, 691, 692, 693, 694, 695, 696, 697, 698, 699, 700, 701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716, 717, 718, 719, 720, 721, 722, 723, 724, 725, 726, 727, 728, 729, 730, 731, 732, 733, 734, 735, 736, 737, 738, 739, 740, 741, 742, 743, 744, 745, 746, 747, 748, 749, 750, 751, 752, 753, 754, 755, 756, 757, 758, 759, 760, 761, 762, 763, 764, 765, 766, 767, 768, 769, 770, 771, 772, 773, 774, 775, 776, 777, 778, 779, 780, 781, 782, 783, 784, 785, 786, 787, 788, 789, 790, 791, 792, 793, 794, 795, 796, 797, 798, 799, 800, 801, 802, 803, 804, 805, 806, 807, 808, 809, 810, 811, 812, 813, 814, 815, 816, 817, 818, 819, 820, 821, 822, 823, 824, 825, 826, 827, 828, 829, 830, 831, 832, 833, 834, 835, 836, 837, 838, 839, 840, 841, 842, 843, 844, 845, 846, 847, 848, 849, 850, 851, 852, 853, 854, 855, 856, 857, 858, 859, 860, 861, 862, 863, 864, 865, 866, 867, 868, 869, 870, 871, 872, 873, 874, 875, 876, 877, 878, 879, 880, 881, 882, 883, 884, 885, 886, 887, 888, 889, 890, 891, 892, 893, 894, 895, 896, 897, 898, 899, 900, 901, 902, 903, 904, 905, 906, 907, 908, 909, 910, 911, 912, 913, 914, 915, 916, 917, 918, 919, 920, 921, 922, 923, 924, 925, 926, 927, 928, 929, 930, 931, 932, 933, 934, 935, 936, 937, 938, 939, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 952, 953, 954, 955, 956, 957, 958, 959, 960, 961, 962, 963, 964, 965, 966, 967, 968, 969, 970, 971, 972, 973, 974, 975, 976, 977, 978, 979, 980, 981, 982, 983, 984, 985, 986, 987, 988, 989, 990, 991, 992, 993, 994, 995, 996, 997, 998, 999, 1000]
Saliency map size: (224, 224, 3)
Pre- and post-process functions#
The explainer can apply pre-processing internally during model inference, allowing you to provide a raw image as input to the explainer.
To enable this, define preprocess_fn
and provide it to the explainer
constructor. By default, preprocess_fn
is an identity function that
passes the input without any changes, assuming it is preprocessed
beforehand.
In Auto mode, the explainer tries to run the White-Box mode first. If it fails, the corresponding exception will be raised, and the Black-Box mode will be enabled as a fallback.
The Black-Box mode requires access to the output logits
(activated
or not). Therefore, in such cases, postprocess_fn
is required, which
accepts the raw IR model output and returns logits (see below for a
reference).
def preprocess_fn(x: np.ndarray) -> np.ndarray:
# Implementing own pre-process function based on model's implementation
x = cv2.resize(src=x, dsize=(224, 224))
# Add batch dimension
x = np.expand_dims(x, 0)
return x
def postprocess_fn(x: OVDict):
# Implementing own post-process function based on model's implementation
# Return "logits" model output
return x[0]
# Create explainer object
explainer = xai.Explainer(
model=model,
task=xai.Task.CLASSIFICATION,
preprocess_fn=preprocess_fn,
postprocess_fn=postprocess_fn,
)
explanation = explainer(image, targets=retriever_class_index)
INFO:openvino_xai:Target insertion layer is not provided - trying to find it in auto mode.
INFO:openvino_xai:Using ReciproCAM method (for CNNs).
INFO:openvino_xai:Explaining the model in white-box mode.
Visualization Parameters#
resize (True by default): If True, resize saliency map to the input image size.
colormap (True by default): If True, apply colormap to the grayscale saliency map.
overlay (False by default): If True, generate overlay of the saliency map over the input image.
original_input_image (None by default): Provide the original, unprocessed image to apply the overlay. This ensures the overlay is not applied to a preprocessed image, which may be resized or normalized and lose readability.
overlay_weight (0.5 by default): Weight of the saliency map when overlaying the input data with the saliency map.
# Create explainer object
explainer = xai.Explainer(model=model, task=xai.Task.CLASSIFICATION)
# Return overlayed image
explanation = explainer(
preprocessed_image,
targets=[retriever_class_index], # target can be a single label index, label name or a list of indices/names
overlay=True, # False by default
original_input_image=image, # to apply overlay on the original image instead of preprocessed one that was used for the explainer
)
retriever_sal_map = explanation.saliency_map[retriever_class_index]
plt.imshow(retriever_sal_map)
# Save saliency map
output = base_artifacts_dir / "overlay"
explanation.save(output)
INFO:openvino_xai:Assigning preprocess_fn to identity function assumes that input images were already preprocessed by user before passing it to the model. Please define preprocessing function OR preprocess images beforehand.
INFO:openvino_xai:Target insertion layer is not provided - trying to find it in auto mode.
INFO:openvino_xai:Using ReciproCAM method (for CNNs).
INFO:openvino_xai:Explaining the model in white-box mode.
# Return low-resolution saliency map
explanation = explainer(
preprocessed_image,
targets=[retriever_class_index], # target can be a single label index, label name or a list of indices/names
overlay=False, # False by default
)
retriever_sal_map = explanation.saliency_map[retriever_class_index]
plt.imshow(retriever_sal_map)
# Save saliency map
output = base_artifacts_dir / "colormap"
explanation.save(output)
# Return low-resolution gray-scale saliency map
explanation = explainer(
preprocessed_image,
targets=[retriever_class_index], # target can be a single label index, label name or a list of indices/names
resize=False, # True by default
colormap=False, # True by default
)
retriever_sal_map = explanation.saliency_map[retriever_class_index]
plt.imshow(retriever_sal_map, cmap="gray")
# Save saliency map
output = base_artifacts_dir / "grayscale"
explanation.save(output)
White-Box explainer#
ReciproCAM explain method#
The White-Box explainer treats the model as a white box and needs to make inner modifications. It adds extra XAI nodes after the backbone to estimate which activations are important for model prediction.
If a method is not specified, the XAI branch will be generated using the ReciproCAM method.
By default, the insertion of the XAI branch will be done automatically by searching for the correct node.
It works quickly and precisely, requiring only one model inference.
# Create explainer object
explainer = xai.Explainer(
model=model,
task=xai.Task.CLASSIFICATION,
preprocess_fn=preprocess_fn,
# defaults to ExplainMode.AUTO
explain_mode=ExplainMode.WHITEBOX,
# ReciproCAM is the default XAI method for CNNs
explain_method=xai.Method.RECIPROCAM,
)
INFO:openvino_xai:Target insertion layer is not provided - trying to find it in auto mode.
INFO:openvino_xai:Using ReciproCAM method (for CNNs).
INFO:openvino_xai:Explaining the model in white-box mode.
Insert XAI branch#
It’s possible to update the model with an XAI branch using the
insert_xai
functional API.
insert_xai
will return an OpenVINO model with the XAI branch
inserted and an additional saliency_map
output.
This helps to avoid OpenVINO XAI dependency in the inference environment.
Note: XAI branch introduce an additional computational overhead (usually less than a single model forward pass).
# insert XAI branch
model_xai: ov.Model
model_xai = xai.insert_xai(
model,
task=xai.Task.CLASSIFICATION,
explain_method=xai.Method.RECIPROCAM,
target_layer="MobilenetV3/Conv_1/Conv2D", # MobileNet V3
embed_scaling=True,
)
INFO:openvino_xai:Target insertion layer MobilenetV3/Conv_1/Conv2D is provided.
INFO:openvino_xai:Using ReciproCAM method (for CNNs).
INFO:openvino_xai:Insertion of the XAI branch into the model was successful.
Import ImageNet label names and add them to saliency maps#
If label_names
are not provided to the explainer call, the saved
saliency map will have the predicted class index, not the name. For
example, image_name_target_206.jpg
instead of
image_name_target_retriever.jpg
.
To conveniently view label names in saliency maps, we provide ImageNet label names information to the explanation call.
imagenet_filename = download_file(
"https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/datasets/imagenet/imagenet_2012.txt",
directory="data",
)
imagenet_classes = imagenet_filename.read_text().splitlines()
'data/imagenet_2012.txt' already exists.
imagenet_labels = []
for label in imagenet_classes:
class_label = " ".join(label.split(" ")[1:])
first_class_label = class_label.split(",")[0].replace(" ", "_")
imagenet_labels.append(first_class_label)
print(" ".join(imagenet_labels[:10]))
tench goldfish great_white_shark tiger_shark hammerhead electric_ray stingray cock hen ostrich
# The model description states that for this model, class 0 is a background.
# Therefore, a background must be added at the beginning of imagenet_classes.
imagenet_labels = ["background"] + imagenet_labels
# Create explainer object
explainer = xai.Explainer(
model=model,
task=xai.Task.CLASSIFICATION,
preprocess_fn=preprocess_fn,
explain_mode=ExplainMode.WHITEBOX,
)
# Adding ImageNet label names.
explanation = explainer(
image,
# Return saliency maps for 2 named labels
targets=["flat-coated_retriever", "microwave"], # Also label indices [206, 652] are possible as target
label_names=imagenet_labels,
)
INFO:openvino_xai:Target insertion layer is not provided - trying to find it in auto mode.
INFO:openvino_xai:Using ReciproCAM method (for CNNs).
INFO:openvino_xai:Explaining the model in white-box mode.
# Save saliency map
output = base_artifacts_dir / "label_names"
explanation.save(output)
Below in base_artifacts_dir / "label_names"
you can see saved
saliency maps with label name on it:
# See saliency mas saved in `output` with predicted label in image name
for file_name in output.glob("*"):
print(file_name)
artifacts/label_names/target_microwave.jpg
artifacts/label_names/target_flat-coated_retriever.jpg
Activation map explain method#
The Activation Map method shows a general attention map without respect to specific classes. It can be useful for understanding which areas the model identifies as important.
If the explanation method is set to Method.ACTIVATIONMAP
, instead of
saliency maps for each class, the activation map is returned as
explanation.saliency_map["per_image_map"]
.
# Create explainer object
explainer = xai.Explainer(
model=model,
task=xai.Task.CLASSIFICATION,
preprocess_fn=preprocess_fn,
explain_mode=ExplainMode.WHITEBOX,
explain_method=xai.Method.ACTIVATIONMAP,
)
explanation = explainer(image, targets=-1, overlay=True)
activation_map = explanation.saliency_map["per_image_map"]
plt.imshow(activation_map)
plt.show()
INFO:openvino_xai:Target insertion layer is not provided - trying to find it in auto mode.
INFO:openvino_xai:Using ActivationMap method (for CNNs).
INFO:openvino_xai:Explaining the model in white-box mode.