Stable Diffusion v2.1 using Optimum-Intel OpenVINO and multiple Intel Hardware

This tutorial is also available as a Jupyter notebook that can be cloned directly from GitHub. See the installation guide for instructions to run this tutorial locally on Windows, Linux or macOS.

Github

image0

Table of contents:

Optimum Intel is the interface between the Transformers and Diffusers libraries and the different tools and libraries provided by Intel to accelerate end-to-end pipelines on Intel architectures. More details in this repository.

Note

We suggest you to create a different environment and run the following installation command there.

%pip install -q "optimum-intel[openvino,diffusers]" "ipywidgets"

import warnings
warnings.filterwarnings('ignore')

Showing Info Available Devices ⇑

The available_devices property shows the available devices in your system. The “FULL_DEVICE_NAME” option to ie.get_property() shows the name of the device. Check what is the ID name for the discrete GPU, if you have integrated GPU (iGPU) and discrete GPU (dGPU), it will show device_name="GPU.0" for iGPU and device_name="GPU.1" for dGPU. If you just have either an iGPU or dGPU that will be assigned to "GPU"

Note

For more details about GPU with OpenVINO visit this link. If you have been facing any issue in Ubuntu 20.04 or Windows 11 read this blog.

from openvino.runtime import Core

ie = Core()
devices = ie.available_devices

for device in devices:
    device_name = ie.get_property(device, "FULL_DEVICE_NAME")
    print(f"{device}: {device_name}")

CPU: Intel(R) Xeon(R) Gold 6348 CPU @ 2.60GHz
GPU: Intel(R) Data Center GPU Flex 170 (dGPU)

Using full precision model in CPU with StableDiffusionPipeline. ⇑

from diffusers import StableDiffusionPipeline

model_id = "stabilityai/stable-diffusion-2-1-base"
pipe = StableDiffusionPipeline.from_pretrained(model_id)
pipe.save_pretrained("./stabilityai_cpu")
prompt = "red car in snowy forest"
output_cpu = pipe(prompt, num_inference_steps=17).images[0]
output_cpu.save("image_cpu.png")
output_cpu

Fetching 13 files:   0%|          | 0/13 [00:00<?, ?it/s]

Downloading model.safetensors:   0%|          | 0.00/1.36G [00:00<?, ?B/s]

Downloading (…)ch_model.safetensors:   0%|          | 0.00/335M [00:00<?, ?B/s]

Downloading (…)ch_model.safetensors:   0%|          | 0.00/3.46G [00:00<?, ?B/s]

0%|          | 0/17 [00:00<?, ?it/s]
../_images/236-stable-diffusion-v2-optimum-demo-comparison-with-output_7_5.png

Using full precision model in CPU with OVStableDiffusionPipeline. ⇑

from optimum.intel.openvino import OVStableDiffusionPipeline

model_id = "stabilityai/stable-diffusion-2-1-base"
ov_pipe = OVStableDiffusionPipeline.from_pretrained(model_id, export=True, compile=False)
ov_pipe.reshape(batch_size=1, height=512, width=512, num_images_per_prompt=1)
ov_pipe.save_pretrained("./openvino_ir")
ov_pipe.compile()

Framework not specified. Using pt to export to ONNX.
Keyword arguments {'subfolder': '', 'config': {'_class_name': 'StableDiffusionPipeline', '_diffusers_version': '0.10.0.dev0', 'feature_extractor': ['transformers', 'CLIPImageProcessor'], 'requires_safety_checker': False, 'safety_checker': [None, None], 'scheduler': ['diffusers', 'PNDMScheduler'], 'text_encoder': ['transformers', 'CLIPTextModel'], 'tokenizer': ['transformers', 'CLIPTokenizer'], 'unet': ['diffusers', 'UNet2DConditionModel'], 'vae': ['diffusers', 'AutoencoderKL']}} are not expected by StableDiffusionPipeline and will be ignored.
Using framework PyTorch: 2.0.1+cu117

============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

Using framework PyTorch: 2.0.1+cu117
Saving external data to one file...

============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

Using framework PyTorch: 2.0.1+cu117
Using framework PyTorch: 2.0.1+cu117

============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

Compiling the text_encoder...
Compiling the vae_decoder...
Compiling the unet...

prompt = "red car in snowy forest"
output_cpu_ov = ov_pipe(prompt, num_inference_steps=17).images[0]
output_cpu_ov.save("image_ov_cpu.png")
output_cpu_ov

0%|          | 0/18 [00:00<?, ?it/s]
../_images/236-stable-diffusion-v2-optimum-demo-comparison-with-output_10_1.png

Using full precision model in dGPU with OVStableDiffusionPipeline. ⇑

The model in this notebook is FP32 precision. And thanks to the new feature of OpenVINO 2023.0 you do not need to convert the model to FP16 for running the inference on GPU.

ov_pipe.to("GPU")
ov_pipe.compile()

Compiling the text_encoder...
Compiling the vae_decoder...
Compiling the unet...

prompt = "red car in snowy forest"
output_gpu_ov = ov_pipe(prompt, num_inference_steps=17).images[0]
output_gpu_ov.save("image_ov_gpu.png")
output_gpu_ov

0%|          | 0/18 [00:00<?, ?it/s]
../_images/236-stable-diffusion-v2-optimum-demo-comparison-with-output_13_1.png