Stable Diffusion v2.1 using Optimum-Intel OpenVINO and multiple Intel Hardware

This Jupyter notebook can be launched after a local installation only.

Github

This notebook will provide you a way to see different precision models performing in different hardware. This notebook was done for showing case the use of Optimum-Intel-OpenVINO and it is not optimized for running multiple times.

image0

Table of contents:

Optimum Intel is the interface between the Transformers and Diffusers libraries and the different tools and libraries provided by Intel to accelerate end-to-end pipelines on Intel architectures. More details in this repository.

Note: We suggest you to create a different environment and run the following installation command there.

%pip install -q "optimum-intel[openvino,diffusers]@git+https://github.com/huggingface/optimum-intel.git" "ipywidgets" "transformers>=4.33.0" --extra-index-url https://download.pytorch.org/whl/cpu

import warnings
warnings.filterwarnings('ignore')

Showing Info Available Devices

The available_devices property shows the available devices in your system. The “FULL_DEVICE_NAME” option to ie.get_property() shows the name of the device. Check what is the ID name for the discrete GPU, if you have integrated GPU (iGPU) and discrete GPU (dGPU), it will show device_name="GPU.0" for iGPU and device_name="GPU.1" for dGPU. If you just have either an iGPU or dGPU that will be assigned to "GPU"

Note: For more details about GPU with OpenVINO visit this link. If you have been facing any issue in Ubuntu 20.04 or Windows 11 read this blog.

from openvino.runtime import Core

ie = Core()
devices = ie.available_devices

for device in devices:
    device_name = ie.get_property(device, "FULL_DEVICE_NAME")
    print(f"{device}: {device_name}")

CPU: Intel(R) Xeon(R) Gold 6348 CPU @ 2.60GHz
GPU: Intel(R) Data Center GPU Flex 170 (dGPU)

Using full precision model in CPU with StableDiffusionPipeline

from diffusers import StableDiffusionPipeline

import gc

model_id = "stabilityai/stable-diffusion-2-1-base"
pipe = StableDiffusionPipeline.from_pretrained(model_id)
pipe.save_pretrained("./stabilityai_cpu")
prompt = "red car in snowy forest"
output_cpu = pipe(prompt, num_inference_steps=17).images[0]
output_cpu.save("image_cpu.png")
output_cpu

Fetching 13 files:   0%|          | 0/13 [00:00<?, ?it/s]

Downloading model.safetensors:   0%|          | 0.00/1.36G [00:00<?, ?B/s]

Downloading (…)ch_model.safetensors:   0%|          | 0.00/335M [00:00<?, ?B/s]

Downloading (…)ch_model.safetensors:   0%|          | 0.00/3.46G [00:00<?, ?B/s]

0%|          | 0/17 [00:00<?, ?it/s]
../_images/236-stable-diffusion-v2-optimum-demo-comparison-with-output_7_5.png 
del pipe
gc.collect()

Using full precision model in CPU with OVStableDiffusionPipeline

from optimum.intel.openvino import OVStableDiffusionPipeline

model_id = "stabilityai/stable-diffusion-2-1-base"
ov_pipe = OVStableDiffusionPipeline.from_pretrained(model_id, export=True, compile=False)
ov_pipe.reshape(batch_size=1, height=512, width=512, num_images_per_prompt=1)
ov_pipe.save_pretrained("./openvino_ir")
ov_pipe.compile()

Framework not specified. Using pt to export to ONNX.
Keyword arguments {'subfolder': '', 'config': {'_class_name': 'StableDiffusionPipeline', '_diffusers_version': '0.10.0.dev0', 'feature_extractor': ['transformers', 'CLIPImageProcessor'], 'requires_safety_checker': False, 'safety_checker': [None, None], 'scheduler': ['diffusers', 'PNDMScheduler'], 'text_encoder': ['transformers', 'CLIPTextModel'], 'tokenizer': ['transformers', 'CLIPTokenizer'], 'unet': ['diffusers', 'UNet2DConditionModel'], 'vae': ['diffusers', 'AutoencoderKL']}} are not expected by StableDiffusionPipeline and will be ignored.
Using framework PyTorch: 2.0.1+cu117

============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

Using framework PyTorch: 2.0.1+cu117
Saving external data to one file...

============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

Using framework PyTorch: 2.0.1+cu117
Using framework PyTorch: 2.0.1+cu117

============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

Compiling the text_encoder...
Compiling the vae_decoder...
Compiling the unet...

prompt = "red car in snowy forest"
output_cpu_ov = ov_pipe(prompt, num_inference_steps=17).images[0]
output_cpu_ov.save("image_ov_cpu.png")
output_cpu_ov

0%|          | 0/18 [00:00<?, ?it/s]
../_images/236-stable-diffusion-v2-optimum-demo-comparison-with-output_11_1.png

Using full precision model in dGPU with OVStableDiffusionPipeline

The model in this notebook is FP32 precision. And thanks to the new feature of OpenVINO 2023.0 you do not need to convert the model to FP16 for running the inference on GPU.

ov_pipe.to("GPU")
ov_pipe.compile()

Compiling the text_encoder...
Compiling the vae_decoder...
Compiling the unet...

prompt = "red car in snowy forest"
output_gpu_ov = ov_pipe(prompt, num_inference_steps=17).images[0]
output_gpu_ov.save("image_ov_gpu.png")
output_gpu_ov

del ov_pipe
gc.collect()