OpenVINO™ Test Drive#

OpenVINO™ Test Drive is a cross-platform graphic user interface application for running and testing AI models, both generative and vision based. It can run directly on your computer or on edge devices using OpenVINO™ Runtime.

OpenVINO™ Test Drive is developed under the openvino_testdrive repository.

Use OpenVINO™ Test Drive to:

  • Chat with LLMs and evaluate model performance on your computer or edge device;

  • Experiment with different text prompts to generate images, using Stable Diffusion and Stable DiffusionXL models;

  • Transcribe speech from video, using Whisper models, including generation of timestamps;

  • Run inference of models trained by Intel® Geti™ and visualize the results.

Installation (Windows)#

Important

For Intel® NPU, use the latest available version of Intel® NPU Driver.

  1. Download the latest archive from the release repository. To verify the integrity of the downloaded package, use the SHA-256 file attached.

  2. Extract the zip file and run the MSIX installation package. Click the Install button to proceed.

  3. Launch OpenVINO™ Test Drive, clicking the application name in the Windows app list.

Quick start#

When starting the application, you can import an LLM model from Hugging Face Hub or upload an Intel® Geti™ model from a local drive.

Text generation and LLM performance evaluation#

  1. Find a model on Hugging Face and import it.

    how to import a model to test drive

  2. Chat with LLMs via the Playground tab. You can export an LLM by clicking the Export model button.

    chatting with llm models in test drive

  3. Use the Performance metrics tab to get model performance metrics on your computer or an edge device.

    verifying llm performance in test drive

Retrieval-Augmented Generation with LLMs#

  1. Upload files and create knowledge base for RAG (Retrieval-Augmented Generation), using Knowledge base tab.

    creating a knowledge base for RAG in test drive

    The knowledge base can be used for text generation with LLM models.

    using a knowledge base for text generation with LLMs

  2. You can also upload a document directly, using the Playground` tab.

    uploading a document to the knowledge base

Image analysis with Visual Language Models (VLMs)#

  1. Import a VLM for image analysis.

    importing a visual language model for image analysis

  2. Select the VLM from the My models section, upload an image and analyze it.

    importing a visual language model for image analysis

Video transcription with Whisper models#

  1. Import a Whisper model for video transcription.

    importing a Whisper model for video transcription

  2. Select the speech-to-text LLM from the My models section, and upload a video for transcription.

    importing a visual language model for image analysis

    You can search for words in the transcript or download it.

  3. Use the Performance metrics tab to get performance metrics of the LLM on your computer or an edge device.

Image generation with LLMs#

  1. Import an image generation LLM from a predefined set of popular models or from Hugging Face, using Import model -> Hugging Face.

  2. Select the LLM from the My models section and start the chat to generate an image. You can export the model by clicking the Export model button.

    image generation with a chosen LLM

    You can download the generated image.

  3. Use the Performance metrics tab to get performance metrics of the LLM on your computer or an edge device.

Inference of models trained with Intel® Geti™#

  1. Download the deployment code for a model in the OpenVINO IR format trained by Intel® Geti™ (refer to the Intel® Geti™ documentation for more details).

    verifying llm performance in test drive

  2. Import the deployment code into OpenVINO™ Test Drive, using the Import model and then Local disk buttons.

  3. Use the Live inference tab to run and visualize results of inference of individual images.

  4. For batch inference, use the Batch inference tab and provide paths to the folder with input images, as well as one for batch inference results. You can do so by filling out the Source folder and Destination folder fields. Click Start to start batch inference.

Build the Application#

  1. Make sure you Install flutter SDK and all its platform-specific dependencies.

  2. Build the bindings and place them in the ./bindings folder.

    OpenVINO™ Test Drive uses bindings to OpenVINO™ GenAI and OpenVINO™ Model API, which are located in the ./openvino_bindings folder. Refer to the GitHub page for more details.

  3. Start the application, using the following command:

    flutter run

Additional Resources#

  • OpenVINO™ - a software toolkit for optimizing and deploying deep learning models.

  • GenAI Repository and OpenVINO Tokenizers - resources and tools for developing and optimizing Generative AI applications.

  • Intel® Geti™ - software for building computer vision models.

  • OpenVINO™ Model API - a set of wrapper classes for particular tasks and model architectures. It simplifies routine procedures, preprocessing and postprocessing of data.