Performance Information F.A.Q.#
How often do performance benchmarks get updated?
New performance benchmarks are typically published on every
major.minor release of the Intel® Distribution of OpenVINO™ toolkit.
Where can I find the models used in the performance benchmarks?
All models used are published on Hugging Face.
Will there be any new models added to the list used for benchmarking?
The models used in the performance benchmarks were chosen based on general adoption and usage in deployment scenarios. New models that support a diverse set of workloads and usage are added periodically.
How can I run the benchmark results on my own?
All of the performance benchmarks on traditional network models are generated using the open-source tool within the Intel® Distribution of OpenVINO™ toolkit called benchmark_app.
For diffusers (Stable-Diffusion) and foundational models (aka LLMs) please use the OpenVINO GenAI opensource repo OpenVINO GenAI tools/llm_bench
For a simple instruction on testing performance, see the Getting Performance Numbers Guide.
Where can I find a more detailed description of the workloads used for benchmarking?
The image size used in inference depends on the benchmarked network. The table below presents the list of input sizes for each network model and a link to more information on that model:
Model |
Public Network |
Task |
Input Size |
|---|---|---|---|
THUDM |
Transformer |
128K |
|
Hugginface |
Text-To-Text Decoder-only |
8K |
|
Meta AI |
Auto regressive language |
4K |
|
Meta AI |
Auto regressive language |
8K |
|
Meta AI |
Auto regressive language |
128K |
|
Mistral AI |
Auto regressive language |
4096 |
|
Huggingface |
Auto regressive language |
4096 |
|
Huggingface |
Auto regressive language |
128K |
|
Hugginface |
Latent Diffusion Model |
77 |
|
BERT |
question / answer |
128 |
|
Yolov8nano |
object detection |
608x608 |
Where can I purchase the specific hardware used in the benchmarking?
Intel partners with vendors all over the world. For a list of Hardware Manufacturers, see the Intel® AI: In Production Partners & Solutions Catalog. For more details, see the Supported Devices article.
How can I optimize my models for better performance or accuracy?
Set of guidelines and recommendations to optimize models are available in the optimization guide. Join the conversation in the Community Forum for further support.
Why are INT8 optimized models used for benchmarking on CPUs with no VNNI support?
The benefit of low-precision optimization extends beyond processors supporting VNNI through Intel® DL Boost. The reduced bit width of INT8 compared to FP32 allows Intel® CPU to process the data faster. Therefore, it offers better throughput on any converted model, regardless of the intrinsically supported low-precision optimizations within Intel® hardware. For comparison on boost factors for different network models and a selection of Intel® CPU architectures, including AVX-2 with Intel® Core™ i7-8700T, and AVX-512 (VNNI) with Intel® Xeon® 5218T and Intel® Xeon® 8270, refer to the Model Accuracy for INT8 and FP32 Precision
Where can I search for OpenVINO™ performance results based on HW-platforms?
The website format has changed in order to support more common approach of searching for the performance results of a given neural network model on different HW-platforms. As opposed to reviewing performance of a given HW-platform when working with different neural network models.
How is Latency measured?
Latency is measured by running the OpenVINO™ Runtime in synchronous mode. In this mode, each frame or image is processed through the entire set of stages (pre-processing, inference, post-processing) before the next frame or image is processed. This KPI is relevant for applications where the inference on a single image is required. For example, the analysis of an ultra sound image in a medical application or the analysis of a seismic image in the oil & gas industry. Other use cases include real or near real-time applications, e.g. the response of industrial robot to changes in its environment and obstacle avoidance for autonomous vehicles, where a quick response to the result of the inference is required.