Intel® Distribution of OpenVINO™ toolkit Benchmark Results

This benchmark setup includes a single machine on which both the benchmark application and the OpenVINO™ installation reside.

The benchmark application loads the Inference Engine (SW) at run time and executes inferences on the specified hardware inference engine, (CPU, GPU or VPU). The benchmark application measures the time spent on actual inferencing (excluding any pre or post processing) and then reports on the inferences per second (or Frames Per Second). For more information on the benchmark application, please also refer to the entry 5 of the FAQ section.

Devices similar to the ones we have used for benchmarking can be accessed using Intel® DevCloud for the Edge, a remote development environment with access to Intel® hardware and the latest versions of the Intel® Distribution of the OpenVINO™ Toolkit. Learn more or Register here.

Measuring inference performance involves many variables and is extremely use-case and application dependent. We use the below four parameters for measurements, which are key elements to consider for a successful deep learning inference application:

  • Throughput - Measures the number of inferences delivered within a latency threshold. (for example, number of Frames Per Second - FPS). When deploying a system with deep learning inference, select the throughput that delivers the best trade-off between latency and power for the price and performance that meets your requirements.
  • Value - While throughput is important, what is more critical in edge AI deployments is the performance efficiency or performance-per-cost. Application performance in throughput per dollar of system cost is the best measure of value.
  • Efficiency - System power is a key consideration from the edge to the data center. When selecting deep learning solutions, power efficiency (throughput/watt) is a critical factor to consider. Intel designs provide excellent power efficiency for running deep learning workloads.
  • Latency - This measures the synchronous execution of inference requests and is reported in milliseconds. Each inference request (for example: preprocess, infer, postprocess) is allowed to complete before the next is started. This performance metric is relevant in usage scenarios where a single image input needs to be acted upon as soon as possible. An example would be the healthcare sector where medical personnel only request analysis of a single ultra sound scanning image or in real-time or near real-time applications for example an industrial robot's response to actions in its environment or obstacle avoidance for autonomous vehicles.

Platform Configurations

Intel® Distribution of OpenVINO™ toolkit performance benchmark numbers are based on release 2021.3.

Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. Performance results are based on testing as of March 15, 2021 and may not reflect all publicly available updates. See configuration disclosure for details. No product can be absolutely secure.

Performance varies by use, configuration and other factors. Learn more at www.intel.com/PerformanceIndex.

Your costs and results may vary.

© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.

Intel optimizations, for Intel compilers or other products, may not optimize to the same degree for non-Intel products.

Testing by Intel done on: see test date for each HW platform below.

CPU Inference Engines

Intel® Xeon® E-2124G Intel® Xeon® W1290P Intel® Xeon® Silver 4216R
Motherboard ASUS* WS C246 PRO ASUS* WS W480-ACE Intel® Server Board S2600STB
CPU Intel® Xeon® E-2124G CPU @ 3.40GHz Intel® Xeon® W-1290P CPU @ 3.70GHz Intel® Xeon® Silver 4216R CPU @ 2.20GHz
Hyper Threading OFF ON ON
Turbo Setting ON ON ON
Memory 2 x 16 GB DDR4 2666MHz 4 x 16 GB DDR4 @ 2666MHz 12 x 32 GB DDR4 2666MHz
Operating System Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS
Kernel Version 5.3.0-24-generic 5.3.0-24-generic 5.3.0-24-generic
BIOS Vendor American Megatrends Inc.* American Megatrends Inc. Intel Corporation
BIOS Version 0904 607 SE5C620.86B.02.01.
0009.092820190230
BIOS Release April 12, 2019 May 29, 2020 September 28, 2019
BIOS Settings Select optimized default settings,
save & exit 		Select optimized default settings,
save & exit 		Select optimized default settings,
change power policy
to "performance",
save & exit
Batch size 1 1 1
Precision INT8 INT8 INT8
Number of concurrent inference requests 4 5 32
Test Date March 15, 2021 March 15, 2021 March 15, 2021
Power dissipation, TDP in Watt 71 125 125
CPU Price on Mach 15th, 2021, USD
Prices may vary 		213 539 1,002

CPU Inference Engines (continue)

Intel® Xeon® Gold 5218T Intel® Xeon® Platinum 8270 Intel® Xeon® Platinum 8380
Motherboard Intel® Server Board S2600STB Intel® Server Board S2600STB Intel Corporation / WilsonCity
CPU Intel® Xeon® Gold 5218T CPU @ 2.10GHz Intel® Xeon® Platinum 8270 CPU @ 2.70GHz Intel® Xeon® Platinum 8380 CPU @ 2.30GHz
Hyper Threading ON ON ON
Turbo Setting ON ON ON
Memory 12 x 32 GB DDR4 2666MHz 12 x 32 GB DDR4 2933MHz 16 x 16 GB DDR4 3200MHz
Operating System Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS
Kernel Version 5.3.0-24-generic 5.3.0-24-generic 5.3.0-24-generic
BIOS Vendor Intel Corporation Intel Corporation Intel Corporation
BIOS Version SE5C620.86B.02.01.
0009.092820190230 		SE5C620.86B.02.01.
0009.092820190230 		WLYDCRB1.SYS.0020.
P86.2103050636
BIOS Release September 28, 2019 September 28, 2019 March 5, 2021
BIOS Settings Select optimized default settings,
change power policy to "performance",
save & exit 		Select optimized default settings,
change power policy to "performance",
save & exit 		Select optimized default settings,
change power policy to "performance",
save & exit
Batch size 1 1 1
Precision INT8 INT8 INT8
Number of concurrent inference requests 32 52 80
Test Date March 15, 2021 March 15, 2021 March 22, 2021
Power dissipation, TDP in Watt 105 205 270
CPU Price, USD
Prices may vary 		1,349 (on Mach 15th, 2021) 7,405 (on Mach 15th, 2021) 8,099 (on March 26th, 2021)

CPU Inference Engines (continue)

Intel® Core™ i7-8700T Intel® Core™ i9-10920X 11th Gen Intel® Core™ i7-1185G7
Motherboard GIGABYTE* Z370M DS3H-CF ASUS* PRIME X299-A II Intel Corporation
internal/Reference
Validation Platform
CPU Intel® Core™ i7-8700T CPU @ 2.40GHz Intel® Core™ i9-10920X CPU @ 3.50GHz 11th Gen Intel® Core™ i7-1185G7 @ 3.00GHz
Hyper Threading ON ON ON
Turbo Setting ON ON ON
Memory 4 x 16 GB DDR4 2400MHz 4 x 16 GB DDR4 2666MHz 2 x 8 GB DDR4 3200MHz
Operating System Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS
Kernel Version 5.3.0-24-generic 5.3.0-24-generic 5.8.0-05-generic
BIOS Vendor American Megatrends Inc.* American Megatrends Inc.* Intel Corporation
BIOS Version F11 505 TGLSFWI1.R00.3425.
A00.2010162309
BIOS Release March 13, 2019 December 17, 2019 October 16, 2020
BIOS Settings Select optimized default settings,
set OS type to "other",
save & exit 		Default Settings Default Settings
Batch size 1 1 1
Precision INT8 INT8 INT8
Number of concurrent inference requests 4 24 4
Test Date March 15, 2021 March 15, 2021 March 15, 2021
Power dissipation, TDP in Watt 35 165 28
CPU Price on Mach 15th, 2021, USD
Prices may vary 		303 700 426

CPU Inference Engines (continue)

Intel® Core™ i5-8500 Intel® Core™ i5-10500TE
Motherboard ASUS* PRIME Z370-A GIGABYTE* Z490 AORUS PRO AX
CPU Intel® Core™ i5-8500 CPU @ 3.00GHz Intel® Core™ i5-10500TE CPU @ 2.30GHz
Hyper Threading OFF ON
Turbo Setting ON ON
Memory 2 x 16 GB DDR4 2666MHz 2 x 16 GB DDR4 @ 2666MHz
Operating System Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS
Kernel Version 5.3.0-24-generic 5.3.0-24-generic
BIOS Vendor American Megatrends Inc.* American Megatrends Inc.*
BIOS Version 2401 F3
BIOS Release July 12, 2019 March 25, 2020
BIOS Settings Select optimized default settings,
save & exit 		Select optimized default settings,
set OS type to "other",
save & exit
Batch size 1 1
Precision INT8 INT8
Number of concurrent inference requests 3 4
Test Date March 15, 2021 March 15, 2021
Power dissipation, TDP in Watt 65 35
CPU Price on Mach 15th, 2021, USD
Prices may vary 		192 195

CPU Inference Engines (continue)

Intel Atom® x5-E3940 Intel Atom® x6425RE Intel® Core™ i3-8100
Motherboard Intel Corporation /
ElkhartLake LPDDR4x T3 CRB 		GIGABYTE* Z390 UD
CPU Intel Atom® Processor E3940 @ 1.60GHz Intel Atom® x6425RE
Processor @ 1.90GHz 		Intel® Core™ i3-8100 CPU @ 3.60GHz
Hyper Threading OFF OFF OFF
Turbo Setting ON ON OFF
Memory 1 x 8 GB DDR3 1600MHz 2 x 4GB DDR4 3200 MHz 4 x 8 GB DDR4 2400MHz
Operating System Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS
Kernel Version 5.3.0-24-generic 5.8.0-050800-generic 5.3.0-24-generic
BIOS Vendor American Megatrends Inc.* Intel Corporation American Megatrends Inc.*
BIOS Version 5.12 EHLSFWI1.R00.2463.
A03.2011200425 		F8
BIOS Release September 6, 2017 November 22, 2020 May 24, 2019
BIOS Settings Default settings Default settings Select optimized default settings,
set OS type to "other",
save & exit
Batch size 1 1 1
Precision INT8 INT8 INT8
Number of concurrent inference requests 4 4 4
Test Date March 15, 2021 March 15, 2021 March 15, 2021
Power dissipation, TDP in Watt 9.5 12 65
CPU Price, USD
Prices may vary 		34 (on March 15th, 2021) 59 (on March 26th, 2021) 117 (on March 15th, 2021)

Accelerator Inference Engines

Intel® Neural Compute Stick 2 Intel® Vision Accelerator Design
with Intel® Movidius™ VPUs (Mustang-V100-MX8)
VPU 1 X Intel® Movidius™ Myriad™ X MA2485 8 X Intel® Movidius™ Myriad™ X MA2485
Connection USB 2.0/3.0 PCIe X4
Batch size 1 1
Precision FP16 FP16
Number of concurrent inference requests 4 32
Power dissipation, TDP in Watt 2.5 30
CPU Price, USD
Prices may vary 		69 (from March 15, 2021) 1180 (from March 15, 2021)
Host Computer Intel® Core™ i7 Intel® Core™ i5
Motherboard ASUS* Z370-A II Uzelinfo* / US-E1300
CPU Intel® Core™ i7-8700 CPU @ 3.20GHz Intel® Core™ i5-6600 CPU @ 3.30GHz
Hyper Threading ON OFF
Turbo Setting ON ON
Memory 4 x 16 GB DDR4 2666MHz 2 x 16 GB DDR4 2400MHz
Operating System Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS
Kernel Version 5.0.0-23-generic 5.0.0-23-generic
BIOS Vendor American Megatrends Inc.* American Megatrends Inc.*
BIOS Version 411 5.12
BIOS Release September 21, 2018 September 21, 2018
Test Date March 15, 2021 March 15, 2021

Please follow this link for more detailed configuration descriptions: Configuration Details

Results may vary. For workloads and configurations visit: www.intel.com/PerformanceIndex and Legal Information.