Intel® Distribution of OpenVINO™ toolkit Benchmark Results

This benchmark setup includes a single machine on which both the benchmark application and the OpenVINO™ installation reside.

The benchmark application loads the OpenVINO™ Runtime and executes inferences on the specified hardware (CPU, GPU or VPU). The benchmark application measures the time spent on actual inferencing (excluding any pre or post processing) and then reports on the inferences per second (or Frames Per Second). To learn how to benchmark your own model in OpenVINO, visit the benchmark application documentation.

Measuring inference performance involves many variables and is extremely use-case and application dependent. We use the below four parameters for measurements, which are key elements to consider for a successful deep learning inference application:

  • Throughput - Measures the number of inferences delivered within a latency threshold. (for example, number of Frames Per Second - FPS). When deploying a system with deep learning inference, select the throughput that delivers the best trade-off between latency and power for the price and performance that meets your requirements.

  • Value - While throughput is important, what is more critical in edge AI deployments is the performance efficiency or performance-per-cost. Application performance in throughput per dollar of system cost is the best measure of value.

  • Efficiency - System power is a key consideration from the edge to the data center. When selecting deep learning solutions, power efficiency (throughput/watt) is a critical factor to consider. Intel designs provide excellent power efficiency for running deep learning workloads.

  • Latency - This measures the synchronous execution of inference requests and is reported in milliseconds. Each inference request (for example: preprocess, infer, postprocess) is allowed to complete before the next is started. This performance metric is relevant in usage scenarios where a single image input needs to be acted upon as soon as possible. An example would be the healthcare sector where medical personnel only request analysis of a single ultra sound scanning image or in real-time or near real-time applications for example an industrial robot’s response to actions in its environment or obstacle avoidance for autonomous vehicles.

bert-base-cased [124]

bert-large-uncased-whole-word-masking-squad-int8-0001 [384]

deeplabv3-TF [513x513]

densenet-121-TF [224x224]

efficientdet-d0 [512x512]

faster-rcnn-resnet50-coco-TF [600x1024]

inception-v4-TF [299x299]

mobilenet-ssd-CF [300x300]

mobilenet-v2-pytorch [224x224]

resnet-18-pytorch [224x224]

resnet_50_TF [224x224]

ssd-resnet34-1200-onnx [1200x1200]

unet-camvid-onnx-0001 [368x480]

yolo-v3-tiny-tf [416x416]

yolo_v4-tf [608x608]

Platform Configurations

Intel® Distribution of OpenVINO™ toolkit performance benchmark numbers are based on release 2022.1.

Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. Performance results are based on testing as of March 17, 2022 and may not reflect all publicly available updates. See configuration disclosure for details. No product can be absolutely secure.

Performance varies by use, configuration and other factors. Learn more at www.intel.com/PerformanceIndex.

Your costs and results may vary.

© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.

Intel optimizations, for Intel compilers or other products, may not optimize to the same degree for non-Intel products.

Testing by Intel done on: see test date for each HW platform below.

CPU Inference Engines

Configuration

Intel® Xeon® E-2124G

Intel® Xeon® W1290P

Motherboard

ASUS* WS C246 PRO

ASUS* WS W480-ACE

CPU

Intel® Xeon® E-2124G CPU @ 3.40GHz

Intel® Xeon® W-1290P CPU @ 3.70GHz

Hyper Threading

OFF

ON

Turbo Setting

ON

ON

Memory

2 x 16 GB DDR4 2666MHz

4 x 16 GB DDR4 @ 2666MHz

Operating System

Ubuntu* 20.04.3 LTS

Ubuntu* 20.04.3 LTS

Kernel Version

5.4.0-42-generic

5.4.0-42-generic

BIOS Vendor

American Megatrends Inc.*

American Megatrends Inc.

BIOS Version

1901

2301

BIOS Release

September 24, 2021

July 8, 2021

BIOS Settings

Select optimized default settings, save & exit

Select optimized default settings, save & exit

Batch size

1

1

Precision

INT8

INT8

Number of concurrent inference requests

4

5

Test Date

March 17, 2022

March 17, 2022

Rated maximum TDP/socket in Watt

71

125

CPU Price/socket on Feb 14, 2022, USD Prices may vary

213

539

CPU Inference Engines (continue)

Configuration

Intel® Xeon® Silver 4216R

Intel® Xeon® Silver 4316

Motherboard

Intel® Server Board S2600STB

Intel Corporation / WilsonCity

CPU

Intel® Xeon® Silver 4216R CPU @ 2.20GHz

Intel® Xeon® Silver 4316 CPU @ 2.30GHz

Hyper Threading

ON

ON

Turbo Setting

ON

ON

Memory

12 x 32 GB DDR4 2666MHz

16 x 32 GB DDR4 @ 2666MHz

Operating System

Ubuntu* 20.04.3 LTS

Ubuntu* 20.04.3 LTS

Kernel Version

5.3.0-24-generic

5.4.0-100-generic

BIOS Vendor

Intel Corporation

Intel Corporation

BIOS Version

SE5C620.86B.02.01. 0013.121520200651

WLYDCRB1.SYS.0021. P41.2109200451

BIOS Release

December 15, 2020

September 20, 2021

BIOS Settings

Select optimized default settings, change power policy to “performance”, save & exit

Select optimized default settings, save & exit

Batch size

1

1

Precision

INT8

INT8

Number of concurrent inference requests

32

10

Test Date

March 17, 2022

March 17, 2022

Rated maximum TDP/socket in Watt

125

150

CPU Price/socket on June 21, 2021, USD Prices may vary

1,002

1083

CPU Inference Engines (continue)

Configuration

Intel® Xeon® Gold 5218T

Intel® Xeon® Platinum 8270

Intel® Xeon® Platinum 8380

Motherboard

Intel® Server Board S2600STB

Intel® Server Board S2600STB

Intel Corporation / WilsonCity

CPU

Intel® Xeon® Gold 5218T CPU @ 2.10GHz

Intel® Xeon® Platinum 8270 CPU @ 2.70GHz

Intel® Xeon® Platinum 8380 CPU @ 2.30GHz

Hyper Threading

ON

ON

ON

Turbo Setting

ON

ON

ON

Memory

12 x 32 GB DDR4 2666MHz

12 x 32 GB DDR4 2933MHz

16 x 16 GB DDR4 3200MHz

Operating System

Ubuntu* 20.04.3 LTS

Ubuntu* 20.04.3 LTS

Ubuntu* 20.04.1 LTS

Kernel Version

5.3.0-24-generic

5.3.0-24-generic

5.4.0-64-generic

BIOS Vendor

Intel Corporation

Intel Corporation

Intel Corporation

BIOS Version

SE5C620.86B.02.01. 0013.121520200651

SE5C620.86B.02.01. 0013.121520200651

WLYDCRB1.SYS.0020. P86.2103050636

BIOS Release

December 15, 2020

December 15, 2020

March 5, 2021

BIOS Settings

Select optimized default settings, change power policy to “performance”, save & exit

Select optimized default settings, change power policy to “performance”, save & exit

Select optimized default settings, change power policy to “performance”, save & exit

Batch size

1

1

1

Precision

INT8

INT8

INT8

Number of concurrent inference requests

32

52

80

Test Date

March 17, 2022

March 17, 2022

March 17, 2022

Rated maximum TDP/socket in Watt

105

205

270

CPU Price/socket on Feb 14, 2022, USD Prices may vary

1,349

7,405

8,099

CPU Inference Engines (continue)

Configuration

Intel® Core™ i9-10920X

Intel® Core™ i9-10900TE

Intel® Core™ i9-12900

Motherboard

ASUS* PRIME X299-A II

B595

Intel Corporation internal/Reference Validation Platform

CPU

Intel® Core™ i9-10920X CPU @ 3.50GHz

Intel® Core™ i9-10900TE CPU @ 1.80GHz

12th Gen Intel® Core™ i9-12900

Hyper Threading

ON

ON

OFF

Turbo Setting

ON

ON

Memory

4 x 16 GB DDR4 2666MHz

2 x 8 GB DDR4 @ 2400 MHz

4 x 8 GB DDR4 4800MHz

Operating System

Ubuntu 20.04.3 LTS

Ubuntu 20.04.3 LTS

Microsoft Windows 10 Pro

Kernel Version

5.4.0-42-generic

5.4.0-42-generic

10.0.19043 N/A Build 19043

BIOS Vendor

American Megatrends Inc.*

American Megatrends Inc.*

Intel Corporation

BIOS Version

1004

Z667AR10.BIN

ADLSFWI1.R00.2303. B00.2107210432

BIOS Release

March 19, 2021

July 15, 2020

July 21, 2021

BIOS Settings

Default Settings

Default Settings

Default Settings

Batch size

1

1

1

Precision

INT8

INT8

INT8

Number of concurrent inference requests

24

5

4

Test Date

March 17, 2022

March 17, 2022

March 17, 2022

Rated maximum TDP/socket in Watt

165

35

65

CPU Price/socket on Feb 14, 2022, USD Prices may vary

700

444

519

CPU Inference Engines (continue)

Configuration

Intel® Core™ i7-8700T

Intel® Core™ i7-1185G7

Motherboard

GIGABYTE* Z370M DS3H-CF

Intel Corporation internal/Reference Validation Platform

CPU

Intel® Core™ i7-8700T CPU @ 2.40GHz

Intel® Core™ i7-1185G7 @ 3.00GHz

Hyper Threading

ON

ON

Turbo Setting

ON

ON

Memory

4 x 16 GB DDR4 2400MHz

2 x 8 GB DDR4 3200MHz

Operating System

Ubuntu 20.04.3 LTS

Ubuntu 20.04.3 LTS

Kernel Version

5.4.0-42-generic

5.8.0-050800-generic

BIOS Vendor

American Megatrends Inc.*

Intel Corporation

BIOS Version

F14c

TGLSFWI1.R00.4391. A00.2109201819

BIOS Release

March 23, 2021

September 20, 2021

BIOS Settings

Select optimized default settings, set OS type to “other”, save & exit

Default Settings

Batch size

1

1

Precision

INT8

INT8

Number of concurrent inference requests

4

4

Test Date

March 17, 2022

March 17, 2022

Rated maximum TDP/socket in Watt

35

28

CPU Price/socket on Feb 14, 2022, USD Prices may vary

303

426

CPU Inference Engines (continue)

Configuration

Intel® Core™ i3-8100

Intel® Core™ i5-8500

Intel® Core™ i5-10500TE

Motherboard

GIGABYTE* Z390 UD

ASUS* PRIME Z370-A

GIGABYTE* Z490 AORUS PRO AX

CPU

Intel® Core™ i3-8100 CPU @ 3.60GHz

Intel® Core™ i5-8500 CPU @ 3.00GHz

Intel® Core™ i5-10500TE CPU @ 2.30GHz

Hyper Threading

OFF

OFF

ON

Turbo Setting

OFF

ON

ON

Memory

4 x 8 GB DDR4 2400MHz

2 x 16 GB DDR4 2666MHz

2 x 16 GB DDR4 @ 2666MHz

Operating System

Ubuntu* 20.04.3 LTS

Ubuntu* 20.04.3 LTS

Ubuntu* 20.04.3 LTS

Kernel Version

5.3.0-24-generic

5.4.0-42-generic

5.4.0-42-generic

BIOS Vendor

American Megatrends Inc.*

American Megatrends Inc.*

American Megatrends Inc.*

BIOS Version

F8

3004

F21

BIOS Release

May 24, 2019

July 12, 2021

November 23, 2021

BIOS Settings

Select optimized default settings, set OS type to “other”, save & exit

Select optimized default settings, save & exit

Select optimized default settings, set OS type to “other”, save & exit

Batch size

1

1

1

Precision

INT8

INT8

INT8

Number of concurrent inference requests

4

3

4

Test Date

March 17, 2022

March 17, 2022

March 17, 2022

Rated maximum TDP/socket in Watt

65

65

35

CPU Price/socket on Feb 14, 2022, USD Prices may vary

117

192

195

CPU Inference Engines (continue)

Configuration

Intel Atom® x5-E3940

Intel Atom® x6425RE

Intel® Celeron® 6305E

Motherboard

Intel Corporation internal/Reference Validation Platform

Intel Corporation internal/Reference Validation Platform

Intel Corporation internal/Reference Validation Platform

CPU

Intel Atom® Processor E3940 @ 1.60GHz

Intel Atom® x6425RE Processor @ 1.90GHz

Intel® Celeron® 6305E @ 1.80GHz

Hyper Threading

OFF

OFF

OFF

Turbo Setting

ON

ON

ON

Memory

1 x 8 GB DDR3 1600MHz

2 x 4GB DDR4 3200MHz

2 x 8 GB DDR4 3200MHz

Operating System

Ubuntu* 20.04.3 LTS

Ubuntu* 20.04.3 LTS

Ubuntu 20.04.3 LTS

Kernel Version

5.4.0-42-generic

5.13.0-27-generic

5.13.0-1008-intel

BIOS Vendor

American Megatrends Inc.*

Intel Corporation

Intel Corporation

BIOS Version

5.12

EHLSFWI1.R00.3273. A01.2106300759

TGLIFUI1.R00.4064.A02.2102260133

BIOS Release

September 6, 2017

June 30, 2021

February 26, 2021

BIOS Settings

Default settings

Default settings

Default settings

Batch size

1

1

1

Precision

INT8

INT8

INT8

Number of concurrent inference requests

4

4

4

Test Date

March 17, 2022

March 17, 2022

March 17, 2022

Rated maximum TDP/socket in Watt

9.5

12

15

CPU Price/socket on Feb 14, 2022, USD Prices may vary

34

59

107

Accelerator Inference Engines

Configuration

Intel® Neural Compute Stick 2

Intel® Vision Accelerator Design with Intel® Movidius™ VPUs (Mustang-V100-MX8)

VPU

1 X Intel® Movidius™ Myriad™ X MA2485

8 X Intel® Movidius™ Myriad™ X MA2485

Connection

USB 2.0/3.0

PCIe X4

Batch size

1

1

Precision

FP16

FP16

Number of concurrent inference requests

4

32

Rated maximum TDP/socket in Watt

2.5

30

CPU Price/socket on Feb 14, 2022, USD Prices may vary

69

492

Host Computer

Intel® Core™ i7

Intel® Core™ i5

Motherboard

ASUS* Z370-A II

Uzelinfo* / US-E1300

CPU

Intel® Core™ i7-8700 CPU @ 3.20GHz

Intel® Core™ i5-6600 CPU @ 3.30GHz

Hyper Threading

ON

OFF

Turbo Setting

ON

ON

Memory

4 x 16 GB DDR4 2666MHz

2 x 16 GB DDR4 2400MHz

Operating System

Ubuntu* 20.04.3 LTS

Ubuntu* 20.04.3 LTS

Kernel Version

5.0.0-23-generic

5.0.0-23-generic

BIOS Vendor

American Megatrends Inc.*

American Megatrends Inc.*

BIOS Version

411

5.12

BIOS Release

September 21, 2018

September 21, 2018

Test Date

March 17, 2022

March 17, 2022

For more detailed configuration descriptions, see Configuration Details.