OpenVINO™ Model Server is an open-source, production-grade inference platform that exposes a set of models via a convenient inference API over gRPC or HTTP/REST. It employs the inference engine libraries for from the Intel® Distribution of OpenVINO™ toolkit to extend workloads across Intel® hardware including CPU, GPU and others.
OpenVINO™ Model Server is measured in multiple-client-single-server configuration using two hardware platforms connected by ethernet network. The network bandwidth depends on the platforms as well as models under investigation and it is set to not be a bottleneck for workload intensity. This connection is dedicated only to the performance measurements. The benchmark setup is consists of four main parts:
OpenVINO™ Model Server performance benchmark numbers are based on release 2021.3. Performance results are based on testing as of March 15, 2021 and may not reflect all publicly available updates.
Platform with Intel® Xeon® Gold 6252
Server Platform | Client Platform | |
---|---|---|
Motherboard | Intel® Server Board S2600WF H48104-872 | Inspur YZMB-00882-104 NF5280M5 |
Memory | Hynix 16 x 16GB @ 2666 MT/s DDR4 | Samsung 16 x 16GB @ 2666 MT/s DDR4 |
CPU | Intel® Xeon® Gold 6252 CPU @ 2.10GHz | Intel® Xeon® Platinum 8260M CPU @ 2.40GHz |
Selected CPU Flags | Hyper Threading, Turbo Boost, DL Boost | Hyper Threading, Turbo Boost, DL Boost |
CPU Thermal Design Power | 150 W | 162 W |
Operating System | Ubuntu 20.04.2 LTS | Ubuntu 20.04.2 LTS |
Kernel Version | 5.4.0-65-generic | 5.4.0-54-generic |
BIOS Vendor | Intel® Corporation | American Megatrends Inc. |
BIOS Version and Release Date | SE5C620.86B.02.01, date: 03/26/2020 | 4.1.16, date: 06/23/2020 |
Docker Version | 20.10.3 | 20.10.3 |
Network Speed | 40 Gb/s |
Platform with Intel® Core™ i9-10920X
Server Platform | Client Platform | |
---|---|---|
Motherboard | ASUSTeK COMPUTER INC. PRIME X299-A II | ASUSTeK COMPUTER INC. PRIME Z370-P |
Memory | Corsair 4 x 16GB @ 2666 MT/s DDR4 | Corsair 4 x 16GB @ 2133 MT/s DDR4 |
CPU | Intel® Core™ i9-10920X CPU @ 3.50GHz | Intel® Core™ i7-8700T CPU @ 2.40GHz |
Selected CPU Flags | Hyper Threading, Turbo Boost, DL Boost | Hyper Threading, Turbo Boost |
CPU Thermal Design Power | 165 W | 35 W |
Operating System | Ubuntu 20.04.1 LTS | Ubuntu 20.04.1 LTS |
Kernel Version | 5.4.0-52-generic | 5.4.0-56-generic |
BIOS Vendor | American Megatrends Inc. | American Megatrends Inc. |
BIOS Version and Release Date | 0603, date: 03/05/2020 | 2401, date: 07/15/2019 |
Docker Version | 19.03.13 | 19.03.14 |
Network Speed | 10 Gb/s |
Platform with Intel® Core™ i7-8700T
Server Platform | Client Platform | |
---|---|---|
Motherboard | ASUSTeK COMPUTER INC. PRIME Z370-P | ASUSTeK COMPUTER INC. PRIME X299-A II |
Memory | Corsair 4 x 16GB @ 2133 MT/s DDR4 | Corsair 4 x 16GB @ 2666 MT/s DDR4 |
CPU | Intel® Core™ i7-8700T CPU @ 2.40GHz | Intel® Core™ i9-10920X CPU @ 3.50GHz |
Selected CPU Flags | Hyper Threading, Turbo Boost | Hyper Threading, Turbo Boost, DL Boost |
CPU Thermal Design Power | 35 W | 165 W |
Operating System | Ubuntu 20.04.1 LTS | Ubuntu 20.04.1 LTS |
Kernel Version | 5.4.0-56-generic | 5.4.0-52-generic |
BIOS Vendor | American Megatrends Inc. | American Megatrends Inc. |
BIOS Version and Release Date | 2401, date: 07/15/2019 | 0603, date: 03/05/2020 |
Docker Version | 19.03.14 | 19.03.13 |
Network Speed | 10 Gb/s |
Platform with Intel® Core™ i5-8500
Server Platform | Client Platform | |
---|---|---|
Motherboard | ASUSTeK COMPUTER INC. PRIME Z370-A | Gigabyte Technology Co., Ltd. Z390 UD |
Memory | Corsair 2 x 16GB @ 2133 MT/s DDR4 | 029E 4 x 8GB @ 2400 MT/s DDR4 |
CPU | Intel® Core™ i5-8500 CPU @ 3.00GHz | Intel® Core™ i3-8100 CPU @ 3.60GHz |
Selected CPU Flags | Turbo Boost | - |
CPU Thermal Design Power | 65 W | 65 W |
Operating System | Ubuntu 20.04.1 LTS | Ubuntu 20.04.1 LTS |
Kernel Version | 5.4.0-52-generic | 5.4.0-52-generic |
BIOS Vendor | American Megatrends Inc. | American Megatrends Inc. |
BIOS Version and Release Date | 2401, date: 07/12/2019 | F10j, date: 09/16/2020 |
Docker Version | 19.03.13 | 20.10.0 |
Network Speed | 40 Gb/s |
Platform with Intel® Core™ i3-8100
Server Platform | Client Platform | |
---|---|---|
Motherboard | Gigabyte Technology Co., Ltd. Z390 UD | ASUSTeK COMPUTER INC. PRIME Z370-A |
Memory | 029E 4 x 8GB @ 2400 MT/s DDR4 | Corsair 2 x 16GB @ 2133 MT/s DDR4 |
CPU | Intel® Core™ i3-8100 CPU @ 3.60GHz | Intel® Core™ i5-8500 CPU @ 3.00GHz |
Selected CPU Flags | - | Turbo Boost |
CPU Thermal Design Power | 65 W | 65 W |
Operating System | Ubuntu 20.04.1 LTS | Ubuntu 20.04.1 LTS |
Kernel Version | 5.4.0-52-generic | 5.4.0-52-generic |
BIOS Vendor | American Megatrends Inc. | American Megatrends Inc. |
BIOS Version and Release Date | F10j, date: 09/16/2020 | 2401, date: 07/12/2019 |
Docker Version | 20.10.0 | 19.03.13 |
Network Speed | 40 Gb/s |
Results may vary. For workloads and configurations visit: www.intel.com/PerformanceIndex and Legal Information.