Most Efficient Large Language Models for AI PC#

This page is regularly updated to help you identify the best-performing LLMs on the Intel® Core™ Ultra processor family and AI PCs. The current data is as of OpenVINO 2024.4, 20 Nov. 2024.

The tables below list the key performance indicators for inference on built-in GPUs.

9-288V

Search:

Topology	Precision	Input Size	2nd latency (ms)
baichuan2-7b-chat	INT4-MIXED	32	77.3
baichuan2-7b-chat	INT4-MIXED	1024	86.4
bloomz-560m	INT4-MIXED	32	11.1
bloomz-560m	INT8-CW	32	13.9
bloomz-560m	INT4-MIXED	1024	15.2
bloomz-560m	INT8-CW	1024	16.1
bloomz-560m	FP16	32	17.5
bloomz-560m	FP16	1024	19.8
dolly-v2-3b	INT4-MIXED	32	49.9
dolly-v2-3b	INT8-CW	32	55.9

Showing 1 to 10 of 82 entries

…

7-268V

Search:

Topology	Precision	Input Size	2nd latency (ms)
baichuan2-13b-chat	INT4-MIXED	32	93.2
baichuan2-7b-chat	INT4-MIXED	32	57.2
baichuan2-7b-chat	INT4-MIXED	1024	63.5
baichuan2-7b-chat	INT8-CW	32	91.8
baichuan2-7b-chat	INT8-CW	1024	96.8
bloomz-560m	INT8-CW	32	15.1
bloomz-560m	INT4-MIXED	32	15.4
bloomz-560m	INT8-CW	1024	15.8
bloomz-560m	INT4-MIXED	1024	16.4
bloomz-560m	FP16	32	18.4

Showing 1 to 10 of 116 entries

…

7-155H

Search:

Topology	Precision	Input Size	2nd latency (ms)
baichuan2-7b-chat	INT4-MIXED	32	76.8
baichuan2-7b-chat	INT4-MIXED	1024	80.6
bloomz-560m	FP16	1024	23.8
bloomz-560m	FP16	32	25.1
codegen25-7b	INT4-MIXED	32	63.5
codegen25-7b	INT4-MIXED	1024	67.4
decilm-7b-instruct	INT4-MIXED	36	65.3
dolly-v2-3b	INT4-MIXED	32	29.4
dolly-v2-3b	INT4-MIXED	1024	32
dolly-v2-3b	INT8-CW	32	42.5

Showing 1 to 10 of 95 entries

…

All models listed here were tested with the following parameters:

Framework: PyTorch
Beam: 1
Batch size: 1

Get system descriptions [PDF]

Get the data in .csv [CSV]