Most Efficient Large Language Models for AI PC

This page is regularly updated to help you identify the best-performing LLMs on the Intel® Core™ Ultra processor family and AI PCs.

The tables below list the key performance indicators for a selection of Large Language Models, running on an Intel® Core™ Ultra 7-165H based system, on built-in GPUs.

OpenVINO

Model name	Throughput: (tokens/sec. 2nd token)	1st token latency (msec)	Max RSS memory used. (MB)	Input tokens	Output tokens	Model Precision	Beam	Batch size	Framework
OPT-2.7b	20.2	2757	7084	937	128	INT4	1	1	PT
Phi-3-mini-4k-instruct	19.9	2776	7028	1062	128	INT4	1	1	PT
Orca-mini-3b	19.2	2966	7032	1024	128	INT4	1	1	PT
Phi-2	17.8	2162	7032	1024	128	INT4	1	1	PT
Stable-Zephyr-3b-dpo	17.0	1791	7007	946	128	INT4	1	1	PT
ChatGLM3-6b	16.5	3569	6741	1024	128	INT4	1	1	PT
Dolly-v2-3b	15.8	6891	6731	1024	128	INT4	1	1	PT
Stablelm-3b-4e1t	15.7	2051	7018	1024	128	INT4	1	1	PT
Red-Pajama-Incite-Chat-3b-V1	14.8	6582	7028	1020	128	INT4	1	1	PT
Falcon-7b-instruct	14.5	4552	7033	1049	128	INT4	1	1	PT
Codegen25-7b	13.3	3982	6732	1024	128	INT4	1	1	PT
GPT-j-6b	13.2	7213	6882	1024	128	INT4	1	1	PT
Stablelm-7b	12.8	6339	7013	1020	128	INT4	1	1	PT
Llama-3-8b	12.8	4356	6953	1024	128	INT4	1	1	PT
Llama-2-7b-chat	12.3	4205	6906	1024	128	INT4	1	1	PT
Llama-7b	11.7	4315	6927	1024	128	INT4	1	1	PT
Mistral-7b-v0.1	10.5	4462	7242	1007	128	INT4	1	1	PT
Zephyr-7b-beta	10.5	4500	7039	1024	128	INT4	1	1	PT
Qwen1.5-7b-chat	9.9	4318	7034	1024	128	INT4	1	1	PT
Baichuan2-7b-chat	9.8	4668	6724	1024	128	INT4	1	1	PT
Qwen-7b-chat	9.0	5141	6996	1024	128	INT4	1	1	PT

OpenVINO Model Server

Product	Model	Framework	Precision	Node	Request Rate	Throughput [tok/s]	TPOT Mean Latency
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8380	0.2	92.75	75.75
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8380	0.3	137.89	98.6
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8380	0.4	182.68	144.36
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8380	0.5	227.02	238.54
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8380	0.6	259.06	679.07
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8380	0.7	267.24	785.75
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8380	0.8	267.77	815.11
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8380	0.9	270.01	827.09
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8380	1.0	268.92	840.1
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8380	2.0	269.6	847.81
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8380	inf	270.55	839.37
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8480+	0.2	92.63	63.23
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8480+	0.4	183.51	105.0
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8480+	0.6	272.59	95.34
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8480+	0.8	359.28	126.61
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8480+	1.0	442.69	169.24
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8480+	1.2	521.61	195.94
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8480+	1.4	589.34	267.43
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8480+	1.6	650.25	291.68
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8480+	1.8	655.39	308.64
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8480+	2.0	680.45	302.09
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8480+	inf	702.42	307.82
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8580	0.2	92.89	54.69
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8580	0.4	184.37	77.0
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8580	0.6	273.06	101.81
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8580	0.8	360.22	135.38
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8580	1.0	442.46	170.65
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8580	1.2	519.5	208.44
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8580	1.4	590.11	252.86
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8580	1.6	651.09	286.93
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8580	1.8	670.74	298.02
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8580	2.0	684.4	299.41
ovms	meta-llama/Llama-2-7b-chat-hf	PT	INT8-CW	Xeon Platinum 8580	inf	701.91	305.9
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8380	0.2	79.24	73.06
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8380	0.3	118.42	90.31
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8380	0.4	157.04	113.23
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8380	0.5	193.85	203.97
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8380	0.6	232.36	253.17
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8380	0.7	260.56	581.45
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8380	0.8	271.97	761.05
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8380	0.9	273.36	787.74
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8380	1.0	272.54	811.37
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8380	2.0	278.07	809.3
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8380	inf	275.71	810.89
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8480+	0.2	78.3	60.37
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8480+	0.4	156.42	69.27
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8480+	0.6	232.27	77.79
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8480+	0.8	307.37	90.07
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8480+	1.0	380.61	104.71
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8480+	1.2	452.18	127.36
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8480+	1.4	519.44	156.18
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8480+	1.6	587.62	169.44
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8480+	1.8	649.94	198.44
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8480+	2.0	707.46	234.44
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8480+	inf	799.46	265.5
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8580	0.2	78.61	54.12
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8580	0.4	156.19	70.38
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8580	0.6	232.36	81.83
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8580	0.8	307.01	101.66
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8580	1.0	376.36	139.62
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8580	1.2	447.75	158.53
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8580	1.4	519.74	160.26
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8580	1.6	582.37	190.22
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8580	1.8	635.46	231.31
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8580	2.0	698.38	247.77
ovms	meta-llama/Meta-Llama-3-8B-Instruct	PT	INT8-CW	Xeon Platinum 8580	inf	843.51	252.12
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8380	0.2	87.18	74.96
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8380	0.3	130.74	92.67
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8380	0.4	172.94	117.03
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8380	0.5	214.71	172.69
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8380	0.6	255.45	282.74
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8380	0.7	280.38	629.68
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8380	0.8	280.55	765.16
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8380	0.9	289.65	765.65
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8380	1.0	290.67	783.47
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8380	2.0	284.14	815.09
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8380	inf	290.39	793.52
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8480+	0.2	88.9	60.04
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8480+	0.4	176.5	70.24
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8480+	0.6	262.04	77.01
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8480+	0.8	346.01	95.29
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8480+	1.0	427.37	114.16
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8480+	1.2	507.86	138.56
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8480+	1.4	582.58	150.72
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8480+	1.6	655.61	166.64
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8480+	1.8	717.9	216.76
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8480+	2.0	774.3	233.49
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8480+	inf	873.93	245.31
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8580	0.2	88.92	56.33
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8580	0.4	175.99	72.72
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8580	0.6	261.96	84.24
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8580	0.8	346.78	101.67
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8580	1.0	427.85	128.33
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8580	1.2	506.17	150.01
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8580	1.4	581.72	167.61
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8580	1.6	651.97	190.91
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8580	1.8	713.2	222.56
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8580	2.0	771.17	232.08
ovms	mistralai/Mistral-7B-v0.1	PT	INT8-CW	Xeon Platinum 8580	inf	839.74	253.74

For complete information on the system config, see: Hardware Platforms [PDF]

To view the data in an editable form, you can download the .csv files here:

Click for OpenVINO LLM results [CSV]

Click for OpenVINO Model Server results [CSV]