Most Efficient Large Language Models for AI PC

This page is regularly updated to help you identify the best-performing LLMs on the Intel® Core™ Ultra processor family and AI PCs. The current data is as of OpenVINO 2026.4, 23 September 2026.

The tables below list the key performance indicators for inference on built-in GPUs.

7-368H

Topology	Precision	Input Size	1st latency (ms)	2nd latency (ms)	2nd token per sec
tiny-llama-1.1b-chat	INT4-MIXED	32	14.4	6.6	151.5151515
tiny-llama-1.1b-chat	INT4-MIXED	1024	40	6.8	147.0588235
tiny-llama-1.1b-chat	INT4-MIXED	32	14.9	6.9	144.9275362
tiny-llama-1.1b-chat	INT4-MIXED	1024	54.7	7.2	138.8888889
minicpm-1b-sft	INT4-MIXED	31	34.7	10.1	99.00990099
minicpm-1b-sft	INT4-MIXED	31	37	10.2	98.03921569
minicpm-1b-sft	INT4-MIXED	31	34.9	10.2	98.03921569
minicpm-1b-sft	INT4-MIXED	1014	75.1	10.7	93.45794393
minicpm-1b-sft	INT4-MIXED	1014	64.4	10.7	93.45794393
minicpm-1b-sft	INT4-MIXED	1014	66.6	10.8	92.59259259
tiny-llama-1.1b-chat	INT8-CW	32	16.3	10.8	92.59259259
tiny-llama-1.1b-chat	INT8-CW	1024	47.3	11.1	90.09009009
gemma-2b-it	INT4-MIXED	32	19.2	15.3	65.35947712
gemma-2b-it	INT4-MIXED	1024	91.3	15.6	64.1025641
minicpm-1b-sft	INT8-CW	31	43.6	15.7	63.69426752
gemma-2b-it	INT4-MIXED	32	21	15.8	63.29113924
gemma-2b-it	INT4-MIXED	1024	106.7	16.1	62.11180124
minicpm-1b-sft	INT8-CW	1014	72.2	16.5	60.60606061
gemma-2-2b	INT4-MIXED	33	26.9	17.2	58.13953488
gemma-2-2b	INT4-MIXED	1025	108.4	17.4	57.47126437
gemma-2-2b	INT4-MIXED	33	29.2	17.5	57.14285714
gemma-2-2b	INT4-MIXED	1025	130.1	17.9	55.86592179
tiny-llama-1.1b-chat	FP16	32	21.5	19.7	50.76142132
tiny-llama-1.1b-chat	FP16	1024	62.1	20	50
gemma-2b-it	INT8-CW	32	28.1	24.9	40.16064257
gemma-2b-it	INT8-CW	1024	111.5	25.3	39.5256917
gemma-2-2b	INT8-CW	33	32.5	26.2	38.16793893
gemma-2-2b	INT8-CW	1025	125.3	26.9	37.17472119
minicpm-1b-sft	FP16	31	35.4	27.9	35.84229391
minicpm-1b-sft	FP16	1014	87.8	28.9	34.60207612
codellama-7b	INT4-MIXED	32	38.2	32.5	30.76923077
codellama-7b	INT4-MIXED	32	42.1	34	29.41176471
codellama-7b	INT4-MIXED	1024	228.2	34.3	29.15451895
neural-chat-7b-v3-3	INT4-MIXED	32	41	34.6	28.9017341
zephyr-7b-beta	INT4-MIXED	32	41.1	34.6	28.9017341
zephyr-7b-beta	INT4-MIXED	1024	239.6	35.8	27.93296089
codellama-7b	INT4-MIXED	1024	280.7	35.9	27.8551532
neural-chat-7b-v3-3	INT4-MIXED	1024	239.3	35.9	27.8551532
neural-chat-7b-v3-3	INT4-MIXED	32	45.2	36.2	27.62430939
qwen-7b-chat	INT4-MIXED	32	43.1	37.1	26.9541779
neural-chat-7b-v3-3	INT4-MIXED	1024	299.9	37.5	26.66666667
qwen-7b-chat	INT4-MIXED	32	45.4	38.2	26.17801047
bloomz-7b1	INT4-MIXED	32	45.6	38.6	25.90673575
qwen-7b-chat	INT4-MIXED	1024	244.6	38.9	25.70694087
bloomz-7b1	INT4-MIXED	32	49.2	40	25
qwen-7b-chat	INT4-MIXED	1024	256.7	40	25
bloomz-7b1	INT4-MIXED	1024	325.8	40.4	24.75247525
bloomz-7b1	INT4-MIXED	1024	386.1	41.7	23.98081535
zephyr-7b-beta	INT4-MIXED	32	50.7	41.8	23.92344498
zephyr-7b-beta	INT4-MIXED	1024	258	43	23.25581395
baichuan2-7b-chat	INT4-MIXED	32	51.5	43.1	23.20185615
baichuan2-7b-chat	INT4-MIXED	1024	317.4	45	22.22222222
gemma-2b-it	FP16	32	49.4	46.5	21.50537634
gemma-2b-it	FP16	1024	166.1	46.7	21.41327623
qwen-7b-chat	INT4-MIXED	32	56.4	49.1	20.36659878
gemma-2-2b	FP16	33	53.1	50.2	19.92031873
qwen-7b-chat	INT4-MIXED	1024	266.3	51	19.60784314
gemma-2-2b	FP16	1025	162	51.1	19.56947162
dolly-v2-12b	INT4-MIXED	32	73	59.3	16.86340641
lcm-sdxl	INT8-CW	1024	62	61.4	16.28664495
codellama-7b	INT8-CW	32	67.4	61.5	16.2601626
lcm-sdxl	INT8-CW	32	62	61.5	16.2601626
dolly-v2-12b	INT4-MIXED	1024	656.3	61.9	16.15508885
codellama-7b	INT8-CW	1024	267.4	64.3	15.55209953
lcm-sdxl	INT8-HYBRID	32	65.7	64.9	15.40832049
baichuan2-7b-chat	INT8-CW	32	71.6	65	15.38461538
lcm-sdxl	INT8-HYBRID	1024	65.9	65.1	15.3609831
neural-chat-7b-v3-3	INT8-CW	32	71.8	65.5	15.26717557
zephyr-7b-beta	INT8-CW	32	72	65.5	15.26717557
qwen-7b-chat	INT8-CW	32	72	66	15.15151515
neural-chat-7b-v3-3	INT8-CW	1024	284.9	66.8	14.97005988
zephyr-7b-beta	INT8-CW	1024	283.7	66.8	14.97005988
lcm-sdxl	INT8-CW	32	67.9	67.5	14.81481481
baichuan2-7b-chat	INT8-CW	1024	305.4	67.7	14.77104874
lcm-sdxl	INT8-CW	1024	67.8	67.7	14.77104874
bloomz-7b1	INT8-CW	32	73.4	68.2	14.6627566
qwen-7b-chat	INT8-CW	1024	286.9	68.8	14.53488372
bloomz-7b1	INT8-CW	1024	389.2	70.9	14.10437236
starcoder	INT4-MIXED	32	89.2	76.2	13.12335958
starcoder	INT4-MIXED	1024	701.8	78.5	12.7388535
lcm-sdxl	FP16	32	87.6	88	11.36363636
lcm-sdxl	FP16	1024	87.5	92.1	10.8577633

7-255H

Topology	Precision	Input Size	1st latency (ms)	2nd latency (ms)	2nd token per sec
distil-large-v2	INT8-CW	prompt0	226.1	9.1	109.8901099
distil-large-v2	INT8-CW	prompt1	283.9	9.9	101.010101
minicpm4-0.5b	INT4-MIXED	32	32.7	10.4	96.15384615
minicpm4-0.5b	INT4-MIXED	1024	63.7	10.6	94.33962264
minicpm4-0.5b	INT4-MIXED	32	42.6	11	90.90909091
minicpm4-0.5b	INT4-MIXED	32	37.6	11	90.90909091
whisper-large-v3-turbo	INT8-CW	prompt1	338.1	11	90.90909091
distil-large-v2	INT4-MIXED	prompt1	342.2	11.2	89.28571429
minicpm4-0.5b	INT8-CW	32	36.7	11.2	89.28571429
minicpm4-0.5b	INT4-MIXED	1024	72.8	11.3	88.49557522
distil-large-v2	INT4-MIXED	prompt0	254.6	11.4	87.71929825
minicpm4-0.5b	INT4-MIXED	1024	69.6	11.4	87.71929825
whisper-large-v3-turbo	INT4-MIXED	prompt1	364	11.5	86.95652174
minicpm4-0.5b	INT8-CW	1024	69.1	11.6	86.20689655
nuextract-tiny	INT8-CW	32	36.9	11.7	85.47008547
whisper-large-v3-turbo	INT8-CW	prompt0	266.8	11.7	85.47008547
nuextract-tiny	INT4-MIXED	32	46.8	11.9	84.03361345
nuextract-tiny	INT4-MIXED	1024	68.7	12.3	81.30081301
nuextract-tiny	INT8-CW	1024	67	12.3	81.30081301
whisper-large-v3-turbo	INT4-MIXED	prompt0	277.6	12.4	80.64516129
gemma-3-270m	INT4-MIXED	32	39.1	12.7	78.74015748
nanollava	INT4-MIXED	760	127.1	12.7	78.74015748
gemma-3-270m	INT4-MIXED	1024	51.4	12.8	78.125
qwen2.5-coder-0.5b-instruct	INT4-MIXED	32	40	12.9	77.51937984
lfm2.5-350m	INT4-MIXED	32	49.6	13	76.92307692
nanollava	INT4-MIXED	1752	170.8	13	76.92307692
qwen2.5-coder-0.5b-instruct	INT4-MIXED	1024	60.3	13.2	75.75757576
nanollava	INT8-CW	760	112.3	13.4	74.62686567
qwen2.5-coder-0.5b-instruct	INT4-MIXED	32	38.2	13.4	74.62686567
qwen2.5-coder-0.5b-instruct	INT4-MIXED	1024	69.7	13.5	74.07407407
nanollava	INT8-CW	1752	167.3	13.6	73.52941176
lfm2.5-350m	INT4-MIXED	1024	87.5	13.7	72.99270073
gemma-3-270m	INT8-CW	32	40	13.8	72.46376812
gemma-3-270m	INT8-CW	1024	53.8	13.9	71.94244604
qwen2.5-coder-0.5b-instruct	INT4-MIXED	1024	76.1	13.9	71.94244604
qwen2.5-coder-0.5b-instruct	INT4-MIXED	32	48.6	13.9	71.94244604
whisper-small	INT4-MIXED	prompt0	124.8	14.1	70.92198582
whisper-small	INT4-MIXED	prompt0	140.5	14.2	70.42253521
lfm2.5-350m	INT8-CW	32	47.2	15	66.66666667
whisper-small	INT4-MIXED	prompt1	207.4	15	66.66666667
whisper-small	INT8-CW	prompt1	193.1	15	66.66666667
whisper-small	INT4-MIXED	prompt1	200.1	15.2	65.78947368
whisper-small	INT4-MIXED	prompt1	188.6	15.3	65.35947712
whisper-small	INT8-CW	prompt0	136	15.3	65.35947712
lfm2.5-350m	INT8-CW	1024	91.4	15.8	63.29113924
gemma-3-270m	FP16	32	41.5	15.9	62.89308176
minicpm5-1b	INT4-MIXED	32	55.9	16.1	62.11180124
tiny-llama-1.1b-chat	INT4-MIXED	32	68.1	16.1	62.11180124
whisper-large-v3-turbo	FP16	prompt0	321.9	16.1	62.11180124
gemma-3-270m	FP16	1024	53.3	16.2	61.72839506
distil-large-v2	FP16	prompt0	294.1	16.5	60.60606061
distil-large-v2	FP16	prompt1	396.1	16.5	60.60606061
whisper-large-v3-turbo	FP16	prompt1	429.9	16.5	60.60606061
deepseek-r1-distill-qwen-1.5b	INT4-MIXED	32	62.5	16.7	59.88023952
tiny-llama-1.1b-chat	INT4-MIXED	1024	144	16.9	59.17159763
whisper-small	INT4-MIXED	prompt0	143.9	16.9	59.17159763
minicpm5-1b	INT4-MIXED	1024	115.6	17.1	58.47953216
deepseek-r1-distill-qwen-1.5b	INT4-MIXED	1024	127.8	17.2	58.13953488
llama-3.2-1b-instruct	INT4-MIXED	32	42	17.5	57.14285714
minicpm5-1b	INT8-CW	32	41.7	17.5	57.14285714
tiny-llama-1.1b-chat	INT4-MIXED	32	42.4	17.5	57.14285714
tiny-llama-1.1b-chat	INT4-MIXED	1024	116.2	18.1	55.24861878
qwen2.5-coder-0.5b-instruct	INT8-CW	32	54	18.2	54.94505495
qwen2.5-coder-0.5b-instruct	INT8-CW	1024	77.2	18.3	54.64480874
qwen3.5-0.8b	INT4-MIXED	85	104.3	18.3	54.64480874
llama-3.2-1b-instruct	INT4-MIXED	1024	130.2	18.4	54.34782609
whisper-small	FP16	prompt1	244.9	18.4	54.34782609
minicpm5-1b	INT8-CW	1024	108.5	18.5	54.05405405
qwen3.5-0.8b	INT4-MIXED	85	108.6	18.5	54.05405405
qwen3.5-0.8b	INT4-MIXED	1024	220.8	18.7	53.47593583
tiny-llama-1.1b-chat	INT8-CW	32	51.9	18.9	52.91005291
whisper-small	FP16	prompt0	141.3	18.9	52.91005291
qwen3.5-0.8b	INT4-MIXED	85	98	19	52.63157895
qwen3.5-0.8b	INT4-MIXED	1024	222.7	19.4	51.54639175
tiny-llama-1.1b-chat	INT8-CW	1024	117.6	19.6	51.02040816
qwen3.5-0.8b	INT4-MIXED	1024	237.3	19.7	50.76142132
qwen3.5-0.8b	INT8-CW	85	114.4	20	50
qwen3.5-0.8b	INT8-CW	1024	235.1	20.6	48.54368932
llama-3.2-1b-instruct	INT4-MIXED	32	42.4	21	47.61904762
exaone-4.0-1.2b	INT8-CW	32	55.5	21.1	47.39336493
qwen3-asr-0.6b	INT4-MIXED	prompt0	203.7	21.5	46.51162791
exaone-4.0-1.2b	INT4-MIXED	32	78.3	21.6	46.2962963
qwen2.5-coder-1.5b-instruct	INT4-MIXED	32	75.1	21.7	46.08294931
llama-3.2-1b-instruct	INT8-CW	32	48.5	21.8	45.87155963
llama-3.2-1b-instruct	INT4-MIXED	1024	113	21.9	45.66210046
gemma-3-1b-it	INT4-MIXED	1024	117.3	22	45.45454545
gemma-3-1b-it	INT4-MIXED	32	68.3	22	45.45454545
lfm2.5-1.2b-instruct	INT4-MIXED	32	65.3	22	45.45454545
exaone-4.0-1.2b	INT8-CW	1024	178.5	22.1	45.24886878
lfm2-1.2b	INT4-MIXED	32	62.8	22.3	44.84304933
qwen3-asr-0.6b	INT4-MIXED	prompt0	180.9	22.3	44.84304933
exaone-4.0-1.2b	INT4-MIXED	1024	191.8	22.6	44.24778761
lfm2.5-350m	FP16	32	69.1	22.6	44.24778761
llama-3.2-1b-instruct	INT8-CW	1024	124.1	22.6	44.24778761
qwen2.5-coder-1.5b-instruct	INT4-MIXED	1024	147.5	22.6	44.24778761
gemma-3-1b-it	INT4-MIXED	32	68.2	23.1	43.29004329
gemma-3-1b-it	INT4-MIXED	32	58.5	23.1	43.29004329
minicpm4-0.5b	FP16	32	50.5	23.1	43.29004329
lfm2.5-1.2b-instruct	INT4-MIXED	1024	169.3	23.2	43.10344828
lfm2-1.2b	INT4-MIXED	1024	169.4	23.4	42.73504274
qwen3-asr-0.6b	INT4-MIXED	prompt1	492.4	23.5	42.55319149
qwen3-asr-0.6b	INT4-MIXED	prompt1	485.8	23.6	42.37288136
gemma-3-1b-it	INT4-MIXED	1024	123	23.7	42.19409283
gemma-3-1b-it	INT4-MIXED	1024	140.6	23.9	41.84100418
lfm2.5-350m	FP16	1024	99.5	24	41.66666667
qwen2.5-coder-1.5b-instruct	INT4-MIXED	32	81.4	24.1	41.49377593
qwen3-asr-0.6b	INT4-MIXED	prompt0	196.6	24.2	41.32231405
exaone-4.0-1.2b	INT4-MIXED	32	88.9	24.3	41.15226337
minicpm4-0.5b	FP16	1024	89.8	24.3	41.15226337
lfm2-1.2b	INT4-MIXED	32	76.4	24.6	40.6504065
lfm2.5-1.2b-instruct	INT4-MIXED	32	77.8	24.6	40.6504065
qwen3-asr-0.6b	INT8-CW	prompt0	188.3	24.6	40.6504065
qwen2.5-coder-1.5b-instruct	INT4-MIXED	1024	182.9	24.7	40.48582996
exaone-4.0-1.2b	INT4-MIXED	1024	220.9	25	40
nuextract-tiny	FP16	32	81.3	25	40
qwen2.5-1.5b-instruct	INT4-MIXED	32	88.7	25	40
exaone-4.0-1.2b	INT4-MIXED	32	69.6	25.3	39.5256917
qwen3-asr-0.6b	INT8-CW	prompt1	467.6	25.3	39.5256917
qwen2.5-coder-0.5b-instruct	FP16	32	67.3	25.4	39.37007874
qwen3-asr-0.6b	INT4-MIXED	prompt1	511.8	25.4	39.37007874
qwen2.5-1.5b-instruct	INT4-MIXED	1024	184.3	25.5	39.21568627
lfm2-1.2b	INT4-MIXED	1024	196.5	25.6	39.0625
minicpm-1b-sft	INT4-MIXED	31	98.1	25.6	39.0625
nuextract-tiny	FP16	1024	85.3	25.6	39.0625
qwen2.5-1.5b-instruct	INT4-MIXED	32	67.5	25.7	38.91050584
qwen2.5-coder-0.5b-instruct	FP16	1024	90.5	25.7	38.91050584
lfm2.5-1.2b-instruct	INT4-MIXED	1024	200.7	25.8	38.75968992
exaone-4.0-1.2b	INT4-MIXED	1024	194.5	26.1	38.31417625
deepseek-r1-distill-qwen-1.5b	INT4-MIXED	32	87.7	26.2	38.16793893
minicpm-1b-sft	INT4-MIXED	31	92.3	26.4	37.87878788
qwen2.5-coder-1.5b-instruct	INT4-MIXED	32	69.1	26.4	37.87878788
qwen2.5-coder-1.5b-instruct	INT4-MIXED	1024	156.8	26.5	37.73584906
deepseek-r1-distill-qwen-1.5b	INT4-MIXED	1024	194.7	26.8	37.31343284
minicpm-1b-sft	INT4-MIXED	1014	169.5	26.8	37.31343284
nanollava	FP16	760	176.8	26.8	37.31343284
qwen2.5-1.5b-instruct	INT4-MIXED	1024	165	26.9	37.17472119
lfm2-1.2b	INT4-MIXED	32	55.2	27	37.03703704
lfm2.5-1.2b-instruct	INT4-MIXED	32	67.8	27.1	36.900369
deepseek-r1-distill-qwen-1.5b	INT8-CW	32	75.9	27.3	36.63003663
lfm2.5-1.2b-instruct	INT4-MIXED	1024	176.6	27.3	36.63003663
minicpm-1b-sft	INT4-MIXED	31	95.5	27.4	36.49635036
lfm2-1.2b	INT4-MIXED	1024	168.7	27.5	36.36363636
nanollava	FP16	1752	225.7	27.5	36.36363636
qwen2.5-1.5b-instruct	INT8-CW	32	70.5	27.8	35.97122302
qwen2.5-coder-1.5b-instruct	INT8-CW	32	65.8	27.8	35.97122302
gemma-2b-it	INT4-MIXED	32	67.4	27.9	35.84229391
lfm2-1.2b	INT4-MIXED	32	86	28	35.71428571
deepseek-r1-distill-qwen-1.5b	INT8-CW	1024	164.3	28.1	35.58718861
lfm2.5-1.2b-instruct	INT4-MIXED	32	60.7	28.1	35.58718861
qwen2.5-1.5b-instruct	INT8-CW	1024	164	28.2	35.46099291
qwen2.5-coder-1.5b-instruct	INT8-CW	1024	164.4	28.3	35.33568905
gemma-3-1b-it	INT8-CW	32	88.7	28.4	35.21126761
gemma-2b-it	INT4-MIXED	1024	214.2	28.6	34.96503497
lfm2-1.2b	INT8-CW	32	77	28.7	34.84320557
gemma-3-1b-it	INT8-CW	1024	143.9	29	34.48275862
lfm2.5-1.2b-instruct	INT4-MIXED	1024	186	29	34.48275862
minicpm-1b-sft	INT8-CW	31	95	29.2	34.24657534
lfm2-1.2b	INT4-MIXED	1024	183.6	29.3	34.12969283
gemma-2b-it	INT4-MIXED	1024	269.1	29.5	33.89830508
lfm2.5-1.2b-instruct	INT8-CW	32	69.6	29.5	33.89830508
minicpm-1b-sft	INT4-MIXED	1014	201.6	29.5	33.89830508
minicpm-1b-sft	INT4-MIXED	1014	174.9	29.6	33.78378378
gemma-2b-it	INT4-MIXED	32	73.6	29.7	33.67003367
lfm2.5-1.2b-instruct	INT8-CW	1024	199.4	30.1	33.22259136
minicpm-1b-sft	INT8-CW	1014	182.7	30.4	32.89473684
lfm2-1.2b	INT8-CW	1024	198.3	30.7	32.5732899
lfm2-8b-a1b	INT4-MIXED	32	166.7	32.2	31.05590062
lfm2-8b-a1b	INT4-MIXED	1024	1150.3	33.4	29.94011976
gemma-2-2b	INT4-MIXED	33	82.8	34.8	28.73563218
phi-2	INT4-MIXED	32	140.5	34.9	28.65329513
smollm3-3b	INT4-MIXED	32	107	35	28.57142857
gemma-4-e2b-it	INT4-MIXED	274	427.5	35.2	28.40909091
gemma-2-2b	INT4-MIXED	1025	262.4	35.4	28.24858757
gemma-4-e2b-it	INT4-MIXED	1024	592	36.1	27.70083102
smollm3-3b	INT4-MIXED	1024	289.3	36.1	27.70083102
llama-3.2-3b-instruct	INT4-MIXED	32	71.6	36.3	27.54820937
qwen2.5-coder-3b-instruct	INT4-MIXED	32	106.3	36.4	27.47252747
gemma-2-2b	INT4-MIXED	33	90.5	36.6	27.32240437
minicpm5-1b	FP16	32	77.3	36.7	27.2479564
stable-zephyr-3b-dpo	INT4-MIXED	32	130.1	36.7	27.2479564
stablelm-3b-4e1t	INT4-MIXED	32	115.6	36.8	27.17391304
phi-2	INT4-MIXED	1024	328.1	37	27.02702703
afm-4.5b	INT4-MIXED	32	83.2	37.1	26.9541779
llama-3.2-3b-instruct	INT4-MIXED	1024	275.6	37.2	26.88172043
smollm3-3b	INT4-MIXED	32	123.4	37.2	26.88172043
gemma-2-2b	INT4-MIXED	1025	313.4	37.4	26.73796791
qwen2.5-coder-3b-instruct	INT4-MIXED	1024	272.4	37.6	26.59574468
tiny-llama-1.1b-chat	FP16	32	54.7	37.7	26.52519894
afm-4.5b	INT4-MIXED	1024	431.8	38	26.31578947
minicpm5-1b	FP16	1024	151.1	38	26.31578947
phi-2	INT4-MIXED	32	129.4	38.1	26.24671916
llama-3.2-3b-instruct	INT4-MIXED	32	114.7	38.2	26.17801047
qwen3.5-0.8b	FP16	85	149.2	38.8	25.77319588
smollm3-3b	INT4-MIXED	1024	377.5	38.8	25.77319588
stable-zephyr-3b-dpo	INT4-MIXED	1024	339.8	39	25.64102564
stablelm-3b-4e1t	INT4-MIXED	1024	333.1	39	25.64102564
qwen2.5-coder-3b-instruct	INT4-MIXED	32	132.5	39.1	25.57544757
gemma-4-e2b-it	INT4-MIXED	274	454.1	39.3	25.44529262
tiny-llama-1.1b-chat	FP16	1024	177.2	39.4	25.38071066
phi-3-mini-4k-instruct	INT4-MIXED	32	111.7	39.6	25.25252525
granite-4.0-h-micro	INT4-MIXED	33	182.3	39.7	25.18891688
phi-3-mini-128k-instruct	INT4-MIXED	32	119.6	39.7	25.18891688
phi-3.5-mini-instruct	INT4-MIXED	32	129.8	39.9	25.06265664
qwen3.5-0.8b	FP16	1024	296	39.9	25.06265664
stable-zephyr-3b-dpo	INT4-MIXED	32	145.6	39.9	25.06265664
llama-3.2-3b-instruct	INT4-MIXED	1024	381.7	40	25
gemma-4-e2b-it	INT4-MIXED	1024	661	40.1	24.93765586
qwen2.5-coder-3b-instruct	INT4-MIXED	1024	414.1	40.2	24.87562189
granite-4.0-h-micro	INT4-MIXED	1033	767.8	40.5	24.69135802
gemma-2b-it	INT8-CW	32	59.6	40.6	24.63054187
phi-2	INT4-MIXED	1024	422.1	41	24.3902439
qwen3-asr-0.6b	FP16	prompt1	592.5	41.3	24.21307506
gemma-2b-it	INT8-CW	1024	237.1	41.4	24.15458937
lfm2-8b-a1b	INT8-CW	32	191	41.4	24.15458937
llama-3.2-3b-instruct	INT4-MIXED	32	92	41.8	23.92344498
phi-3-mini-128k-instruct	INT4-MIXED	1024	444.9	41.8	23.92344498
phi-3-mini-4k-instruct	INT4-MIXED	1024	442.2	41.8	23.92344498
phi-3.5-mini-instruct	INT4-MIXED	1024	441.7	41.9	23.86634845
gemma-3-1b-it	FP16	32	75.9	42	23.80952381
phi-3-mini-128k-instruct	INT4-MIXED	32	119.6	42	23.80952381
stable-zephyr-3b-dpo	INT4-MIXED	1024	407.4	42.2	23.69668246
granite-4.0-h-micro	INT4-MIXED	33	185.7	42.4	23.58490566
gemma-3-1b-it	FP16	1024	174.8	42.5	23.52941176
gemma-4-e2b-it	INT8-CW	274	430.3	42.8	23.36448598
lfm2-8b-a1b	INT8-CW	1024	1083.3	42.8	23.36448598
llama-3.2-3b-instruct	INT4-MIXED	1024	294.5	42.8	23.36448598
granite-4.0-h-micro	INT4-MIXED	1033	867.7	42.9	23.31002331
qwen3-asr-0.6b	FP16	prompt0	247	42.9	23.31002331
phi-3.5-vision-instruct	INT4-MIXED	802	320.9	43.3	23.09468822
gemma-4-e2b-it	INT8-CW	1024	607.3	43.6	22.93577982
gemma-2-2b	INT8-CW	33	77.5	44	22.72727273
phi-3-mini-128k-instruct	INT4-MIXED	1024	596.2	44	22.72727273
phi-3.5-vision-instruct	INT4-MIXED	1032	614.8	44	22.72727273
phi-2	INT8-CW	32	126.6	45.1	22.172949
phi-3-mini-4k-instruct	INT4-MIXED	32	111.1	45.2	22.12389381
stablelm-3b-4e1t	INT4-MIXED	32	116.3	45.4	22.02643172
gemma-2-2b	INT8-CW	1025	277.4	45.6	21.92982456
llama-3.2-1b-instruct	FP16	32	66.9	45.6	21.92982456
stablelm-3b-4e1t	INT8-CW	32	139	45.8	21.83406114
phi-3.5-mini-instruct	INT4-MIXED	32	118.7	46	21.73913043
qwen3-4b	INT4-MIXED	32	135.9	46.1	21.69197397
stable-zephyr-3b-dpo	INT8-CW	32	116.8	46.2	21.64502165
phi-4-mini-reasoning	INT4-MIXED	32	148.8	46.4	21.55172414
phi-4-mini-instruct	INT4-MIXED	32	153.3	46.6	21.45922747
llama-3.2-1b-instruct	FP16	1024	195.6	46.7	21.41327623
phi-3-mini-4k-instruct	INT4-MIXED	1024	585.9	47.8	20.92050209
phi-4-mini-instruct	INT4-MIXED	1024	388.5	48	20.83333333
phi-4-mini-reasoning	INT4-MIXED	1024	388.1	48	20.83333333
qwen3-4b	INT4-MIXED	1024	413.3	48.2	20.74688797
phi-2	INT8-CW	1024	352	48.4	20.66115702
stablelm-3b-4e1t	INT4-MIXED	1024	359.3	48.4	20.66115702
phi-4-mini-reasoning	INT4-MIXED	32	159.3	48.5	20.6185567
phi-3.5-mini-instruct	INT4-MIXED	1024	610.6	48.7	20.5338809
gemma-3-4b-it	INT4-MIXED	32	152.4	48.8	20.49180328
stablelm-3b-4e1t	INT8-CW	1024	364.4	49.2	20.32520325
stable-zephyr-3b-dpo	INT8-CW	1024	363	49.3	20.28397566
gemma-3-4b-it	INT4-MIXED	1024	421.1	49.8	20.08032129
exaone-4.0-1.2b	FP16	32	107	50	20
llama-3.2-3b-instruct	INT8-CW	32	89	50	20
phi-4-mini-reasoning	INT4-MIXED	1024	495.2	50.3	19.88071571
qwen2.5-coder-3b-instruct	INT8-CW	32	107.1	50.4	19.84126984
smollm3-3b	INT8-CW	32	95.3	50.5	19.8019802
phi-3-mini-4k-instruct	INT4-MIXED	32	115.6	50.7	19.72386588
phi-3.5-mini-instruct	INT4-MIXED	32	114.9	51	19.60784314
afm-4.5b	INT8-CW	32	70.8	51.1	19.56947162
exaone-4.0-1.2b	FP16	1024	266.7	51.3	19.49317739
gemma-3-4b-it	INT4-MIXED	32	150.3	51.7	19.34235977
llama-3.2-3b-instruct	INT8-CW	1024	316.9	51.7	19.34235977
qwen3-vl-4b-instruct	INT4-MIXED	4937	1278.7	51.8	19.30501931
qwen3-vl-4b-instruct	INT4-MIXED	4907	1248.4	51.9	19.26782274
lfm2-1.2b	FP16	32	92.5	52	19.23076923
lfm2.5-1.2b-instruct	FP16	32	92.7	52.1	19.19385797
lfm2.5-1.2b-instruct	FP16	1024	276.7	52.2	19.15708812
qwen2.5-coder-3b-instruct	INT8-CW	1024	389.8	52.2	19.15708812
lfm2-1.2b	FP16	1024	267.9	52.3	19.12045889
gemma-3-4b-it	INT4-MIXED	1024	530.5	52.4	19.08396947
phi-4-mini-instruct	INT4-MIXED	32	160.6	52.4	19.08396947
smollm3-3b	INT8-CW	1024	318.8	52.6	19.01140684
afm-4.5b	INT8-CW	1024	325.2	52.7	18.97533207
phi-3-mini-4k-instruct	INT4-MIXED	1024	445.6	53.2	18.79699248
phi-3.5-mini-instruct	INT4-MIXED	1024	451.7	53.4	18.72659176
phi-4-mini-instruct	INT4-MIXED	1024	539	53.5	18.69158879
qwen2.5-coder-3b-instruct	INT4-MIXED	32	58.1	53.5	18.69158879
smollm3-3b	INT4-MIXED	32	66.8	53.8	18.58736059
qwen2.5-coder-3b-instruct	INT4-MIXED	1024	288.9	54.3	18.41620626
smollm3-3b	INT4-MIXED	1024	309.9	54.6	18.31501832
qwen3-vl-4b-instruct	INT4-MIXED	4937	1283.2	54.8	18.24817518
minicpm-1b-sft	FP16	31	131.7	55	18.18181818
deepseek-r1-distill-qwen-1.5b	FP16	32	101.1	55.3	18.08318264
phi-4-mini-instruct	INT4-MIXED	32	145.5	55.3	18.08318264
qwen3-vl-4b-instruct	INT4-MIXED	4907	1248.6	55.3	18.08318264
granite-4.0-h-micro	INT8-CW	33	185.8	55.7	17.95332136
deepseek-r1-distill-qwen-1.5b	FP16	1024	239.1	55.8	17.92114695
phi-4-mini-reasoning	INT4-MIXED	32	133.1	56.1	17.82531194
granite-4.0-h-micro	INT8-CW	1033	792.2	56.5	17.69911504
minicpm-1b-sft	FP16	1014	254.4	57	17.54385965
qwen2.5-3b-instruct-gptq	INT4-MIXED	32	77.8	57.2	17.48251748
phi-4-mini-instruct	INT4-MIXED	1024	404.5	57.4	17.42160279
phi-4-mini-reasoning	INT4-MIXED	1024	407	57.4	17.42160279
granite-4.0-h-micro	INT4-MIXED	33	110.9	57.5	17.39130435
qwen2.5-coder-1.5b-instruct	FP16	32	91.1	57.5	17.39130435
whisper-large-v3	INT4-MIXED	prompt1	549.4	57.5	17.39130435
qwen2.5-1.5b-instruct	FP16	32	93.5	57.6	17.36111111
granite-4.0-h-micro	INT4-MIXED	1033	743.8	57.8	17.30103806
whisper-large-v3	INT8-CW	prompt1	539.4	57.9	17.27115717
whisper-large-v3	INT4-MIXED	prompt0	489.8	58	17.24137931
gemma-3n-e2b-it	INT4-MIXED	275	710.9	58.1	17.21170396
codellama-7b	INT4-MIXED	32	114.7	58.2	17.18213058
gemma-3n-e2b-it	INT4-MIXED	1025	1074.1	58.2	17.18213058
phi-3-mini-4k-instruct	INT8-CW	32	98.5	58.3	17.15265866
qwen2.5-1.5b-instruct	FP16	1024	241.1	58.3	17.15265866
llama-2-7b-chat-hf	INT4-MIXED	32	103.3	58.5	17.09401709
qwen2.5-3b-instruct-gptq	INT4-MIXED	1024	307.9	58.5	17.09401709
qwen2.5-coder-1.5b-instruct	FP16	1024	235.2	58.5	17.09401709
phi-3-mini-128k-instruct	INT8-CW	32	97.8	58.6	17.06484642
phi-3.5-mini-instruct	INT8-CW	32	99.9	58.7	17.03577513
whisper-large-v3	INT8-CW	prompt0	468.6	58.7	17.03577513
qwen3-4b	INT4-MIXED	32	116.1	60.2	16.61129568
gemma-3-4b-it	INT4-MIXED	32	128.7	61.3	16.3132137
biomistral-7b-slerp	INT4-MIXED	7	92.4	61.4	16.28664495
codellama-7b	INT4-MIXED	32	105.4	61.4	16.28664495
codellama-7b	INT4-MIXED	1024	617	61.5	16.2601626
falcon-7b-instruct	INT4-MIXED	32	125	61.5	16.2601626
gemma-3-4b-it	INT4-MIXED	1024	443.4	61.5	16.2601626
zephyr-7b-beta	INT4-MIXED	32	112.9	61.6	16.23376623
phi-4-multimodal-instruct	INT4-MIXED	1362	1277.5	61.7	16.20745543
neural-chat-7b-v3-3	INT4-MIXED	32	102.2	61.8	16.18122977
phi-4-multimodal-instruct	INT4-MIXED	578	529.9	61.8	16.18122977
phi-3.5-vision-instruct	INT8-CW	802	322.4	61.9	16.15508885
lfm2-24b-a2b	INT4-MIXED	32	459.8	62.1	16.10305958
phi-4-multimodal-instruct	INT4-MIXED	786	492.5	62.1	16.10305958
phi-3-mini-4k-instruct	INT8-CW	1024	490.8	62.2	16.07717042
phi-4-multimodal-instruct	INT4-MIXED	1570	860.1	62.2	16.07717042
llama-2-7b-chat-hf	INT4-MIXED	1024	627	62.3	16.05136437
mistral-7b-instruct-v0.3	INT4-MIXED	32	101.4	62.4	16.02564103
phi-3.5-mini-instruct	INT8-CW	1024	495	62.4	16.02564103
phi-4-mini-instruct	INT8-CW	32	128.1	62.4	16.02564103
mistral-7b-instruct-v0.2	INT4-MIXED	32	106.3	62.5	16
phi-4-mini-reasoning	INT8-CW	32	127.1	62.5	16
phi-3-mini-128k-instruct	INT8-CW	1024	493.8	62.6	15.97444089
phi-3.5-vision-instruct	INT8-CW	1032	555.1	62.8	15.92356688
qwen3-4b	INT4-MIXED	1024	434	63	15.87301587
neural-chat-7b-v3-3	INT4-MIXED	1024	648.2	63.3	15.79778831
zephyr-7b-beta	INT4-MIXED	1024	661.2	63.3	15.79778831
minicpm3-4b	INT4-MIXED	32	263.1	63.6	15.72327044
falcon-7b-instruct	INT4-MIXED	1024	658.2	63.8	15.67398119
gemma-3-4b-it	INT8-CW	32	136	64.1	15.60062402
mistral-7b-instruct-v0.2	INT4-MIXED	1024	651.3	64.1	15.60062402
phi-4-mini-reasoning	INT8-CW	1024	446	64.1	15.60062402
qwen3-4b	INT8-CW	32	112.2	64.1	15.60062402
mistral-7b-instruct-v0.3	INT4-MIXED	1024	660.6	64.2	15.57632399
phi-4-mini-instruct	INT8-CW	1024	439.3	64.3	15.55209953
qwen2.5-7b-instruct	INT4-MIXED	32	89.9	64.4	15.52795031
qwen2.5-7b-instruct-1m	INT4-MIXED	32	107.3	64.4	15.52795031
gemma-3n-e2b-it	INT4-MIXED	275	785.8	64.5	15.50387597
biomistral-7b-slerp	INT4-MIXED	7	83.8	64.6	15.47987616
minicpm-o-2_6	INT4-MIXED	238	653.1	64.7	15.45595054
codellama-7b	INT4-MIXED	1024	882.3	64.8	15.43209877
minicpm-v-2_6	INT4-MIXED	228	657.9	64.9	15.40832049
neural-chat-7b-v3-3	INT4-MIXED	32	114	65	15.38461538
mistral-7b-instruct-v0.2	INT4-MIXED	32	111.3	65.1	15.3609831
mistral-7b-instruct-v0.3	INT4-MIXED	32	115.5	65.1	15.3609831
mistral-7b-instruct-v0.1	INT4-MIXED	32	128.9	65.2	15.33742331
qwen2.5-vl-7b-instruct	INT4-MIXED	32	141.2	65.4	15.29051988
gemma-3-4b-it	INT8-CW	1024	460.9	65.8	15.19756839
qwen2.5-7b-instruct-1m	INT4-MIXED	1024	607.8	65.9	15.17450683
bloomz-7b1	INT4-MIXED	32	104.5	66	15.15151515
qwen2.5-7b-instruct	INT4-MIXED	1024	606.3	66.1	15.12859304
gemma-3n-e2b-it	INT8-CW	275	731.1	66.2	15.10574018
qwen3-vl-4b-instruct	INT4-MIXED	4907	1264.4	66.2	15.10574018
qwen-7b-chat	INT4-MIXED	32	127.6	66.4	15.06024096
neural-chat-7b-v3-3	INT4-MIXED	1024	911.7	66.5	15.03759398
qwen3-vl-4b-instruct	INT4-MIXED	4937	1289.5	66.6	15.01501502
whisper-large-v3	FP16	prompt1	600	66.6	15.01501502
deepseek-r1-distill-llama-8b	INT4-MIXED	35	118.3	66.9	14.94768311
llama-3-8b-instruct	INT4-MIXED	32	112.4	66.9	14.94768311
mistral-7b-instruct-v0.3	INT4-MIXED	1024	941.1	67	14.92537313
llama-3.1-8b-instruct	INT4-MIXED	32	106.4	67.3	14.85884101
mistral-7b-instruct-v0.1	INT4-MIXED	1025	921.1	67.3	14.85884101
qwen3-4b	INT8-CW	1024	454.4	67.3	14.85884101
gemma-3n-e2b-it	INT4-MIXED	1025	1154.7	67.4	14.83679525
mistral-7b-instruct-v0.2	INT4-MIXED	1024	941.8	67.4	14.83679525
qwen2.5-vl-7b-instruct	INT4-MIXED	1024	670.2	67.5	14.81481481
gpt-oss-20b	INT4-MIXED	32	252.6	67.8	14.74926254
gemma-3n-e2b-it	INT8-CW	1025	1128.6	68.2	14.6627566
qwen2.5-7b-instruct	INT4-MIXED	32	101.3	68.5	14.59854015
gpt-oss-20b	INT4-MIXED	1024	3036.6	68.6	14.57725948
deepseek-r1-distill-qwen-7b	INT4-MIXED	32	110.7	68.7	14.55604076
minicpm3-4b	INT4-MIXED	1024	711.4	68.9	14.5137881
qwen2-7b-instruct	INT4-MIXED	32	113.2	69.1	14.47178003
qwen2.5-7b-instruct-1m	INT4-MIXED	32	111.1	69.2	14.45086705
deepseek-r1-distill-llama-8b	INT4-MIXED	1135	744.2	69.3	14.43001443
llama-3-8b-instruct	INT4-MIXED	1024	649.6	69.3	14.43001443
llama-3.1-8b-instruct	INT4-MIXED	1024	657.9	69.3	14.43001443
minicpm-o-2_6	INT4-MIXED	238	662	69.3	14.43001443
bloomz-7b1	INT4-MIXED	1024	837.1	69.4	14.4092219
minicpm-v-2_6	INT4-MIXED	228	636.9	69.5	14.38848921
minicpm3-4b	INT4-MIXED	32	286.6	69.5	14.38848921
minicpm4-8b	INT4-MIXED	32	127.4	69.5	14.38848921
qwen3-8b	INT4-MIXED	32	137.6	69.5	14.38848921
bloomz-7b1	INT4-MIXED	32	117.7	69.7	14.3472023
qwen-7b-chat	INT4-MIXED	1024	631.3	69.9	14.30615165
deepseek-r1-distill-qwen-7b	INT4-MIXED	1024	833.2	70	14.28571429
qwen2.5-7b-instruct	INT4-MIXED	1024	853.3	70	14.28571429
whisper-large-v3	FP16	prompt0	538.2	70.1	14.26533524
fara-7b	INT4-MIXED	32	235.7	70.2	14.24501425
qwen2.5-vl-7b-instruct	INT4-MIXED	32	141.4	70.3	14.22475107
stable-diffusion-xl-1.0-inpainting-0.1	INT8-CW	32	71.8	70.3	14.22475107
qwen2-7b-instruct	INT4-MIXED	1024	851.4	70.4	14.20454545
minicpm-v-4_5	INT4-MIXED	217	682.2	70.5	14.18439716
qwen2.5-7b-instruct-1m	INT4-MIXED	1024	854.3	70.5	14.18439716
minicpm4-8b	INT4-MIXED	1024	739	71.3	14.02524544
qwen3-8b	INT4-MIXED	1024	689.6	71.3	14.02524544
llama-3-8b-instruct	INT4-MIXED	32	121.4	71.4	14.00560224
qwen2.5-vl-7b-instruct	INT4-MIXED	1024	891.8	71.5	13.98601399
falcon-7b-instruct	INT4-MIXED	32	129.6	71.6	13.96648045
fara-7b	INT4-MIXED	1024	1080.6	71.7	13.94700139
llava-next-video-7b-hf	INT4-MIXED	2945	593.9	72.1	13.86962552
minicpm3-4b	INT4-MIXED	32	231.6	72.1	13.86962552
minicpm4-8b	INT4-MIXED	32	112.2	72.7	13.75515818
bloomz-7b1	INT4-MIXED	1024	1079.7	72.8	13.73626374
llama-3-8b-instruct	INT4-MIXED	1024	915.9	72.8	13.73626374
stable-diffusion-xl-1.0-inpainting-0.1	INT8-CW	32	76.4	72.8	13.73626374
qwen3-vl-4b-instruct	INT8-CW	4907	1277.1	72.9	13.71742112
qwen3-vl-4b-instruct	INT8-CW	4937	1331.3	73	13.69863014
falcon-7b-instruct	INT4-MIXED	1024	919.5	73.3	13.6425648
qwen3-8b	INT4-MIXED	32	149.4	74.2	13.47708895
minicpm4-8b	INT4-MIXED	1024	1011.6	74.5	13.42281879
minicpm-v-4_5	INT4-MIXED	217	707.5	74.6	13.40482574
minicpm3-4b	INT8-CW	32	257.7	74.7	13.38688086
baichuan2-7b-chat	INT4-MIXED	32	135.1	74.8	13.36898396
minicpm3-4b	INT4-MIXED	1024	850.3	75.6	13.22751323
qwen3_8b_draft_eagle3	INT4-MIXED	32	110.6	75.9	13.17523057
qwen3-8b	INT4-MIXED	1024	959	76.1	13.14060447
deepseek-r1-distill-qwen-7b	INT4-MIXED	32	108.8	76.6	13.05483029
phi-4-multimodal-instruct	INT4-MIXED	786	526.6	77.8	12.85347044
minicpm3-4b	INT4-MIXED	1024	748.1	78	12.82051282
deepseek-r1-distill-qwen-7b	INT4-MIXED	1024	614.1	78.2	12.78772379
phi-4-multimodal-instruct	INT4-MIXED	578	685.6	78.2	12.78772379
baichuan2-7b-chat	INT4-MIXED	1024	878.4	78.6	12.72264631
videochat-flash-qwen2_5-7b_internvideo2-1b	INT4-MIXED	14531	60873.7	78.6	12.72264631
deepseek-r1-distill-llama-8b	INT4-MIXED	35	126.1	78.7	12.7064803
qwen3_8b_draft_eagle3	INT4-MIXED	1024	909.1	79.3	12.61034048
phi-4-multimodal-instruct	INT4-MIXED	1570	909	79.4	12.59445844
qwen3.5-9b	INT4-MIXED	83	325.3	79.4	12.59445844
gemma-7b-it	INT4-MIXED	32	112.9	79.5	12.57861635
phi-4-multimodal-instruct	INT4-MIXED	1362	1610.4	79.5	12.57861635
qwen3.5-9b	INT4-MIXED	1024	1120.3	79.5	12.57861635
phi-4-multimodal-instruct	INT8-CW	578	576.9	80.1	12.48439451
phi-4-multimodal-instruct	INT8-CW	786	510.6	80.1	12.48439451
gpt-oss-20b	INT4-MIXED	32	283.4	80.2	12.46882793
deepseek-r1-distill-llama-8b	INT4-MIXED	1135	769.1	80.3	12.45330012
phi-4-multimodal-instruct	INT8-CW	1362	1370.9	81.4	12.28501229
phi-4-multimodal-instruct	INT8-CW	1570	857.6	81.5	12.26993865
gemma-7b-it	INT4-MIXED	1024	769.2	81.6	12.25490196
llama-2-7b-chat-hf	INT4-MIXED	32	97.3	82.1	12.18026797
qwen3.5-9b	INT4-MIXED	83	356.9	82.3	12.15066829
minicpm3-4b	INT8-CW	1024	786.4	82.5	12.12121212
deepseek-r1-distill-llama-8b	INT4-MIXED	35	128.5	82.8	12.07729469
gemma-3n-e4b-it	INT4-MIXED	275	868.6	82.8	12.07729469
gemma-2b-it	FP16	32	91.7	83.1	12.03369434
gemma-3n-e4b-it	INT4-MIXED	1025	1410	83.1	12.03369434
qwen3.5-9b	INT4-MIXED	1024	1369.3	83.2	12.01923077
gemma-2b-it	FP16	1024	391.7	83.4	11.99040767
gpt-oss-20b	INT4-MIXED	1024	3255.1	83.5	11.9760479
llama-3.1-8b-instruct	INT4-MIXED	32	134.3	83.5	11.9760479
gemma-7b-it	INT4-MIXED	32	116.8	83.7	11.9474313
qwen2-7b-instruct	INT4-MIXED	32	105.2	83.7	11.9474313
qwen2-7b-instruct-gptq	INT4-MIXED	32	99.3	83.8	11.93317422
deepseek-r1-distill-llama-8b	INT4-MIXED	1135	888.3	84	11.9047619
qwen2.5-7b-instruct	INT4-MIXED	32	103.1	84.1	11.89060642
minicpm-v-2_6	INT4-MIXED	228	603.4	84.5	11.83431953
mistral-7b-instruct-v0.2	INT4-MIXED	32	98.3	84.5	11.83431953
mistral-7b-instruct-v0.3	INT4-MIXED	32	107.1	84.6	11.82033097
llama-2-7b-chat-hf	INT4-MIXED	1024	628.9	84.8	11.79245283
llama-3.1-8b-instruct	INT4-MIXED	1024	889.2	84.9	11.77856302
qwen-7b-chat	INT4-MIXED	32	122.3	84.9	11.77856302
qwen2-7b-instruct	INT4-MIXED	1024	603.2	85.1	11.75088132
qwen2.5-vl-7b-instruct	INT4-MIXED	32	143.9	85.2	11.7370892
gemma-3n-e4b-it	INT4-MIXED	275	858.8	85.4	11.70960187
qwen2-7b-instruct-gptq	INT4-MIXED	1024	612.6	85.4	11.70960187
zephyr-7b-beta	INT4-MIXED	32	114.2	85.4	11.70960187
qwen2.5-7b-instruct	INT4-MIXED	1024	626.2	85.7	11.66861144
gemma-7b-it	INT4-MIXED	1024	1085.2	86.4	11.57407407
mistral-7b-instruct-v0.2	INT4-MIXED	1024	649.7	86.5	11.56069364
gemma-3n-e4b-it	INT4-MIXED	1025	1512.7	86.6	11.54734411
mistral-7b-instruct-v0.3	INT4-MIXED	1024	660.8	86.7	11.53402537
qwen2.5-vl-7b-instruct	INT4-MIXED	1024	672.9	87.1	11.48105626
zephyr-7b-beta	INT4-MIXED	1024	662.2	87.4	11.4416476
gemma-2-2b	FP16	33	99.1	88.1	11.3507378
qwen-7b-chat	INT4-MIXED	1024	664.8	88.1	11.3507378
gemma-2-9b-it	INT4-MIXED	32	180.5	88.2	11.33786848
gemma-2-2b	FP16	1025	429.8	89.7	11.14827202
llama-3-8b-instruct	INT4-MIXED	32	113.5	90.2	11.0864745
llama-3-8b-instruct	INT4-MIXED	32	112.4	90.4	11.0619469
llama-3.1-8b-instruct	INT4-MIXED	32	103.7	90.5	11.04972376
phi-2	FP16	32	127.5	91.1	10.97694841
gemma-2-9b-it	INT4-MIXED	1024	889.5	91.2	10.96491228
llama-3-8b-instruct	INT4-MIXED	1024	674.8	92	10.86956522
llama-3-8b-instruct	INT4-MIXED	1024	659.4	92.4	10.82251082
stable-zephyr-3b-dpo	FP16	32	134.3	92.6	10.79913607
stablelm-3b-4e1t	FP16	32	154.2	92.6	10.79913607
llama-3.1-8b-instruct	INT4-MIXED	1024	650.9	92.7	10.78748652
qwen3-8b	INT4-MIXED	32	131.7	93	10.75268817
qwen-7b-chat	INT4-MIXED	32	129.9	93.8	10.66098081
qwen3-8b	INT4-MIXED	1024	717.3	95.1	10.51524711
phi-2	FP16	1024	553.5	95.7	10.44932079
qwen-7b-chat	INT4-MIXED	1024	659.8	97.1	10.29866117
stable-zephyr-3b-dpo	FP16	1024	563.7	97.4	10.26694045
gemma-3n-e2b-it	FP16	275	696.9	97.6	10.24590164
stablelm-3b-4e1t	FP16	1024	562.5	97.8	10.22494888
gemma-3n-e2b-it	FP16	1025	1243.1	98.4	10.16260163
gemma-3n-e4b-it	INT8-CW	275	854.9	98.5	10.15228426
llama-2-7b-chat-hf	INT8-CW	32	130.4	98.7	10.13171226
gemma-3n-e4b-it	INT8-CW	1025	1456	99.5	10.05025126

7-258V

Topology	Precision	Input Size	1st latency (ms)	2nd latency (ms)	2nd token per sec
lfm2.5-350m	INT4-MIXED	1024	39.9	3.6	277.7777778
lfm2.5-350m	INT4-MIXED	32	12.9	3.6	277.7777778
minicpm4-0.5b	INT4-MIXED	32	17.9	4.7	212.7659574
gemma-3-270m	INT4-MIXED	1024	26.5	4.8	208.3333333
gemma-3-270m	INT4-MIXED	32	18.6	4.8	208.3333333
lfm2.5-350m	INT8-CW	32	13.8	4.9	204.0816327
lfm2.5-350m	INT8-CW	1024	37.7	5.1	196.0784314
minicpm4-0.5b	INT4-MIXED	32	23	5.1	196.0784314
minicpm4-0.5b	INT4-MIXED	32	19.7	5.1	196.0784314
gemma-3-270m	INT8-CW	32	20	5.2	192.3076923
gemma-3-270m	INT8-CW	1024	25.7	5.3	188.6792453
qwen2.5-coder-0.5b-instruct	INT4-MIXED	32	18.4	5.4	185.1851852
qwen2.5-coder-0.5b-instruct	INT4-MIXED	32	18.7	5.4	185.1851852
minicpm4-0.5b	INT4-MIXED	1024	39.7	5.5	181.8181818
distil-large-v2	INT4-MIXED	prompt0	158.6	5.6	178.5714286
whisper-large-v3-turbo	INT4-MIXED	prompt0	166.9	5.7	175.4385965
distil-large-v2	INT4-MIXED	prompt1	213.7	5.8	172.4137931
distil-large-v2	INT8-CW	prompt0	155	5.8	172.4137931
distil-large-v2	INT8-CW	prompt1	206.9	5.8	172.4137931
whisper-large-v3-turbo	INT4-MIXED	prompt1	224.1	5.8	172.4137931
minicpm4-0.5b	INT4-MIXED	1024	41.9	5.9	169.4915254
qwen2.5-coder-0.5b-instruct	INT4-MIXED	32	23.1	5.9	169.4915254
whisper-large-v3-turbo	INT8-CW	prompt1	223.5	5.9	169.4915254
minicpm4-0.5b	INT4-MIXED	1024	48.8	6	166.6666667
qwen2.5-coder-0.5b-instruct	INT4-MIXED	1024	42.1	6.1	163.9344262
whisper-large-v3-turbo	INT8-CW	prompt0	161.8	6.1	163.9344262
qwen2.5-coder-0.5b-instruct	INT4-MIXED	1024	43.4	6.2	161.2903226
nuextract-tiny	INT4-MIXED	32	19.7	6.3	158.7301587
whisper-small	INT4-MIXED	prompt0	78.8	6.4	156.25
qwen2.5-coder-0.5b-instruct	INT4-MIXED	1024	51	6.6	151.5151515
whisper-small	INT4-MIXED	prompt0	77.9	6.6	151.5151515
whisper-small	INT4-MIXED	prompt1	136.7	6.7	149.2537313
nuextract-tiny	INT4-MIXED	1024	48.6	6.8	147.0588235
whisper-small	INT4-MIXED	prompt1	134.2	6.8	147.0588235
whisper-small	INT8-CW	prompt0	81.2	6.9	144.9275362
nanollava	INT4-MIXED	760	84.3	7	142.8571429
whisper-small	INT8-CW	prompt1	138.1	7	142.8571429
minicpm4-0.5b	INT8-CW	32	24.7	7.2	138.8888889
nanollava	INT4-MIXED	1752	129.5	7.3	136.9863014
nuextract-tiny	INT8-CW	32	25.5	7.3	136.9863014
whisper-small	INT4-MIXED	prompt0	81.9	7.3	136.9863014
whisper-small	INT4-MIXED	prompt1	134.8	7.3	136.9863014
qwen2.5-coder-0.5b-instruct	INT8-CW	32	26.2	7.6	131.5789474
gemma-3-270m	FP16	32	16.6	7.7	129.8701299
minicpm4-0.5b	INT8-CW	1024	44.2	7.7	129.8701299
whisper-large-v3-turbo	FP16	prompt0	167.5	7.9	126.5822785
whisper-large-v3-turbo	FP16	prompt1	221.8	7.9	126.5822785
distil-large-v2	FP16	prompt0	160.7	8	125
distil-large-v2	FP16	prompt1	220.8	8	125
gemma-3-270m	FP16	1024	28.2	8	125
nanollava	INT8-CW	760	90	8	125
nuextract-tiny	INT8-CW	1024	44.8	8	125
qwen2.5-coder-0.5b-instruct	INT8-CW	1024	47.4	8	125
tiny-llama-1.1b-chat	INT4-MIXED	32	26.1	8.1	123.4567901
lfm2.5-350m	FP16	32	13.3	8.2	121.9512195
lfm2.5-350m	FP16	1024	43.6	8.4	119.047619
minicpm5-1b	INT4-MIXED	32	29.2	8.5	117.6470588
lfm2-1.2b	INT4-MIXED	32	20.9	8.6	116.2790698
lfm2.5-1.2b-instruct	INT4-MIXED	32	19.1	8.6	116.2790698
nanollava	INT8-CW	1752	127.2	8.7	114.9425287
tiny-llama-1.1b-chat	INT4-MIXED	32	26.6	8.7	114.9425287
lfm2-1.2b	INT4-MIXED	32	22	8.8	113.6363636
lfm2.5-1.2b-instruct	INT4-MIXED	1024	92.5	8.8	113.6363636
whisper-small	FP16	prompt0	76.2	8.8	113.6363636
lfm2-1.2b	INT4-MIXED	1024	94.7	8.9	112.3595506
lfm2.5-1.2b-instruct	INT4-MIXED	32	22.2	8.9	112.3595506
tiny-llama-1.1b-chat	INT4-MIXED	1024	92.7	8.9	112.3595506
lfm2-1.2b	INT4-MIXED	1024	98	9	111.1111111
lfm2.5-1.2b-instruct	INT4-MIXED	32	22.3	9	111.1111111
whisper-small	FP16	prompt1	136.1	9	111.1111111
lfm2-1.2b	INT4-MIXED	32	22.6	9.2	108.6956522
lfm2.5-1.2b-instruct	INT4-MIXED	1024	109.1	9.2	108.6956522
lfm2.5-1.2b-instruct	INT4-MIXED	1024	98.1	9.2	108.6956522
lfm2-1.2b	INT4-MIXED	1024	109.3	9.3	107.5268817
minicpm5-1b	INT4-MIXED	1024	81.6	9.3	107.5268817
tiny-llama-1.1b-chat	INT4-MIXED	1024	111.7	9.4	106.3829787
qwen3.5-0.8b	INT4-MIXED	85	55.5	9.9	101.010101
qwen3.5-0.8b	INT4-MIXED	85	57.8	10	100
llama-3.2-1b-instruct	INT4-MIXED	32	22.3	10.1	99.00990099
qwen3-asr-0.6b	INT4-MIXED	prompt0	95.5	10.1	99.00990099
llama-3.2-1b-instruct	INT4-MIXED	32	22.4	10.2	98.03921569
qwen3-asr-0.6b	INT4-MIXED	prompt0	99.2	10.2	98.03921569
qwen3.5-0.8b	INT4-MIXED	85	61.2	10.2	98.03921569
qwen3-asr-0.6b	INT4-MIXED	prompt1	279.6	10.5	95.23809524
qwen3-asr-0.6b	INT8-CW	prompt0	99.7	10.5	95.23809524
qwen3.5-0.8b	INT4-MIXED	1024	133.8	10.5	95.23809524
qwen3-asr-0.6b	INT4-MIXED	prompt1	293.8	10.6	94.33962264
qwen3.5-0.8b	INT4-MIXED	1024	141.8	10.6	94.33962264
gemma-3-1b-it	INT4-MIXED	32	35	10.7	93.45794393
qwen3.5-0.8b	INT4-MIXED	1024	149	10.7	93.45794393
gemma-3-1b-it	INT4-MIXED	32	36.5	10.8	92.59259259
llama-3.2-1b-instruct	INT4-MIXED	1024	93	10.8	92.59259259
gemma-3-1b-it	INT4-MIXED	32	34.6	10.9	91.74311927
gemma-3-1b-it	INT4-MIXED	1024	83.7	10.9	91.74311927
exaone-4.0-1.2b	INT4-MIXED	32	42.6	11	90.90909091
llama-3.2-1b-instruct	INT4-MIXED	1024	103.3	11	90.90909091
gemma-3-1b-it	INT4-MIXED	1024	87.5	11.1	90.09009009
minicpm4-0.5b	FP16	32	23.3	11.1	90.09009009
qwen3-asr-0.6b	INT8-CW	prompt1	288.1	11.1	90.09009009
exaone-4.0-1.2b	INT4-MIXED	32	42.4	11.4	87.71929825
gemma-3-1b-it	INT4-MIXED	1024	93.6	11.4	87.71929825
exaone-4.0-1.2b	INT4-MIXED	32	44.6	11.5	86.95652174
minicpm4-0.5b	FP16	1024	58.3	11.5	86.95652174
nuextract-tiny	FP16	32	26.7	11.5	86.95652174
minicpm5-1b	INT8-CW	32	31.3	11.6	86.20689655
qwen3.5-0.8b	INT8-CW	85	72.9	11.6	86.20689655
qwen3-asr-0.6b	INT4-MIXED	prompt0	135.2	11.9	84.03361345
exaone-4.0-1.2b	INT4-MIXED	1024	143.4	12.1	82.6446281
qwen3.5-0.8b	INT8-CW	1024	151.6	12.1	82.6446281
nanollava	FP16	760	91.7	12.2	81.96721311
qwen2.5-coder-0.5b-instruct	FP16	32	24.3	12.3	81.30081301
qwen2.5-coder-1.5b-instruct	INT4-MIXED	32	33.3	12.3	81.30081301
qwen3-asr-0.6b	INT4-MIXED	prompt1	297.7	12.3	81.30081301
exaone-4.0-1.2b	INT4-MIXED	1024	148.4	12.4	80.64516129
nuextract-tiny	FP16	1024	55.4	12.4	80.64516129
deepseek-r1-distill-qwen-1.5b	INT4-MIXED	32	33.3	12.5	80
minicpm5-1b	INT8-CW	1024	82.6	12.5	80
qwen2.5-1.5b-instruct	INT4-MIXED	32	35.3	12.5	80
qwen2.5-coder-0.5b-instruct	FP16	1024	53.3	12.5	80
exaone-4.0-1.2b	INT4-MIXED	1024	156.1	12.6	79.36507937
lfm2-8b-a1b	INT4-MIXED	32	163.3	12.6	79.36507937
qwen2.5-1.5b-instruct	INT4-MIXED	32	33.8	12.6	79.36507937
qwen2.5-coder-1.5b-instruct	INT4-MIXED	32	37.7	12.6	79.36507937
tiny-llama-1.1b-chat	INT8-CW	32	29.4	12.6	79.36507937
lfm2-8b-a1b	INT4-MIXED	1024	346.2	12.7	78.74015748
qwen2.5-coder-1.5b-instruct	INT4-MIXED	32	34.1	12.7	78.74015748
qwen2.5-coder-1.5b-instruct	INT4-MIXED	1024	122.3	13	76.92307692
deepseek-r1-distill-qwen-1.5b	INT4-MIXED	1024	152.6	13.2	75.75757576
nanollava	FP16	1752	142.7	13.3	75.18796992
qwen2.5-1.5b-instruct	INT4-MIXED	1024	153.1	13.3	75.18796992
qwen2.5-1.5b-instruct	INT4-MIXED	1024	153.4	13.3	75.18796992
qwen2.5-coder-1.5b-instruct	INT4-MIXED	1024	160.3	13.3	75.18796992
qwen2.5-1.5b-instruct	INT4-MIXED	32	36.6	13.4	74.62686567
qwen2.5-1.5b-instruct	INT4-MIXED	32	39.3	13.4	74.62686567
tiny-llama-1.1b-chat	INT8-CW	1024	106.4	13.4	74.62686567
qwen2.5-coder-1.5b-instruct	INT4-MIXED	1024	153.6	13.6	73.52941176
minicpm-1b-sft	INT4-MIXED	31	53.4	13.7	72.99270073
gemma-3-1b-it	INT8-CW	32	38.2	13.9	71.94244604
lfm2.5-1.2b-instruct	INT4-MIXED	32	20.4	13.9	71.94244604
lfm2-1.2b	INT4-MIXED	32	20.6	14	71.42857143
minicpm-1b-sft	INT4-MIXED	31	55	14	71.42857143
lfm2.5-1.2b-instruct	INT8-CW	32	23.1	14.1	70.92198582
qwen2.5-1.5b-instruct	INT4-MIXED	1024	159.8	14.1	70.92198582
qwen2.5-1.5b-instruct	INT4-MIXED	1024	158.7	14.1	70.92198582
gemma-3-1b-it	INT8-CW	1024	93.6	14.2	70.42253521
lfm2-1.2b	INT8-CW	32	27.4	14.2	70.42253521
lfm2.5-1.2b-instruct	INT4-MIXED	1024	104.1	14.2	70.42253521
lfm2-1.2b	INT4-MIXED	1024	105.3	14.3	69.93006993
minicpm-1b-sft	INT4-MIXED	31	56.9	14.3	69.93006993
lfm2.5-1.2b-instruct	INT8-CW	1024	114.9	14.4	69.44444444
lfm2-1.2b	INT8-CW	1024	117.4	14.5	68.96551724
minicpm-1b-sft	INT4-MIXED	1014	119.8	14.5	68.96551724
minicpm-1b-sft	INT4-MIXED	1014	129.1	14.9	67.11409396
llama-3.2-1b-instruct	INT8-CW	32	24	15.1	66.22516556
minicpm-1b-sft	INT4-MIXED	1014	143.4	15.1	66.22516556
qwen3-asr-0.6b	FP16	prompt0	104.5	15.3	65.35947712
deepseek-r1-distill-qwen-1.5b	INT4-MIXED	32	36.3	15.9	62.89308176
llama-3.2-1b-instruct	INT8-CW	1024	104.9	15.9	62.89308176
deepseek-r1-distill-qwen-1.5b	INT4-MIXED	1024	160.8	16.5	60.60606061
qwen3-asr-0.6b	FP16	prompt1	307.4	16.6	60.24096386
exaone-4.0-1.2b	INT8-CW	32	46.9	17.2	58.13953488
qwen2.5-1.5b-instruct	INT4-MIXED	32	63	18	55.55555556
gemma-2b-it	INT4-MIXED	32	27.9	18.1	55.24861878
lfm2-8b-a1b	INT8-CW	32	114.7	18.2	54.94505495
exaone-4.0-1.2b	INT8-CW	1024	152.3	18.3	54.64480874
qwen3.5-0.8b	FP16	85	58.9	18.4	54.34782609
lfm2-8b-a1b	INT8-CW	1024	400.6	18.6	53.76344086
deepseek-r1-distill-qwen-1.5b	INT8-CW	32	38.6	18.7	53.47593583
qwen2.5-1.5b-instruct	INT4-MIXED	32	65.7	18.7	53.47593583
qwen3.5-0.8b	FP16	1024	169.9	18.7	53.47593583
gemma-2b-it	INT4-MIXED	1024	173	18.8	53.19148936
qwen2.5-1.5b-instruct	INT4-MIXED	1024	186.8	18.8	53.19148936
qwen2.5-1.5b-instruct	INT8-CW	32	38.9	18.8	53.19148936
qwen2.5-1.5b-instruct	INT8-CW	32	40.3	18.8	53.19148936
qwen2.5-coder-1.5b-instruct	INT8-CW	32	39.4	18.8	53.19148936
lfm2-24b-a2b	INT4-MIXED	32	242.4	18.9	52.91005291
gemma-2b-it	INT4-MIXED	32	30.5	19	52.63157895
phi-2	INT4-MIXED	32	52.5	19.1	52.35602094
lfm2-24b-a2b	INT4-MIXED	1024	776.8	19.2	52.08333333
minicpm-1b-sft	INT8-CW	31	66.4	19.3	51.8134715
stablelm-3b-4e1t	INT4-MIXED	32	56.3	19.3	51.8134715
deepseek-r1-distill-qwen-1.5b	INT8-CW	1024	139.1	19.4	51.54639175
qwen2.5-1.5b-instruct	INT4-MIXED	1024	192.6	19.4	51.54639175
stable-zephyr-3b-dpo	INT4-MIXED	32	58.5	19.5	51.28205128
gemma-4-e2b-it	INT4-MIXED	274	289	19.6	51.02040816
qwen2.5-1.5b-instruct	INT8-CW	1024	138.2	19.6	51.02040816
qwen2.5-1.5b-instruct	INT8-CW	1024	139.5	19.6	51.02040816
qwen2.5-coder-1.5b-instruct	INT8-CW	1024	138.5	19.6	51.02040816
gemma-2b-it	INT4-MIXED	1024	211.9	19.8	50.50505051
gemma-4-e2b-it	INT4-MIXED	1024	463.7	20	50
whisper-large-v3	INT4-MIXED	prompt0	237.7	20.2	49.5049505
whisper-large-v3	INT4-MIXED	prompt1	298.5	20.3	49.26108374
gemma-2-2b	INT4-MIXED	33	39.3	20.5	48.7804878
gemma-4-e2b-it	INT4-MIXED	274	309.4	20.6	48.54368932
smollm3-3b	INT4-MIXED	32	45.9	20.7	48.30917874
minicpm-1b-sft	INT8-CW	1014	143.8	20.8	48.07692308
phi-2	INT4-MIXED	32	56.8	20.8	48.07692308
minicpm5-1b	FP16	32	27.1	20.9	47.84688995
gemma-4-e2b-it	INT4-MIXED	1024	492.1	21	47.61904762
gemma-2-2b	INT4-MIXED	33	40.3	21.2	47.16981132
gemma-2-2b	INT4-MIXED	1025	204.3	21.5	46.51162791
phi-2	INT4-MIXED	1024	289.2	21.5	46.51162791
smollm3-3b	INT4-MIXED	32	48.5	21.6	46.2962963
stable-zephyr-3b-dpo	INT4-MIXED	1024	288.6	21.7	46.08294931
stablelm-3b-4e1t	INT4-MIXED	1024	287.5	21.7	46.08294931
qwen2.5-coder-3b-instruct	INT4-MIXED	32	47.8	21.8	45.87155963
minicpm5-1b	FP16	1024	94.9	21.9	45.66210046
qwen2.5-coder-3b-instruct	INT4-MIXED	32	46.8	21.9	45.66210046
smollm3-3b	INT4-MIXED	32	47	21.9	45.66210046
stable-zephyr-3b-dpo	INT4-MIXED	32	64.6	21.9	45.66210046
smollm3-3b	INT4-MIXED	1024	228.3	22	45.45454545
gemma-2-2b	INT4-MIXED	1025	229.7	22.4	44.64285714
llama-3.2-3b-instruct	INT4-MIXED	32	38.8	22.4	44.64285714
smollm3-3b	INT4-MIXED	1024	243.9	22.9	43.66812227
granite-4.0-h-micro	INT4-MIXED	33	89.8	23.1	43.29004329
llama-3.2-3b-instruct	INT4-MIXED	32	37.2	23.2	43.10344828
qwen2.5-coder-3b-instruct	INT4-MIXED	1024	223.1	23.2	43.10344828
qwen2.5-coder-3b-instruct	INT4-MIXED	1024	238.8	23.2	43.10344828
smollm3-3b	INT4-MIXED	1024	275.5	23.2	43.10344828
granite-4.0-h-micro	INT4-MIXED	1033	692.5	23.3	42.91845494
qwen2.5-coder-3b-instruct	INT4-MIXED	32	52.3	23.3	42.91845494
llama-3.2-3b-instruct	INT4-MIXED	32	39.4	23.4	42.73504274
phi-2	INT4-MIXED	1024	361.7	23.5	42.55319149
stablelm-3b-4e1t	INT4-MIXED	32	61.5	23.5	42.55319149
gemma-3-1b-it	FP16	32	35.3	23.6	42.37288136
gemma-3-1b-it	FP16	1024	107	23.8	42.01680672
whisper-large-v3	INT8-CW	prompt0	237.9	23.8	42.01680672
llama-3.2-3b-instruct	INT4-MIXED	1024	234.6	23.9	41.84100418
granite-4.0-h-micro	INT4-MIXED	33	94.7	24	41.66666667
granite-4.0-h-micro	INT4-MIXED	1033	711.6	24.1	41.49377593
phi-3-mini-128k-instruct	INT4-MIXED	32	43.4	24.1	41.49377593
phi-3-mini-4k-instruct	INT4-MIXED	32	43.2	24.2	41.32231405
phi-3.5-mini-instruct	INT4-MIXED	32	42	24.2	41.32231405
qwen2.5-1.5b-instruct	INT8-CW	32	70.1	24.2	41.32231405
stable-zephyr-3b-dpo	INT4-MIXED	1024	325.4	24.2	41.32231405
granite-4.0-h-micro	INT4-MIXED	33	97	24.3	41.15226337
granite-4.0-h-micro	INT4-MIXED	1033	766	24.4	40.98360656
qwen2.5-coder-3b-instruct	INT4-MIXED	1024	372.6	24.6	40.6504065
whisper-large-v3	INT8-CW	prompt1	299.7	24.6	40.6504065
llama-3.2-3b-instruct	INT4-MIXED	1024	251.8	24.8	40.32258065
llama-3.2-3b-instruct	INT4-MIXED	1024	273	25.1	39.84063745
qwen2.5-1.5b-instruct	INT8-CW	1024	179.6	25.1	39.84063745
phi-3-mini-4k-instruct	INT4-MIXED	32	45.2	25.2	39.68253968
phi-3.5-mini-instruct	INT4-MIXED	32	44.2	25.3	39.5256917
qwen2.5-3b-instruct-gptq	INT4-MIXED	32	51.5	25.6	39.0625
stablelm-3b-4e1t	INT4-MIXED	1024	316.8	25.6	39.0625
phi-3-mini-128k-instruct	INT4-MIXED	32	44.9	25.8	38.75968992
tiny-llama-1.1b-chat	FP16	32	29.3	26.2	38.16793893
gemma-3n-e2b-it	INT4-MIXED	275	528.8	26.3	38.02281369
gemma-3n-e2b-it	INT4-MIXED	1025	892	26.5	37.73584906
phi-3-mini-128k-instruct	INT4-MIXED	1024	368.4	26.7	37.45318352
phi-3-mini-4k-instruct	INT4-MIXED	1024	366.9	26.7	37.45318352
phi-3.5-mini-instruct	INT4-MIXED	1024	366.4	26.8	37.31343284
qwen2.5-3b-instruct-gptq	INT4-MIXED	1024	240.8	26.9	37.17472119
gemma-3n-e2b-it	INT4-MIXED	275	530.5	27	37.03703704
gemma-3n-e2b-it	INT4-MIXED	1025	919.7	27.1	36.900369
tiny-llama-1.1b-chat	FP16	1024	115.7	27.1	36.900369
phi-3-mini-4k-instruct	INT4-MIXED	32	43.7	27.2	36.76470588
qwen3-4b	INT4-MIXED	32	53.8	27.3	36.63003663
phi-3.5-mini-instruct	INT4-MIXED	32	49	27.4	36.49635036
phi-3-mini-4k-instruct	INT4-MIXED	1024	390.1	27.7	36.10108303
phi-3.5-vision-instruct	INT4-MIXED	802	308.4	27.7	36.10108303
phi-3.5-mini-instruct	INT4-MIXED	1024	390.7	27.8	35.97122302
phi-3-mini-128k-instruct	INT4-MIXED	1024	488.6	28.2	35.46099291
phi-3.5-vision-instruct	INT4-MIXED	1032	802.4	28.2	35.46099291
gemma-3-4b-it	INT4-MIXED	32	62.1	28.3	35.33568905
phi-4-mini-instruct	INT4-MIXED	32	57.9	28.3	35.33568905
phi-4-mini-reasoning	INT4-MIXED	32	58.1	28.4	35.21126761
gpt-oss-20b	INT4-MIXED	32	239.5	28.5	35.0877193
qwen3-4b	INT4-MIXED	32	56	28.5	35.0877193
lfm2-1.2b	FP16	32	33.6	28.8	34.72222222
lfm2.5-1.2b-instruct	FP16	32	33.7	28.9	34.60207612
afm-4.5b	INT4-MIXED	32	47.9	29	34.48275862
lfm2-1.2b	FP16	1024	143.1	29	34.48275862
lfm2.5-1.2b-instruct	FP16	1024	142	29.1	34.36426117
phi-4-mini-instruct	INT4-MIXED	32	60	29.2	34.24657534
phi-4-mini-reasoning	INT4-MIXED	32	58.4	29.2	34.24657534
gemma-2b-it	INT8-CW	32	37.2	29.3	34.12969283
gemma-3-4b-it	INT4-MIXED	1024	335.3	29.3	34.12969283
gemma-3-4b-it	INT4-MIXED	32	63.5	29.3	34.12969283
gpt-oss-20b	INT4-MIXED	1024	679.5	29.3	34.12969283
phi-4-mini-reasoning	INT4-MIXED	32	59.9	29.4	34.01360544
phi-3-mini-4k-instruct	INT4-MIXED	1024	515.6	29.7	33.67003367
phi-3.5-mini-instruct	INT4-MIXED	1024	557.3	29.9	33.44481605
qwen3-4b	INT4-MIXED	1024	337.3	29.9	33.44481605
phi-4-mini-instruct	INT4-MIXED	1024	368	30	33.33333333
phi-4-mini-reasoning	INT4-MIXED	1024	369.8	30	33.33333333
gemma-4-e2b-it	INT8-CW	274	315.2	30.1	33.22259136
gemma-2b-it	INT8-CW	1024	210.4	30.2	33.11258278
gemma-3-4b-it	INT4-MIXED	1024	356.6	30.3	33.00330033
llama-3.2-1b-instruct	FP16	32	34.9	30.4	32.89473684
afm-4.5b	INT4-MIXED	1024	391.1	30.6	32.67973856
gemma-4-e2b-it	INT8-CW	1024	504.9	30.8	32.46753247
phi-4-mini-instruct	INT4-MIXED	1024	390	30.9	32.36245955
phi-4-mini-reasoning	INT4-MIXED	1024	392.1	31	32.25806452
qwen3-4b	INT4-MIXED	1024	355.5	31	32.25806452
llama-3.2-1b-instruct	FP16	1024	134.9	31.1	32.15434084
phi-4-mini-instruct	INT4-MIXED	32	63.8	31.2	32.05128205
gemma-2-2b	INT8-CW	33	46	31.3	31.94888179
phi-4-mini-reasoning	INT4-MIXED	1024	411.2	31.3	31.94888179
qwen3-vl-4b-instruct	INT4-MIXED	4907	1073.2	31.8	31.44654088
qwen3-vl-4b-instruct	INT4-MIXED	4937	1135	31.8	31.44654088
minicpm3-4b	INT4-MIXED	32	122.2	32	31.25
phi-2	INT8-CW	32	59.7	32.5	30.76923077
stable-zephyr-3b-dpo	INT8-CW	32	60.3	32.5	30.76923077
stablelm-3b-4e1t	INT8-CW	32	60.5	32.6	30.67484663
gemma-2-2b	INT8-CW	1025	235.1	32.8	30.48780488
phi-4-mini-instruct	INT4-MIXED	1024	475.3	32.9	30.39513678
qwen3-vl-4b-instruct	INT4-MIXED	4907	1087.2	33	30.3030303
qwen3-vl-4b-instruct	INT4-MIXED	4937	1145.6	33.1	30.21148036
minicpm-1b-sft	FP16	31	52.3	33.4	29.94011976
minicpm3-4b	INT4-MIXED	32	123.3	33.4	29.94011976
qwen3-vl-4b-instruct	INT4-MIXED	4907	1073.1	33.4	29.94011976
qwen3-vl-4b-instruct	INT4-MIXED	4937	1121.2	33.4	29.94011976
minicpm3-4b	INT4-MIXED	32	131.4	33.7	29.6735905
phi-4-multimodal-instruct	INT4-MIXED	578	447.6	34.1	29.3255132
whisper-large-v3	FP16	prompt0	231.8	34.2	29.23976608
qwen2.5-1.5b-instruct	FP16	32	38.2	34.3	29.15451895
deepseek-r1-distill-qwen-1.5b	FP16	32	38.4	34.4	29.06976744
phi-4-multimodal-instruct	INT4-MIXED	786	322.7	34.4	29.06976744
qwen2.5-coder-1.5b-instruct	FP16	32	38.8	34.4	29.06976744
smollm3-3b	INT8-CW	32	53.2	34.4	29.06976744
qwen2.5-coder-3b-instruct	INT8-CW	32	55.4	34.7	28.8184438
exaone-4.0-1.2b	FP16	32	39.3	34.8	28.73563218
stablelm-3b-4e1t	INT8-CW	1024	319.7	34.8	28.73563218
minicpm-1b-sft	FP16	1014	180.3	34.9	28.65329513
phi-2	INT8-CW	1024	319.4	34.9	28.65329513
stable-zephyr-3b-dpo	INT8-CW	1024	320.2	35.1	28.49002849
deepseek-r1-distill-qwen-1.5b	FP16	1024	169.4	35.2	28.40909091
qwen2.5-1.5b-instruct	FP16	1024	167.7	35.3	28.3286119
qwen2.5-coder-1.5b-instruct	FP16	1024	168.9	35.3	28.3286119
whisper-large-v3	FP16	prompt1	293.5	35.3	28.3286119
phi-4-multimodal-instruct	INT4-MIXED	1362	1063.8	35.5	28.16901408
phi-4-multimodal-instruct	INT4-MIXED	1570	631.7	35.6	28.08988764
gemma-3n-e2b-it	INT8-CW	275	551.5	35.7	28.01120448
smollm3-3b	INT8-CW	1024	273.7	35.9	27.8551532
exaone-4.0-1.2b	FP16	1024	178.6	36	27.77777778
gemma-3n-e2b-it	INT8-CW	1025	939.2	36.1	27.70083102
gpt-oss-20b	INT4-MIXED	32	233.3	36.1	27.70083102
qwen2.5-coder-3b-instruct	INT8-CW	1024	269.9	36.1	27.70083102
minicpm3-4b	INT4-MIXED	1024	757.3	36.2	27.62430939
gpt-oss-20b	INT4-MIXED	1024	696.9	37.1	26.9541779
minicpm3-4b	INT4-MIXED	1024	796.8	37.6	26.59574468
gemma-3n-e4b-it	INT4-MIXED	275	566.6	37.9	26.38522427
llama-3.2-3b-instruct	INT8-CW	32	46.9	38	26.31578947
minicpm3-4b	INT4-MIXED	1024	841	38	26.31578947
codellama-7b	INT4-MIXED	32	56.8	38.4	26.04166667
gemma-3n-e4b-it	INT4-MIXED	1025	1107.2	38.4	26.04166667
llama-2-7b-chat-hf	INT4-MIXED	32	56.8	38.5	25.97402597
stable-diffusion-xl-1.0-inpainting-0.1	INT8-CW	32	39.3	38.7	25.83979328
stable-diffusion-xl-1.0-inpainting-0.1	INT8-CW	32	40.1	39.6	25.25252525
llama-3.2-3b-instruct	INT8-CW	1024	270.9	39.7	25.18891688
gemma-3n-e4b-it	INT4-MIXED	275	569	39.8	25.12562814
gemma-3n-e4b-it	INT4-MIXED	1025	1169.3	40.2	24.87562189
phi-4-multimodal-instruct	INT4-MIXED	578	507.6	40.3	24.81389578
qwen3_8b_draft_eagle3	INT4-MIXED	32	65.6	40.3	24.81389578
granite-4.0-h-micro	INT8-CW	33	94.8	40.4	24.75247525
phi-4-multimodal-instruct	INT4-MIXED	786	357.7	40.5	24.69135802
biomistral-7b-slerp	INT4-MIXED	7	47.1	40.6	24.63054187
granite-4.0-h-micro	INT8-CW	1033	740.1	40.6	24.63054187
llama-2-7b-chat-hf	INT4-MIXED	32	60.9	40.6	24.63054187
mistral-7b-instruct-v0.2	INT4-MIXED	32	60.9	40.8	24.50980392
neural-chat-7b-v3-3	INT4-MIXED	32	60.5	40.8	24.50980392
zephyr-7b-beta	INT4-MIXED	32	61	40.8	24.50980392
mistral-7b-instruct-v0.3	INT4-MIXED	32	61.2	40.9	24.44987775
codellama-7b	INT4-MIXED	32	62.1	41	24.3902439
codellama-7b	INT4-MIXED	1024	510.3	41.2	24.27184466
llama-2-7b-chat-hf	INT4-MIXED	1024	512	41.4	24.15458937
falcon-7b-instruct	INT4-MIXED	32	57.8	41.6	24.03846154
phi-4-multimodal-instruct	INT4-MIXED	1362	1209	41.6	24.03846154
phi-4-multimodal-instruct	INT4-MIXED	1570	668.2	41.8	23.92344498
neural-chat-7b-v3-3	INT4-MIXED	1024	536.3	42.7	23.41920375
zephyr-7b-beta	INT4-MIXED	1024	535.3	42.7	23.41920375
mistral-7b-instruct-v0.2	INT4-MIXED	1024	536.1	42.8	23.36448598
mistral-7b-instruct-v0.3	INT4-MIXED	1024	535	42.9	23.31002331
qwen2.5-7b-instruct	INT4-MIXED	32	58.1	42.9	23.31002331
qwen2.5-7b-instruct	INT4-MIXED	32	59	43	23.25581395
mistral-7b-instruct-v0.3	INT4-MIXED	32	65.8	43.1	23.20185615
qwen2.5-7b-instruct-1m	INT4-MIXED	32	57.1	43.1	23.20185615
mistral-7b-instruct-v0.2	INT4-MIXED	32	65.6	43.2	23.14814815
qwen3_8b_draft_eagle3	INT4-MIXED	1024	642.1	43.2	23.14814815
qwen2.5-vl-7b-instruct	INT4-MIXED	32	67.7	43.3	23.09468822
biomistral-7b-slerp	INT4-MIXED	7	50.3	43.4	23.04147465
minicpm-o-2_6	INT4-MIXED	238	508.4	43.4	23.04147465
minicpm-v-2_6	INT4-MIXED	228	499.5	43.4	23.04147465
llama-2-7b-chat-hf	INT4-MIXED	1024	546.8	43.5	22.98850575
mistral-7b-instruct-v0.1	INT4-MIXED	32	67	43.6	22.93577982
mistral-7b-instruct-v0.2	INT4-MIXED	32	67.6	43.6	22.93577982
neural-chat-7b-v3-3	INT4-MIXED	32	66.3	43.6	22.93577982
mistral-7b-instruct-v0.3	INT4-MIXED	32	65.7	43.7	22.88329519
qwen-7b-chat	INT4-MIXED	32	63.7	43.8	22.83105023
codellama-7b	INT4-MIXED	1024	609.6	44.1	22.67573696
falcon-7b-instruct	INT4-MIXED	1024	551.7	44.1	22.67573696
phi-3-mini-4k-instruct	INT8-CW	32	58.9	44.2	22.62443439
phi-3-mini-128k-instruct	INT8-CW	32	58.2	44.3	22.57336343
phi-3.5-mini-instruct	INT8-CW	32	58.8	44.4	22.52252252
deepseek-r1-distill-qwen-7b	INT4-MIXED	32	60.9	44.7	22.37136465
deepseek-r1-distill-llama-8b	INT4-MIXED	35	70.9	44.8	22.32142857
llama-3-8b-instruct	INT4-MIXED	32	65.7	44.8	22.32142857
llama-3.1-8b-instruct	INT4-MIXED	32	65.8	44.8	22.32142857
qwen2.5-7b-instruct	INT4-MIXED	32	61.3	44.8	22.32142857
qwen2.5-7b-instruct	INT4-MIXED	1024	485.6	44.9	22.27171492
qwen2.5-7b-instruct	INT4-MIXED	32	61	44.9	22.27171492
qwen2.5-7b-instruct-1m	INT4-MIXED	1024	495.7	44.9	22.27171492
gemma-3-4b-it	INT8-CW	32	67.8	45	22.22222222
minicpm-v-2_6	INT4-MIXED	228	511.2	45	22.22222222
qwen2-7b-instruct	INT4-MIXED	32	61.1	45	22.22222222
qwen2.5-7b-instruct	INT4-MIXED	1024	494.5	45	22.22222222
qwen2-7b-instruct-gptq	INT4-MIXED	32	61.3	45.1	22.172949
qwen2.5-7b-instruct	INT4-MIXED	32	60.8	45.1	22.172949
qwen2.5-7b-instruct-1m	INT4-MIXED	32	61.1	45.1	22.172949
qwen2.5-vl-7b-instruct	INT4-MIXED	32	70.2	45.1	22.172949
mistral-7b-instruct-v0.2	INT4-MIXED	1024	568.6	45.2	22.12389381
mistral-7b-instruct-v0.3	INT4-MIXED	1024	568.7	45.2	22.12389381
qwen2.5-vl-7b-instruct	INT4-MIXED	1024	513.5	45.2	22.12389381
qwen2-7b-instruct	INT4-MIXED	32	61.6	45.3	22.07505519
qwen2.5-7b-instruct	INT4-MIXED	32	61.6	45.3	22.07505519
qwen2.5-vl-7b-instruct	INT4-MIXED	32	71.2	45.3	22.07505519
fara-7b	INT4-MIXED	32	104.5	45.4	22.02643172
minicpm-v-2_6	INT4-MIXED	228	525.2	45.4	22.02643172
deepseek-r1-distill-qwen-7b	INT4-MIXED	32	61.9	45.5	21.97802198
bloomz-7b1	INT4-MIXED	32	71.3	45.6	21.92982456
minicpm-o-2_6	INT4-MIXED	238	518.9	45.6	21.92982456
mistral-7b-instruct-v0.1	INT4-MIXED	1025	688.7	45.6	21.92982456
mistral-7b-instruct-v0.2	INT4-MIXED	1024	655	45.6	21.92982456
mistral-7b-instruct-v0.3	INT4-MIXED	1024	656.2	45.7	21.88183807
neural-chat-7b-v3-3	INT4-MIXED	1024	653.9	45.7	21.88183807
phi-4-mini-instruct	INT8-CW	32	62.3	45.9	21.78649237
minicpm-v-4_5	INT4-MIXED	217	531.2	46.1	21.69197397
phi-4-mini-reasoning	INT8-CW	32	63.3	46.2	21.64502165
falcon-7b-instruct	INT4-MIXED	32	63.7	46.3	21.59827214
qwen-7b-chat	INT4-MIXED	32	68.2	46.3	21.59827214
qwen3-8b	INT4-MIXED	32	63.4	46.3	21.59827214
minicpm4-8b	INT4-MIXED	32	68.2	46.4	21.55172414
phi-3.5-vision-instruct	INT8-CW	802	334.9	46.4	21.55172414
deepseek-r1-distill-qwen-7b	INT4-MIXED	1024	515.4	46.5	21.50537634
gemma-3-4b-it	INT8-CW	1024	377.1	46.5	21.50537634
qwen2.5-7b-instruct	INT4-MIXED	1024	527.1	46.5	21.50537634
qwen2-7b-instruct	INT4-MIXED	1024	534.1	46.6	21.45922747
qwen2-7b-instruct-gptq	INT4-MIXED	1024	535.9	46.6	21.45922747
qwen2.5-7b-instruct	INT4-MIXED	1024	537.5	46.6	21.45922747
qwen-7b-chat	INT4-MIXED	1024	540.1	46.7	21.41327623
llama-3.1-8b-instruct	INT4-MIXED	1024	538.6	46.8	21.36752137
llama-3-8b-instruct	INT4-MIXED	1024	539.4	46.9	21.32196162
phi-3-mini-128k-instruct	INT8-CW	1024	487	46.9	21.32196162
phi-3-mini-4k-instruct	INT8-CW	1024	486.4	46.9	21.32196162
phi-3.5-vision-instruct	INT8-CW	1032	711.4	46.9	21.32196162
qwen2.5-7b-instruct	INT4-MIXED	1024	606.9	46.9	21.32196162
qwen2.5-7b-instruct-1m	INT4-MIXED	1024	604.9	46.9	21.32196162
deepseek-r1-distill-llama-8b	INT4-MIXED	1135	591	47	21.27659574
deepseek-r1-distill-qwen-7b	INT4-MIXED	1024	600.8	47	21.27659574
llama-3-8b-instruct	INT4-MIXED	32	70.9	47.1	21.23142251
phi-3.5-mini-instruct	INT8-CW	1024	486.1	47.1	21.23142251
qwen2-7b-instruct	INT4-MIXED	1024	612.9	47.1	21.23142251
qwen2.5-7b-instruct	INT4-MIXED	1024	613.6	47.1	21.23142251
qwen2.5-vl-7b-instruct	INT4-MIXED	1024	564.9	47.1	21.23142251
qwen3-4b	INT8-CW	32	59.7	47.1	21.23142251
llama-3-8b-instruct	INT4-MIXED	32	70.5	47.2	21.18644068
llama-3.1-8b-instruct	INT4-MIXED	32	70.4	47.2	21.18644068
qwen2.5-vl-7b-instruct	INT4-MIXED	1024	641.4	47.3	21.14164905
fara-7b	INT4-MIXED	1024	762.5	47.4	21.09704641
deepseek-r1-distill-llama-8b	INT4-MIXED	35	72	47.5	21.05263158
llama-3-8b-instruct	INT4-MIXED	32	70.9	47.7	20.96436059
phi-4-mini-instruct	INT8-CW	1024	410.8	47.8	20.92050209
phi-4-mini-reasoning	INT8-CW	1024	415.2	48.1	20.79002079
afm-4.5b	INT8-CW	32	62.1	48.2	20.74688797
bloomz-7b1	INT4-MIXED	32	77.2	48.2	20.74688797
bloomz-7b1	INT4-MIXED	1024	725.3	48.3	20.70393375
qwen3-8b	INT4-MIXED	32	70.7	48.3	20.70393375
minicpm4-8b	INT4-MIXED	1024	587	48.4	20.66115702
qwen3-8b	INT4-MIXED	1024	561.2	48.4	20.66115702
falcon-7b-instruct	INT4-MIXED	1024	675.9	48.7	20.5338809
qwen-7b-chat	INT4-MIXED	1024	573.3	48.7	20.5338809
minicpm-v-4_5	INT4-MIXED	217	535.4	49	20.40816327
minicpm4-8b	INT4-MIXED	32	73.9	49	20.40816327
llama-3-8b-instruct	INT4-MIXED	1024	561.5	49.1	20.36659878
qwen3-8b	INT4-MIXED	32	72.1	49.1	20.36659878
llama-3-8b-instruct	INT4-MIXED	1024	561.5	49.2	20.32520325
llama-3.1-8b-instruct	INT4-MIXED	1024	562.5	49.3	20.28397566
minicpm4-8b	INT4-MIXED	32	75.2	49.5	20.2020202
deepseek-r1-distill-llama-8b	INT4-MIXED	1135	629.6	49.6	20.16129032
afm-4.5b	INT8-CW	1024	385.3	49.8	20.08032129
llama-3-8b-instruct	INT4-MIXED	1024	647.9	49.8	20.08032129
qwen3-4b	INT8-CW	1024	376.9	49.9	20.04008016
llava-next-video-7b-hf	INT4-MIXED	2945	542.7	50	20
qwen3.5-9b	INT4-MIXED	83	158.6	50.3	19.88071571
qwen3-8b	INT4-MIXED	1024	597.1	50.6	19.76284585
minicpm4-8b	INT4-MIXED	1024	643.8	50.7	19.72386588
bloomz-7b1	INT4-MIXED	1024	853.9	50.8	19.68503937
qwen3.5-9b	INT4-MIXED	1024	827	51.1	19.56947162
qwen3-8b	INT4-MIXED	1024	677	51.2	19.53125
minicpm4-8b	INT4-MIXED	1024	725.5	51.4	19.45525292
zephyr-7b-beta	INT4-MIXED	32	67.8	51.7	19.34235977
baichuan2-7b-chat	INT4-MIXED	32	69.6	52.5	19.04761905
gemma-7b-it	INT4-MIXED	32	77.7	52.5	19.04761905
qwen3-vl-4b-instruct	INT8-CW	4907	1088.8	53.1	18.83239171
qwen3.5-9b	INT4-MIXED	83	164.7	53.1	18.83239171
gemma-4-e2b-it	FP16	274	343.5	53.2	18.79699248
qwen3-vl-4b-instruct	INT8-CW	4937	1123.2	53.3	18.76172608
minicpm3-4b	INT8-CW	32	133.4	53.6	18.65671642
gemma-4-e2b-it	FP16	1024	584.6	53.7	18.62197393
qwen3.5-9b	INT4-MIXED	1024	1022.5	53.8	18.58736059
zephyr-7b-beta	INT4-MIXED	1024	579.3	54	18.51851852
phi-4-multimodal-instruct	INT8-CW	578	490	54.4	18.38235294
phi-4-multimodal-instruct	INT8-CW	786	336.4	54.7	18.28153565
gemma-7b-it	INT4-MIXED	1024	645.2	54.9	18.21493625
baichuan2-7b-chat	INT4-MIXED	1024	665.1	55.3	18.08318264
gemma-7b-it	INT4-MIXED	32	88.8	55.5	18.01801802
phi-4-multimodal-instruct	INT8-CW	1362	1156.9	55.9	17.88908766
phi-4-multimodal-instruct	INT8-CW	1570	638.5	56.1	17.82531194
videochat-flash-qwen2_5-7b_internvideo2-1b	INT4-MIXED	14531	49981.7	56.4	17.73049645
gemma-7b-it	INT4-MIXED	1024	785	57.9	17.27115717
deepseek-r1-distill-llama-8b	INT4-MIXED	35	78.7	58	17.24137931
llama-3.1-8b-instruct	INT4-MIXED	32	76	58	17.24137931
gemma-2-9b-it	INT4-MIXED	32	78.6	58.8	17.00680272
minicpm3-4b	INT8-CW	1024	826.3	59.1	16.92047377
phi-2	FP16	32	65.6	59.4	16.83501684
stable-zephyr-3b-dpo	FP16	32	65.6	59.8	16.72240803
gemma-2-2b	FP16	33	64.6	60	16.66666667
stablelm-3b-4e1t	FP16	32	65.6	60	16.66666667
llama-3.1-8b-instruct	INT4-MIXED	1024	714.9	60.4	16.55629139
deepseek-r1-distill-llama-8b	INT4-MIXED	1135	783.5	60.6	16.50165017
gemma-2-9b-it	INT4-MIXED	32	82.9	61	16.39344262
gemma-3n-e4b-it	INT8-CW	275	577.4	61.2	16.33986928
qwen-7b-chat	INT4-MIXED	32	76	61.4	16.28664495
gemma-3n-e4b-it	INT8-CW	1025	1189.3	61.5	16.2601626
gemma-2-2b	FP16	1025	355.3	61.6	16.23376623
gemma-2-9b-it	INT4-MIXED	32	83.8	61.7	16.20745543
stable-diffusion-xl-1.0-inpainting-0.1	FP16	32	62.6	62	16.12903226
gemma-2-9b-it	INT4-MIXED	1024	713.8	62.2	16.07717042
gemma-2b-it	FP16	32	66.1	62.4	16.02564103
lcm-dreamshaper-v7	INT8-HYBRID	32	64.5	62.9	15.89825119
lcm-dreamshaper-v7	INT8-HYBRID	1024	64.1	63	15.87301587
gemma-2b-it	FP16	1024	258.6	63.2	15.82278481
phi-2	FP16	1024	431.8	63.2	15.82278481
stable-zephyr-3b-dpo	FP16	1024	412	63.6	15.72327044
stablelm-3b-4e1t	FP16	1024	417.3	64	15.625
qwen-7b-chat	INT4-MIXED	1024	577.1	64.3	15.55209953
gemma-2-9b-it	INT4-MIXED	1024	766.9	64.6	15.47987616
gemma-2-9b-it	INT4-MIXED	1024	816.5	65.1	15.3609831
qwen3_8b_draft_eagle3	INT8-CW	1024	642.4	66.1	15.12859304
gemma-3n-e2b-it	FP16	275	567.1	66.2	15.10574018
gemma-3n-e2b-it	FP16	1025	1029.1	66.7	14.99250375
llama-3.2-3b-instruct	FP16	32	73.6	68.7	14.55604076
qwen3_8b_draft_eagle3	INT8-CW	32	104.3	69.4	14.4092219
llama-3.2-3b-instruct	FP16	1024	432.7	70.8	14.12429379
dolly-v2-12b	INT4-MIXED	32	114.2	71.2	14.04494382
gemma-3-12b-it	INT4-MIXED	32	105.9	73.3	13.6425648
llama-2-13b-chat-hf	INT4-MIXED	32	108.1	73.3	13.6425648
qwen2.5-coder-3b-instruct	FP16	32	78	73.9	13.53179973
smollm3-3b	FP16	32	78.5	74.2	13.47708895
dolly-v2-12b	INT4-MIXED	1024	1262.2	75.4	13.26259947
gemma-4-12b-it	INT4-MIXED	278	557.8	75.4	13.26259947
qwen2.5-coder-3b-instruct	FP16	1024	320.8	75.6	13.22751323
smollm3-3b	FP16	1024	323.3	75.7	13.21003963
qwen2.5-7b-instruct	INT8-CW	32	97.7	76.4	13.08900524
qwen2.5-7b-instruct-1m	INT8-CW	32	97.9	76.5	13.07189542
deepseek-r1-distill-qwen-7b	INT8-CW	32	98.6	76.6	13.05483029
gemma-3-12b-it	INT4-MIXED	1024	966.5	76.6	13.05483029
qwen2-7b-instruct	INT8-CW	32	98	76.6	13.05483029
qwen2.5-7b-instruct	INT8-CW	32	98.4	76.6	13.05483029
falcon-7b-instruct	INT8-CW	32	93.4	76.7	13.03780965
fara-7b	INT8-CW	32	141.2	76.7	13.03780965
lcm-dreamshaper-v7	INT8-CW	32	77.9	76.7	13.03780965
minicpm-o-2_6	INT8-CW	238	535.7	76.7	13.03780965
minicpm-v-2_6	INT8-CW	228	563.6	76.7	13.03780965
lcm-dreamshaper-v7	INT8-CW	1024	77.6	77	12.98701299
qwen2.5-vl-7b-instruct	INT8-CW	32	102.4	77.1	12.97016861
gemma-3-12b-it	INT4-MIXED	32	117.3	77.2	12.95336788
lcm-dreamshaper-v7	FP16	32	78.1	77.4	12.91989664
granite-4.0-h-micro	FP16	33	126.2	77.5	12.90322581
lcm-dreamshaper-v7	FP16	1024	78.1	77.6	12.88659794
lcm-dreamshaper-v7	INT8-CW	1024	78.3	77.6	12.88659794
lcm-dreamshaper-v7	INT8-CW	32	78.3	77.8	12.85347044
llama-2-13b-chat-hf	INT4-MIXED	1024	967.7	77.8	12.85347044
granite-4.0-h-micro	FP16	1033	879.9	77.9	12.83697047
gemma-4-12b-it	INT4-MIXED	1024	1132.6	78	12.82051282
llama-2-7b-chat-hf	INT8-CW	32	91.1	78	12.82051282
codellama-7b	INT8-CW	32	90.5	78.1	12.80409731
deepseek-r1-distill-qwen-7b	INT8-CW	1024	583.9	78.3	12.77139208
qwen2-7b-instruct	INT8-CW	1024	586.2	78.3	12.77139208
qwen2.5-7b-instruct	INT8-CW	1024	588.7	78.3	12.77139208
qwen2.5-7b-instruct-1m	INT8-CW	1024	592.5	78.3	12.77139208
qwen2.5-7b-instruct	INT8-CW	1024	597.5	78.4	12.75510204
fara-7b	INT8-CW	1024	756.6	78.6	12.72264631
falcon-7b-instruct	INT8-CW	1024	829.3	78.7	12.7064803
qwen2.5-vl-7b-instruct	INT8-CW	1024	621.1	78.7	12.7064803
phi-3-mini-4k-instruct	FP16	32	86	79.7	12.54705144
phi-3.5-mini-instruct	FP16	32	85.9	79.9	12.51564456
phi-3-mini-128k-instruct	FP16	32	86.4	80	12.5
gemma-3-12b-it	INT4-MIXED	1024	1098.8	80.2	12.46882793
llama-2-7b-chat-hf	INT8-CW	1024	578.7	81.8	12.22493888
qwen3-vl-4b-instruct_draft_eagle3	INT4-MIXED	4907	1103	82.1	12.18026797
phi-4-mini-instruct	FP16	32	90.5	82.2	12.16545012
codellama-7b	INT8-CW	1024	589.4	82.3	12.15066829
neural-chat-7b-v3-3	INT8-CW	32	97.2	82.3	12.15066829
phi-4	INT4-MIXED	32	124	82.3	12.15066829
phi-4-mini-reasoning	FP16	32	90.6	82.3	12.15066829
phi-4-reasoning	INT4-MIXED	32	123.8	82.4	12.13592233
qwen3-vl-4b-instruct_draft_eagle3	INT4-MIXED	4937	1153.2	82.7	12.09189843
baichuan2-7b-chat	INT8-CW	32	97.2	82.8	12.07729469
biomistral-7b-slerp	INT8-CW	7	86.4	83	12.04819277
phi-3.5-vision-instruct	FP16	802	298.8	83.3	12.00480192
qwen-7b-chat	INT8-CW	32	96.5	83.3	12.00480192
gemma-4-12b-it	INT4-MIXED	278	612.6	83.5	11.9760479
mistral-7b-instruct-v0.2	INT8-CW	32	96.1	83.5	11.9760479
mistral-7b-instruct-v0.3	INT8-CW	32	96	83.5	11.9760479
zephyr-7b-beta	INT8-CW	32	96.2	83.5	11.9760479
phi-3-mini-4k-instruct	FP16	1024	610.8	84.1	11.89060642
phi-3.5-mini-instruct	FP16	1024	607.8	84.1	11.89060642
mistral-7b-instruct-v0.1	INT8-CW	32	97.2	84.2	11.87648456
phi-3-mini-128k-instruct	FP16	1024	609.4	84.3	11.8623962
gemma-3-4b-it	FP16	32	93.6	84.4	11.84834123
phi-3.5-vision-instruct	FP16	1032	406.5	84.4	11.84834123
phi-4-mini-instruct	FP16	1024	581.7	84.4	11.84834123
phi-4-mini-reasoning	FP16	1024	593.9	84.5	11.83431953
qwen1.5-14b-chat	INT4-MIXED	32	122	84.5	11.83431953
neural-chat-7b-v3-3	INT8-CW	1024	620.7	85	11.76470588
bloomz-7b1	INT8-CW	32	96.8	85.3	11.72332943
phi-4	INT4-MIXED	1024	1099.2	85.6	11.68224299
phi-4-reasoning	INT4-MIXED	1024	1079.2	85.7	11.66861144
gemma-4-12b-it	INT4-MIXED	1024	1362.9	85.9	11.64144354
mistral-7b-instruct-v0.2	INT8-CW	1024	608.4	85.9	11.64144354
mistral-7b-instruct-v0.3	INT8-CW	1024	611.2	85.9	11.64144354
zephyr-7b-beta	INT8-CW	1024	605.9	86	11.62790698
gemma-3-4b-it	FP16	1024	509.9	86.2	11.60092807
phi-4-reasoning	INT4-MIXED	32	129.3	86.2	11.60092807
baichuan2-7b-chat	INT8-CW	1024	647.1	86.6	11.54734411
llama-3.1-8b-instruct	INT8-CW	32	100.8	86.7	11.53402537
mistral-7b-instruct-v0.1	INT8-CW	1025	648.7	86.9	11.50747986
deepseek-r1-distill-llama-8b	INT8-CW	35	114.1	87	11.49425287
phi-4	INT4-MIXED	32	130.5	87.1	11.48105626
qwen-7b-chat	INT8-CW	1024	613.5	87.1	11.48105626
llama-3-8b-instruct	INT8-CW	32	101.4	87.2	11.46788991
deepseek-r1-distill-qwen-14b	INT4-MIXED	32	131.5	88.3	11.32502831
qwen3-4b	FP16	32	94.9	88.5	11.29943503
bloomz-7b1	INT8-CW	1024	826.1	88.8	11.26126126
qwen1.5-14b-chat	INT4-MIXED	1024	1140.1	89	11.23595506
llama-3.1-8b-instruct	INT8-CW	1024	614.2	89.1	11.22334456
videochat-flash-qwen2_5-7b_internvideo2-1b	INT8-CW	14531	50454.6	89.2	11.21076233
phi-4-reasoning	INT4-MIXED	1024	1149	89.6	11.16071429
deepseek-r1-distill-llama-8b	INT8-CW	1135	679.6	89.7	11.14827202
llama-3-8b-instruct	INT8-CW	1024	617	89.7	11.14827202
llava-next-video-7b-hf	INT8-CW	2945	555.3	89.8	11.13585746
llava-v1.6-mistral-7b-hf	INT8-CW	2944	771.1	90.2	11.0864745
phi-4	INT4-MIXED	1024	1302.3	90.4	11.0619469
phi-4-reasoning	INT4-MIXED	32	138.6	91.2	10.96491228
qwen3-4b	FP16	1024	509.7	91.4	10.94091904
qwen3-8b	INT8-CW	32	103.3	91.4	10.94091904
llama-2-13b-chat-hf	INT4-MIXED	32	137.2	91.8	10.89324619
afm-4.5b	FP16	32	100.7	92	10.86956522
deepseek-r1-distill-qwen-14b	INT4-MIXED	1024	1205.9	92.1	10.8577633
minicpm-v-4_5	INT8-CW	217	512.8	93.3	10.71811361
starcoder	INT4-MIXED	32	152.1	93.9	10.64962726
qwen3-8b	INT8-CW	1024	652.9	94.4	10.59322034
phi-4-reasoning	INT4-MIXED	1024	1369.5	94.5	10.58201058
afm-4.5b	FP16	1024	540.1	94.9	10.5374078
minicpm3-4b	FP16	32	112.6	95	10.52631579
phi-4-multimodal-instruct	FP16	578	647.8	95.3	10.49317943
phi-4-multimodal-instruct	FP16	786	454.6	95.5	10.47120419
gemma-7b-it	INT8-CW	32	117.9	95.8	10.43841336
qwen3.5-9b	INT8-CW	83	188	96.1	10.40582726
minicpm4-8b	INT8-CW	32	108.8	96.3	10.38421599
llama-2-13b-chat-hf	INT4-MIXED	1024	1158.3	96.6	10.35196687
phi-4-multimodal-instruct	FP16	1362	1504.3	96.8	10.33057851
starcoder	INT4-MIXED	1024	1436.3	97	10.30927835
qwen3.5-9b	INT8-CW	1024	849.2	97.1	10.29866117
phi-4-multimodal-instruct	FP16	1570	786.5	97.2	10.28806584
minicpm4-8b	INT8-CW	1024	700.2	98.5	10.15228426
gemma-7b-it	INT8-CW	1024	778.8	99.9	10.01001001
qwen3-vl-4b-instruct	FP16	4907	1328.9	100	10

5-330

Topology	Precision	Input Size	1st latency (ms)	2nd latency (ms)	2nd token per sec
minicpm4-0.5b	INT4-MIXED	32	21.1	7.3	136.9863014
minicpm4-0.5b	INT4-MIXED	32	20.9	7.5	133.3333333
gemma-3-270m	INT4-MIXED	1024	64.9	7.6	131.5789474
gemma-3-270m	INT4-MIXED	32	21	7.6	131.5789474
minicpm4-0.5b	INT4-MIXED	32	21.9	7.6	131.5789474
lfm2.5-350m	INT4-MIXED	32	22.8	7.9	126.5822785
gemma-3-270m	INT8-CW	1024	60.5	8	125
gemma-3-270m	INT8-CW	32	21.3	8	125
lfm2.5-350m	INT4-MIXED	1024	153.9	8.2	121.9512195
minicpm4-0.5b	INT4-MIXED	1024	98.9	8.4	119.047619
minicpm4-0.5b	INT4-MIXED	1024	101.8	8.5	117.6470588
minicpm4-0.5b	INT4-MIXED	1024	127.3	8.7	114.9425287
nuextract-tiny	INT4-MIXED	32	21.8	8.9	112.3595506
qwen2.5-coder-0.5b-instruct	INT4-MIXED	32	21	8.9	112.3595506
qwen2.5-coder-0.5b-instruct	INT4-MIXED	32	21.9	9	111.1111111
qwen2.5-coder-0.5b-instruct	INT4-MIXED	32	21.9	9.2	108.6956522
lfm2.5-350m	INT8-CW	32	26	9.7	103.0927835
qwen2.5-coder-0.5b-instruct	INT4-MIXED	1024	104.3	9.8	102.0408163
nuextract-tiny	INT4-MIXED	1024	122.2	9.9	101.010101
lfm2.5-350m	INT8-CW	1024	140.1	10	100
qwen2.5-coder-0.5b-instruct	INT4-MIXED	1024	106.9	10	100
qwen2.5-coder-0.5b-instruct	INT4-MIXED	1024	134.2	10.2	98.03921569
minicpm4-0.5b	INT8-CW	32	23	10.4	96.15384615
whisper-small	INT4-MIXED	prompt0	133.3	10.4	96.15384615
nanollava	INT4-MIXED	760	151.9	10.5	95.23809524
whisper-small	INT4-MIXED	prompt1	199.1	10.6	94.33962264
whisper-small	INT4-MIXED	prompt1	186.8	10.8	92.59259259
whisper-small	INT4-MIXED	prompt0	134.7	11	90.90909091
nuextract-tiny	INT8-CW	32	22.9	11.1	90.09009009
whisper-small	INT4-MIXED	prompt1	201.6	11.3	88.49557522
whisper-small	INT8-CW	prompt0	141.4	11.3	88.49557522
distil-large-v2	INT8-CW	prompt0	387.7	11.4	87.71929825
distil-large-v2	INT8-CW	prompt1	440	11.4	87.71929825
whisper-small	INT4-MIXED	prompt0	149.1	11.4	87.71929825
whisper-small	INT8-CW	prompt1	197.9	11.4	87.71929825
distil-large-v2	INT4-MIXED	prompt0	430	11.5	86.95652174
minicpm4-0.5b	INT8-CW	1024	109.9	11.5	86.95652174
distil-large-v2	INT4-MIXED	prompt1	491.6	11.7	85.47008547
nanollava	INT4-MIXED	1752	317.7	11.7	85.47008547
qwen2.5-coder-0.5b-instruct	INT8-CW	32	23.5	11.9	84.03361345
whisper-large-v3-turbo	INT8-CW	prompt1	464.5	12.1	82.6446281
gemma-3-270m	FP16	32	21.4	12.2	81.96721311
gemma-3-270m	FP16	1024	69.7	12.3	81.30081301
nanollava	INT8-CW	760	154.2	12.3	81.30081301
whisper-large-v3-turbo	INT8-CW	prompt0	403.4	12.4	80.64516129
nuextract-tiny	INT8-CW	1024	104.3	12.5	80
whisper-large-v3-turbo	INT4-MIXED	prompt1	511.1	12.6	79.36507937
qwen2.5-coder-0.5b-instruct	INT8-CW	1024	116.5	12.7	78.74015748
whisper-large-v3-turbo	INT4-MIXED	prompt0	441.9	12.7	78.74015748
tiny-llama-1.1b-chat	INT4-MIXED	32	29.2	13	76.92307692
nanollava	INT8-CW	1752	292.7	13.9	71.94244604
qwen3.5-0.8b	INT4-MIXED	85	57.1	13.9	71.94244604
minicpm5-1b	INT4-MIXED	32	27.3	14	71.42857143
qwen3.5-0.8b	INT4-MIXED	85	57	14.1	70.92198582
qwen3.5-0.8b	INT4-MIXED	85	63.2	14.3	69.93006993
qwen3-asr-0.6b	INT4-MIXED	prompt0	225.2	14.4	69.44444444
qwen3-asr-0.6b	INT4-MIXED	prompt0	224.7	14.4	69.44444444
qwen3.5-0.8b	INT4-MIXED	1024	337	14.4	69.44444444
whisper-small	FP16	prompt1	197.8	14.5	68.96551724
qwen3-asr-0.6b	INT4-MIXED	prompt0	235.7	14.6	68.49315068
whisper-small	FP16	prompt0	138	14.6	68.49315068
qwen3.5-0.8b	INT4-MIXED	1024	338.1	14.8	67.56756757
tiny-llama-1.1b-chat	INT4-MIXED	1024	217	14.8	67.56756757
qwen3.5-0.8b	INT4-MIXED	1024	390.3	15	66.66666667
qwen3-asr-0.6b	INT4-MIXED	prompt1	680.7	15.3	65.35947712
lfm2.5-1.2b-instruct	INT4-MIXED	32	36.1	15.4	64.93506494
qwen3-asr-0.6b	INT4-MIXED	prompt1	613.6	15.4	64.93506494
lfm2-1.2b	INT4-MIXED	32	35.4	15.6	64.1025641
tiny-llama-1.1b-chat	INT4-MIXED	32	33	15.7	63.69426752
distil-large-v2	FP16	prompt0	433.3	15.8	63.29113924
distil-large-v2	FP16	prompt1	487.8	15.8	63.29113924
lfm2-1.2b	INT4-MIXED	1024	319.7	15.9	62.89308176
lfm2-1.2b	INT4-MIXED	32	40.6	15.9	62.89308176
lfm2.5-1.2b-instruct	INT4-MIXED	1024	362	15.9	62.89308176
lfm2.5-1.2b-instruct	INT4-MIXED	32	40.5	15.9	62.89308176
qwen3-asr-0.6b	INT8-CW	prompt0	231.8	15.9	62.89308176
minicpm5-1b	INT4-MIXED	1024	222.8	16	62.5
qwen3-asr-0.6b	INT4-MIXED	prompt1	603.5	16	62.5
lfm2.5-1.2b-instruct	INT4-MIXED	32	40.6	16.1	62.11180124
lfm2.5-350m	FP16	32	26.3	16.1	62.11180124
lfm2-1.2b	INT4-MIXED	32	40.8	16.2	61.72839506
lfm2-1.2b	INT4-MIXED	1024	361.7	16.3	61.34969325
llama-3.2-1b-instruct	INT4-MIXED	32	32.4	16.3	61.34969325
whisper-large-v3-turbo	FP16	prompt0	449	16.3	61.34969325
whisper-large-v3-turbo	FP16	prompt1	514.2	16.3	61.34969325
lfm2.5-1.2b-instruct	INT4-MIXED	1024	324.9	16.4	60.97560976
lfm2.5-350m	FP16	1024	159.9	16.4	60.97560976
llama-3.2-1b-instruct	INT4-MIXED	32	32.9	16.5	60.60606061
lfm2-1.2b	INT4-MIXED	1024	424.6	16.6	60.24096386
lfm2.5-1.2b-instruct	INT4-MIXED	1024	399	16.6	60.24096386
gemma-3-1b-it	INT4-MIXED	32	30.6	17	58.82352941
exaone-4.0-1.2b	INT4-MIXED	32	32.9	17.3	57.80346821
gemma-3-1b-it	INT4-MIXED	1024	200.3	17.4	57.47126437
tiny-llama-1.1b-chat	INT4-MIXED	1024	295.1	17.5	57.14285714
llama-3.2-1b-instruct	INT4-MIXED	1024	209.5	17.7	56.49717514
exaone-4.0-1.2b	INT4-MIXED	32	37.1	17.8	56.17977528
gemma-3-1b-it	INT4-MIXED	32	33	17.8	56.17977528
exaone-4.0-1.2b	INT4-MIXED	32	37.7	18	55.55555556
qwen3.5-0.8b	INT8-CW	85	64.8	18	55.55555556
gemma-3-1b-it	INT4-MIXED	32	33.6	18.1	55.24861878
llama-3.2-1b-instruct	INT4-MIXED	1024	254.3	18.1	55.24861878
qwen3-asr-0.6b	INT8-CW	prompt1	632.6	18.2	54.94505495
gemma-3-1b-it	INT4-MIXED	1024	205	18.3	54.64480874
gemma-3-1b-it	INT4-MIXED	1024	242.4	18.5	54.05405405
qwen3.5-0.8b	INT8-CW	1024	347.7	18.8	53.19148936
minicpm4-0.5b	FP16	32	23.7	19.3	51.8134715
minicpm5-1b	INT8-CW	32	34.9	19.3	51.8134715
nuextract-tiny	FP16	32	25	19.3	51.8134715
minicpm-1b-sft	INT4-MIXED	31	43.7	19.6	51.02040816
exaone-4.0-1.2b	INT4-MIXED	1024	330.3	19.8	50.50505051
exaone-4.0-1.2b	INT4-MIXED	1024	341	20.3	49.26108374
minicpm4-0.5b	FP16	1024	133.3	20.3	49.26108374
exaone-4.0-1.2b	INT4-MIXED	1024	371.8	20.5	48.7804878
minicpm-1b-sft	INT4-MIXED	31	45.3	20.5	48.7804878
minicpm5-1b	INT8-CW	1024	201.4	20.7	48.30917874
qwen2.5-coder-1.5b-instruct	INT4-MIXED	32	39.5	20.8	48.07692308
minicpm-1b-sft	INT4-MIXED	31	46.2	20.9	47.84688995
deepseek-r1-distill-qwen-1.5b	INT4-MIXED	32	42.2	21.1	47.39336493
nanollava	FP16	760	186	21.3	46.94835681
nuextract-tiny	FP16	1024	128.8	21.4	46.72897196
tiny-llama-1.1b-chat	INT8-CW	32	38.4	21.4	46.72897196
qwen2.5-coder-1.5b-instruct	INT4-MIXED	32	44.5	21.5	46.51162791
qwen2.5-1.5b-instruct	INT4-MIXED	32	43.9	21.6	46.2962963
qwen2.5-coder-0.5b-instruct	FP16	32	26.7	21.8	45.87155963
qwen2.5-coder-1.5b-instruct	INT4-MIXED	32	44.8	21.9	45.66210046
qwen2.5-coder-0.5b-instruct	FP16	1024	143.1	22.4	44.64285714
qwen2.5-coder-1.5b-instruct	INT4-MIXED	1024	283.4	22.6	44.24778761
minicpm-1b-sft	INT4-MIXED	1014	316.1	22.7	44.05286344
gemma-3-1b-it	INT8-CW	32	39.2	22.8	43.85964912
deepseek-r1-distill-qwen-1.5b	INT4-MIXED	1024	343.3	22.9	43.66812227
gemma-3-1b-it	INT8-CW	1024	230.1	23.3	42.91845494
qwen2.5-coder-1.5b-instruct	INT4-MIXED	1024	342.1	23.3	42.91845494
tiny-llama-1.1b-chat	INT8-CW	1024	237.9	23.3	42.91845494
nanollava	FP16	1752	329.8	23.4	42.73504274
qwen2.5-1.5b-instruct	INT4-MIXED	32	47.3	23.4	42.73504274
qwen2.5-1.5b-instruct	INT4-MIXED	1024	343.5	23.4	42.73504274
minicpm-1b-sft	INT4-MIXED	1014	326.8	23.7	42.19409283
qwen2.5-coder-1.5b-instruct	INT4-MIXED	1024	358.6	23.8	42.01680672
minicpm-1b-sft	INT4-MIXED	1014	380.9	24.1	41.49377593
lfm2-8b-a1b	INT4-MIXED	32	382.4	24.6	40.6504065
lfm2.5-1.2b-instruct	INT4-MIXED	32	48.1	25	40
lfm2-1.2b	INT4-MIXED	32	48.9	25.1	39.84063745
lfm2-8b-a1b	INT4-MIXED	1024	895.3	25.1	39.84063745
llama-3.2-1b-instruct	INT8-CW	32	43.3	25.1	39.84063745
qwen2.5-1.5b-instruct	INT4-MIXED	1024	367	25.3	39.5256917
lfm2-1.2b	INT8-CW	32	50.9	25.5	39.21568627
lfm2.5-1.2b-instruct	INT8-CW	32	53.2	25.5	39.21568627
lfm2.5-1.2b-instruct	INT4-MIXED	1024	344	25.6	39.0625
lfm2-1.2b	INT4-MIXED	1024	366.2	25.7	38.91050584
lfm2-1.2b	INT8-CW	1024	361.5	26	38.46153846
lfm2.5-1.2b-instruct	INT8-CW	1024	368.4	26	38.46153846
qwen3-asr-0.6b	FP16	prompt0	269.1	26.3	38.02281369
llama-3.2-1b-instruct	INT8-CW	1024	237.2	26.7	37.45318352
exaone-4.0-1.2b	INT8-CW	32	47.4	27.1	36.900369
qwen3-asr-0.6b	FP16	prompt1	709.2	29.1	34.36426117
exaone-4.0-1.2b	INT8-CW	1024	356.5	29.7	33.67003367
deepseek-r1-distill-qwen-1.5b	INT4-MIXED	32	59.5	30.1	33.22259136
qwen3.5-0.8b	FP16	85	71.3	31.2	32.05128205
minicpm-1b-sft	INT8-CW	31	59.6	31.5	31.74603175
qwen3.5-0.8b	FP16	1024	387.3	31.7	31.54574132
deepseek-r1-distill-qwen-1.5b	INT4-MIXED	1024	347.4	31.8	31.44654088
deepseek-r1-distill-qwen-1.5b	INT8-CW	32	58.9	32.5	30.76923077
qwen2.5-1.5b-instruct	INT8-CW	32	61.5	33	30.3030303
qwen2.5-coder-1.5b-instruct	INT8-CW	32	61.4	33.1	30.21148036
phi-2	INT4-MIXED	32	66	33.3	30.03003003
stable-zephyr-3b-dpo	INT4-MIXED	32	62.4	33.7	29.6735905
deepseek-r1-distill-qwen-1.5b	INT8-CW	1024	318.9	33.8	29.58579882
stablelm-3b-4e1t	INT4-MIXED	32	63.1	34	29.41176471
gemma-2b-it	INT4-MIXED	32	55.6	34.1	29.3255132
gemma-4-e2b-it	INT4-MIXED	274	601.8	34.1	29.3255132
qwen2.5-coder-1.5b-instruct	INT8-CW	1024	332.8	34.4	29.06976744
minicpm-1b-sft	INT8-CW	1014	357.1	34.5	28.98550725
qwen2.5-1.5b-instruct	INT8-CW	1024	333.7	34.5	28.98550725
gemma-2b-it	INT4-MIXED	32	65.4	35.1	28.49002849
gemma-4-e2b-it	INT4-MIXED	1024	962.2	35.3	28.3286119
gemma-2b-it	INT4-MIXED	1024	381.6	35.6	28.08988764
minicpm5-1b	FP16	32	43	36.1	27.70083102
whisper-large-v3	INT4-MIXED	prompt0	566.6	36.2	27.62430939
gemma-2-2b	INT4-MIXED	33	57.1	36.4	27.47252747
gemma-2b-it	INT4-MIXED	1024	477.5	36.6	27.32240437
minicpm5-1b	FP16	1024	256.5	37.6	26.59574468
stable-zephyr-3b-dpo	INT4-MIXED	32	78.2	37.7	26.52519894
whisper-large-v3	INT4-MIXED	prompt1	620.4	37.8	26.45502646
phi-2	INT4-MIXED	1024	701.8	37.9	26.38522427
gemma-4-e2b-it	INT4-MIXED	274	651.3	38.3	26.10966057
stable-zephyr-3b-dpo	INT4-MIXED	1024	785.2	38.3	26.10966057
stablelm-3b-4e1t	INT4-MIXED	1024	788.4	38.3	26.10966057
gemma-2-2b	INT4-MIXED	33	58.4	38.7	25.83979328
gemma-2-2b	INT4-MIXED	1025	514.4	38.7	25.83979328
gemma-4-e2b-it	INT4-MIXED	1024	1107.4	39.4	25.38071066
smollm3-3b	INT4-MIXED	32	69.6	39.4	25.38071066
gemma-3n-e2b-it	INT4-MIXED	275	1043.8	39.5	25.3164557
phi-2	INT4-MIXED	32	84.6	39.6	25.25252525
llama-3.2-3b-instruct	INT4-MIXED	32	69.3	39.9	25.06265664
gemma-3n-e2b-it	INT4-MIXED	1025	2136.4	40	25
qwen2.5-coder-3b-instruct	INT4-MIXED	32	69.9	40.2	24.87562189
smollm3-3b	INT4-MIXED	32	80.8	40.4	24.75247525
tiny-llama-1.1b-chat	FP16	32	49.1	40.7	24.57002457
gemma-3n-e2b-it	INT4-MIXED	275	1034.1	40.8	24.50980392
gemma-2-2b	INT4-MIXED	1025	587.9	41	24.3902439
granite-4.0-h-micro	INT4-MIXED	33	109.1	41	24.3902439
smollm3-3b	INT4-MIXED	32	83.3	41	24.3902439
llama-3.2-3b-instruct	INT4-MIXED	32	80.6	41.1	24.33090024
qwen2.5-coder-3b-instruct	INT4-MIXED	32	81.2	41.1	24.33090024
granite-4.0-h-micro	INT4-MIXED	1033	1903.9	41.4	24.15458937
llama-3.2-3b-instruct	INT4-MIXED	32	81.3	41.4	24.15458937
gemma-3n-e2b-it	INT4-MIXED	1025	2186.8	41.6	24.03846154
gemma-3-1b-it	FP16	32	51.2	42.2	23.69668246
tiny-llama-1.1b-chat	FP16	1024	319.5	42.2	23.69668246
whisper-large-v3	INT8-CW	prompt0	514	42.2	23.69668246
stable-zephyr-3b-dpo	INT4-MIXED	1024	871	42.3	23.64066194
granite-4.0-h-micro	INT4-MIXED	33	110.1	42.4	23.58490566
gemma-3-1b-it	FP16	1024	292.3	42.6	23.4741784
smollm3-3b	INT4-MIXED	1024	529.5	42.6	23.4741784
stablelm-3b-4e1t	INT4-MIXED	32	88.4	42.6	23.4741784
granite-4.0-h-micro	INT4-MIXED	33	110.6	42.7	23.41920375
granite-4.0-h-micro	INT4-MIXED	1033	1971.2	42.9	23.31002331
whisper-large-v3	INT8-CW	prompt1	571.1	42.9	23.31002331
granite-4.0-h-micro	INT4-MIXED	1033	2047.2	43.1	23.20185615
qwen2.5-coder-3b-instruct	INT4-MIXED	1024	504.2	43.4	23.04147465
phi-3-mini-128k-instruct	INT4-MIXED	32	81	43.5	22.98850575
phi-3.5-mini-instruct	INT4-MIXED	32	80	43.5	22.98850575
phi-3-mini-4k-instruct	INT4-MIXED	32	80.4	43.6	22.93577982
smollm3-3b	INT4-MIXED	1024	566.2	43.6	22.93577982
llama-3.2-3b-instruct	INT4-MIXED	1024	550.4	43.7	22.88329519
smollm3-3b	INT4-MIXED	1024	671.5	44	22.72727273
phi-2	INT4-MIXED	1024	951.2	44.3	22.57336343
qwen2.5-coder-3b-instruct	INT4-MIXED	1024	528	44.3	22.57336343
llama-3.2-3b-instruct	INT4-MIXED	1024	592.9	44.7	22.37136465
phi-3-mini-4k-instruct	INT4-MIXED	32	96.8	44.8	22.32142857
llama-3.2-3b-instruct	INT4-MIXED	1024	677.2	44.9	22.27171492
phi-3.5-mini-instruct	INT4-MIXED	32	95.8	45	22.22222222
phi-3-mini-128k-instruct	INT4-MIXED	32	100.8	45.6	21.92982456
qwen2.5-3b-instruct-gptq	INT4-MIXED	32	87.4	46.7	21.41327623
stablelm-3b-4e1t	INT4-MIXED	1024	893	47.2	21.18644068
lfm2-1.2b	FP16	32	63.8	47.8	20.92050209
qwen2.5-coder-3b-instruct	INT4-MIXED	32	95.4	48	20.83333333
lfm2.5-1.2b-instruct	FP16	32	64.4	48.1	20.79002079
lfm2-1.2b	FP16	1024	471.9	48.3	20.70393375
phi-3-mini-128k-instruct	INT4-MIXED	1024	863.5	48.4	20.66115702
phi-3.5-mini-instruct	INT4-MIXED	1024	867.6	48.4	20.66115702
llama-3.2-1b-instruct	FP16	32	57.5	48.5	20.6185567
phi-3-mini-4k-instruct	INT4-MIXED	1024	861.1	48.5	20.6185567
lfm2.5-1.2b-instruct	FP16	1024	534.2	48.6	20.57613169
qwen2.5-3b-instruct-gptq	INT4-MIXED	1024	552.5	48.9	20.44989775
gemma-3-4b-it	INT4-MIXED	32	85.9	49.6	20.16129032
phi-3-mini-4k-instruct	INT4-MIXED	1024	907.6	49.6	20.16129032
phi-3.5-vision-instruct	INT4-MIXED	802	568.9	49.6	20.16129032
phi-4-mini-instruct	INT4-MIXED	32	88.4	49.8	20.08032129
phi-3.5-mini-instruct	INT4-MIXED	1024	910.6	49.9	20.04008016
phi-4-mini-reasoning	INT4-MIXED	32	85.8	49.9	20.04008016
llama-3.2-1b-instruct	FP16	1024	314.7	50	20
gemma-2b-it	INT8-CW	32	86.3	50.2	19.92031873
phi-3-mini-128k-instruct	INT4-MIXED	1024	1117.1	50.5	19.8019802
qwen3-4b	INT4-MIXED	32	88.9	50.6	19.76284585
phi-3.5-vision-instruct	INT4-MIXED	1032	1729.8	50.7	19.72386588
phi-4-mini-instruct	INT4-MIXED	32	100.5	51	19.60784314
gemma-3-4b-it	INT4-MIXED	32	99.5	51.1	19.56947162
phi-4-mini-reasoning	INT4-MIXED	32	99.3	51.1	19.56947162
qwen2.5-coder-3b-instruct	INT4-MIXED	1024	838.4	51.2	19.53125
gemma-4-e2b-it	INT8-CW	274	681.9	51.3	19.49317739
gemma-3-4b-it	INT4-MIXED	32	101.9	51.5	19.41747573
phi-4-mini-reasoning	INT4-MIXED	32	103.1	51.5	19.41747573
gemma-2b-it	INT8-CW	1024	454.7	51.8	19.30501931
gemma-3-4b-it	INT4-MIXED	1024	747.3	52	19.23076923
qwen3-4b	INT4-MIXED	32	103.9	52.3	19.12045889
gemma-4-e2b-it	INT8-CW	1024	1105.2	52.5	19.04761905
afm-4.5b	INT4-MIXED	32	104.1	52.6	19.01140684
phi-3-mini-4k-instruct	INT4-MIXED	32	114.5	53.1	18.83239171
phi-3.5-mini-instruct	INT4-MIXED	32	117.7	53.2	18.79699248
gemma-3-4b-it	INT4-MIXED	1024	779.9	53.5	18.69158879
gemma-3-4b-it	INT4-MIXED	1024	844.6	53.9	18.5528757
phi-4-mini-instruct	INT4-MIXED	1024	866.8	54	18.51851852
phi-4-mini-reasoning	INT4-MIXED	1024	865.9	54.1	18.48428835
exaone-4.0-1.2b	FP16	32	63.6	55.1	18.14882033
phi-4-mini-reasoning	INT4-MIXED	1024	914.8	55.2	18.11594203
phi-4-mini-instruct	INT4-MIXED	1024	909.2	55.3	18.08318264
phi-4-mini-reasoning	INT4-MIXED	1024	979.9	55.6	17.98561151
gemma-2-2b	INT8-CW	33	104.5	55.7	17.95332136
phi-2	INT8-CW	32	106.5	55.7	17.95332136
minicpm3-4b	INT4-MIXED	32	103.7	56	17.85714286
afm-4.5b	INT4-MIXED	1024	886.7	56.3	17.76198934
stable-zephyr-3b-dpo	INT8-CW	32	104.8	56.3	17.76198934
qwen3-4b	INT4-MIXED	1024	793.2	56.7	17.6366843
minicpm-1b-sft	FP16	31	68.7	56.8	17.6056338
exaone-4.0-1.2b	FP16	1024	446.7	57.8	17.30103806
phi-3-mini-4k-instruct	INT4-MIXED	1024	1240.5	57.9	17.27115717
minicpm3-4b	INT4-MIXED	32	122.3	58	17.24137931
phi-3.5-mini-instruct	INT4-MIXED	1024	1352.4	58.1	17.21170396
phi-4-mini-instruct	INT4-MIXED	32	117	58.2	17.18213058
qwen3-4b	INT4-MIXED	1024	806.3	58.3	17.15265866
gemma-2-2b	INT8-CW	1025	644	58.6	17.06484642
minicpm3-4b	INT4-MIXED	32	123	58.6	17.06484642
stablelm-3b-4e1t	INT8-CW	32	105.3	59	16.94915254
qwen3-vl-4b-instruct	INT4-MIXED	4907	2389.1	59.5	16.80672269
qwen3-vl-4b-instruct	INT4-MIXED	4937	2470.6	59.5	16.80672269
gemma-3n-e2b-it	INT8-CW	275	1103.4	60	16.66666667
minicpm-1b-sft	FP16	1014	458.6	60.2	16.61129568
phi-2	INT8-CW	1024	806.7	60.3	16.58374793
gemma-3n-e2b-it	INT8-CW	1025	2213.8	60.8	16.44736842
stable-zephyr-3b-dpo	INT8-CW	1024	886.4	61	16.39344262
qwen3-vl-4b-instruct	INT4-MIXED	4907	2380.8	61.1	16.36661211
qwen3-vl-4b-instruct	INT4-MIXED	4937	2462.7	61.2	16.33986928
deepseek-r1-distill-qwen-1.5b	FP16	32	74.1	61.3	16.3132137
qwen2.5-coder-3b-instruct	INT8-CW	32	111.6	61.6	16.23376623
qwen3-vl-4b-instruct	INT4-MIXED	4907	2376	61.6	16.23376623
qwen3-vl-4b-instruct	INT4-MIXED	4937	2464.4	61.6	16.23376623
qwen2.5-coder-1.5b-instruct	FP16	32	73.6	61.7	16.20745543
smollm3-3b	INT8-CW	32	113.6	61.7	16.20745543
qwen2.5-1.5b-instruct	FP16	32	73.9	61.8	16.18122977
phi-4-mini-instruct	INT4-MIXED	1024	1205.9	62.3	16.05136437
deepseek-r1-distill-qwen-1.5b	FP16	1024	438.4	62.7	15.94896332
phi-4-multimodal-instruct	INT4-MIXED	578	983.4	63.1	15.84786054
qwen2.5-1.5b-instruct	FP16	1024	438.3	63.2	15.82278481
qwen2.5-coder-1.5b-instruct	FP16	1024	440.5	63.2	15.82278481
stablelm-3b-4e1t	INT8-CW	1024	903.2	63.8	15.67398119
qwen2.5-coder-3b-instruct	INT8-CW	1024	620.7	64	15.625
smollm3-3b	INT8-CW	1024	634.2	64	15.625
whisper-large-v3	FP16	prompt0	581.1	64	15.625
phi-4-multimodal-instruct	INT4-MIXED	786	691.5	64.1	15.60062402
whisper-large-v3	FP16	prompt1	641.4	64.3	15.55209953
minicpm3-4b	INT4-MIXED	1024	1886.8	64.7	15.45595054
llama-3.2-3b-instruct	INT8-CW	32	114.5	65.6	15.24390244
phi-4-multimodal-instruct	INT4-MIXED	1362	2352.6	66.4	15.06024096
granite-4.0-h-micro	INT8-CW	33	168.6	66.5	15.03759398
minicpm3-4b	INT4-MIXED	1024	1935.9	66.6	15.01501502
granite-4.0-h-micro	INT8-CW	1033	2064.9	67	14.92537313
phi-4-multimodal-instruct	INT4-MIXED	1570	1352.1	67.1	14.90312966
minicpm3-4b	INT4-MIXED	1024	2070	67.3	14.85884101
llama-3.2-3b-instruct	INT8-CW	1024	683.8	68.4	14.61988304
stable-diffusion-xl-1.0-inpainting-0.1	INT8-CW	32	74.1	71.5	13.98601399
codellama-7b	INT4-MIXED	32	123.2	71.9	13.90820584
stable-diffusion-xl-1.0-inpainting-0.1	INT8-CW	32	74.7	72	13.88888889
gemma-3n-e4b-it	INT4-MIXED	275	1100.4	72.6	13.77410468
llama-2-7b-chat-hf	INT4-MIXED	32	126.7	72.7	13.75515818
qwen3_8b_draft_eagle3	INT4-MIXED	32	192.4	73.1	13.67989056
gemma-3n-e4b-it	INT4-MIXED	1025	2665	73.3	13.6425648
biomistral-7b-slerp	INT4-MIXED	7	82.5	73.7	13.56852103
codellama-7b	INT4-MIXED	32	158.9	75.1	13.31557923
falcon-7b-instruct	INT4-MIXED	32	135.6	75.3	13.28021248
llama-2-7b-chat-hf	INT4-MIXED	32	158.8	75.3	13.28021248
gemma-3n-e4b-it	INT4-MIXED	275	1082.4	75.4	13.26259947
neural-chat-7b-v3-3	INT4-MIXED	32	135.7	75.9	13.17523057
gemma-3n-e4b-it	INT4-MIXED	1025	2794.1	76.1	13.14060447
zephyr-7b-beta	INT4-MIXED	32	135.7	76.1	13.14060447
mistral-7b-instruct-v0.2	INT4-MIXED	32	137.2	76.3	13.1061599
phi-3-mini-128k-instruct	INT8-CW	32	143.2	76.3	13.1061599
phi-3.5-mini-instruct	INT8-CW	32	142.7	76.3	13.1061599
mistral-7b-instruct-v0.3	INT4-MIXED	32	135.6	76.7	13.03780965
phi-3-mini-4k-instruct	INT8-CW	32	143.9	76.7	13.03780965
phi-4-multimodal-instruct	INT4-MIXED	578	1168.5	76.7	13.03780965
biomistral-7b-slerp	INT4-MIXED	7	86.3	77.4	12.91989664
phi-4-multimodal-instruct	INT4-MIXED	786	782.1	77.6	12.88659794
codellama-7b	INT4-MIXED	1024	1144.8	77.8	12.85347044
phi-4-mini-instruct	INT8-CW	32	137.9	78.2	12.78772379
phi-4-mini-reasoning	INT8-CW	32	137.8	78.2	12.78772379
llama-2-7b-chat-hf	INT4-MIXED	1024	1128.3	78.7	12.7064803
gemma-3-4b-it	INT8-CW	32	140.6	78.9	12.67427123
mistral-7b-instruct-v0.2	INT4-MIXED	32	169.9	79	12.65822785
mistral-7b-instruct-v0.3	INT4-MIXED	32	169.8	79	12.65822785
neural-chat-7b-v3-3	INT4-MIXED	32	171.3	79.5	12.57861635
mistral-7b-instruct-v0.1	INT4-MIXED	32	173.1	79.8	12.53132832
mistral-7b-instruct-v0.2	INT4-MIXED	32	171.6	79.8	12.53132832
mistral-7b-instruct-v0.3	INT4-MIXED	32	172.2	79.9	12.51564456
phi-4-multimodal-instruct	INT4-MIXED	1362	2769.4	80	12.5
falcon-7b-instruct	INT4-MIXED	1024	1173.1	80.5	12.42236025
qwen2.5-7b-instruct	INT4-MIXED	32	153.7	80.7	12.39157373
qwen2.5-7b-instruct-1m	INT4-MIXED	32	151.5	80.8	12.37623762
phi-4-multimodal-instruct	INT4-MIXED	1570	1453.9	80.9	12.36093943
codellama-7b	INT4-MIXED	1024	1367.4	81.1	12.33045623
llama-2-7b-chat-hf	INT4-MIXED	1024	1230.6	81.3	12.300123
neural-chat-7b-v3-3	INT4-MIXED	1024	1223.2	81.4	12.28501229
phi-4-mini-reasoning	INT8-CW	1024	1014.2	81.5	12.26993865
qwen3-4b	INT8-CW	32	146.9	81.5	12.26993865
qwen2.5-vl-7b-instruct	INT4-MIXED	32	129.5	81.6	12.25490196
deepseek-r1-distill-llama-8b	INT4-MIXED	35	128.4	81.7	12.23990208
deepseek-r1-distill-qwen-7b	INT4-MIXED	32	170.5	81.7	12.23990208
minicpm-v-2_6	INT4-MIXED	228	934.8	81.7	12.23990208
phi-3-mini-128k-instruct	INT8-CW	1024	1155.4	81.7	12.23990208
phi-3.5-mini-instruct	INT8-CW	1024	1149	81.7	12.23990208
phi-3.5-vision-instruct	INT8-CW	802	626.8	81.7	12.23990208
phi-4-mini-instruct	INT8-CW	1024	1012	81.7	12.23990208
zephyr-7b-beta	INT4-MIXED	1024	1221	81.7	12.23990208
minicpm-o-2_6	INT4-MIXED	238	947.9	81.8	12.22493888
phi-3-mini-4k-instruct	INT8-CW	1024	1159.9	81.8	12.22493888
mistral-7b-instruct-v0.2	INT4-MIXED	1024	1220.1	81.9	12.21001221
mistral-7b-instruct-v0.3	INT4-MIXED	1024	1206.3	81.9	12.21001221
gemma-3-4b-it	INT8-CW	1024	881.9	82.1	12.18026797
qwen-7b-chat	INT4-MIXED	32	140.4	82.5	12.12121212
deepseek-r1-distill-qwen-7b	INT4-MIXED	32	174.5	82.7	12.09189843
phi-3.5-vision-instruct	INT8-CW	1032	1541.4	83.1	12.03369434
qwen2-7b-instruct	INT4-MIXED	32	172.3	83.2	12.01923077
qwen2.5-7b-instruct	INT4-MIXED	32	172.5	83.3	12.00480192
llama-3.1-8b-instruct	INT4-MIXED	32	145.8	83.7	11.9474313
llama-3-8b-instruct	INT4-MIXED	32	142.8	83.8	11.93317422
qwen2.5-vl-7b-instruct	INT4-MIXED	32	134.9	84	11.9047619
qwen3_8b_draft_eagle3	INT4-MIXED	1024	1575.3	84	11.9047619
qwen2.5-7b-instruct-1m	INT4-MIXED	32	176.3	84.1	11.89060642
mistral-7b-instruct-v0.3	INT4-MIXED	1024	1314.6	84.2	11.87648456
qwen2-7b-instruct	INT4-MIXED	32	174.7	84.2	11.87648456
qwen2.5-7b-instruct	INT4-MIXED	32	179.5	84.2	11.87648456
deepseek-r1-distill-llama-8b	INT4-MIXED	35	129.4	84.5	11.83431953
mistral-7b-instruct-v0.2	INT4-MIXED	1024	1327.2	84.5	11.83431953
bloomz-7b1	INT4-MIXED	32	140.3	84.6	11.82033097
minicpm-v-2_6	INT4-MIXED	228	938.1	84.7	11.80637544
neural-chat-7b-v3-3	INT4-MIXED	1024	1490.2	84.9	11.77856302
qwen2.5-7b-instruct-1m	INT4-MIXED	1024	1178.3	84.9	11.77856302
mistral-7b-instruct-v0.3	INT4-MIXED	1024	1495.4	85	11.76470588
qwen2.5-7b-instruct	INT4-MIXED	1024	1180.9	85	11.76470588
mistral-7b-instruct-v0.2	INT4-MIXED	1024	1488.2	85.2	11.7370892
qwen-7b-chat	INT4-MIXED	32	172.1	85.3	11.72332943
mistral-7b-instruct-v0.1	INT4-MIXED	1025	1597.5	85.4	11.70960187
qwen2.5-vl-7b-instruct	INT4-MIXED	32	140.5	85.4	11.70960187
minicpm-v-2_6	INT4-MIXED	228	940.8	85.5	11.69590643
qwen2.5-vl-7b-instruct	INT4-MIXED	1024	1222.8	85.6	11.68224299
deepseek-r1-distill-qwen-7b	INT4-MIXED	1024	1248	85.7	11.66861144
qwen3-8b	INT4-MIXED	32	145.4	85.9	11.64144354
minicpm-o-2_6	INT4-MIXED	238	951.7	86	11.62790698
llama-3-8b-instruct	INT4-MIXED	32	175.4	86.1	11.61440186
llama-3-8b-instruct	INT4-MIXED	32	175.1	86.2	11.60092807
fara-7b	INT4-MIXED	32	218.4	86.3	11.58748552
llama-3.1-8b-instruct	INT4-MIXED	32	174.6	86.3	11.58748552
minicpm4-8b	INT4-MIXED	32	152.3	86.5	11.56069364
qwen3-4b	INT8-CW	1024	923.4	86.5	11.56069364
deepseek-r1-distill-qwen-7b	INT4-MIXED	1024	1428.8	86.8	11.52073733
qwen2.5-7b-instruct	INT4-MIXED	1024	1253.4	87.1	11.48105626
falcon-7b-instruct	INT4-MIXED	32	189.4	87.2	11.46788991
llama-3-8b-instruct	INT4-MIXED	32	178.4	87.2	11.46788991
qwen2-7b-instruct	INT4-MIXED	1024	1314.2	87.2	11.46788991
qwen2-7b-instruct-gptq	INT4-MIXED	32	175.8	87.2	11.46788991
afm-4.5b	INT8-CW	32	160.9	87.3	11.45475372
bloomz-7b1	INT4-MIXED	32	166.7	87.5	11.42857143
deepseek-r1-distill-llama-8b	INT4-MIXED	1135	1382.2	87.6	11.41552511
minicpm-v-4_5	INT4-MIXED	217	950.8	87.8	11.38952164
qwen-7b-chat	INT4-MIXED	1024	1207.7	88	11.36363636
qwen2.5-vl-7b-instruct	INT4-MIXED	1024	1301.8	88.1	11.3507378
qwen2.5-7b-instruct	INT4-MIXED	1024	1432.9	88.2	11.33786848
qwen2.5-7b-instruct-1m	INT4-MIXED	1024	1432.1	88.2	11.33786848
qwen2-7b-instruct	INT4-MIXED	1024	1435.6	88.3	11.32502831
qwen3-8b	INT4-MIXED	32	184.4	88.6	11.28668172
llama-3-8b-instruct	INT4-MIXED	1024	1227.8	89	11.23595506
llama-3.1-8b-instruct	INT4-MIXED	1024	1213.2	89.3	11.19820829
qwen2.5-vl-7b-instruct	INT4-MIXED	1024	1489.7	89.4	11.18568233
minicpm4-8b	INT4-MIXED	32	193.1	89.5	11.17318436
qwen3-8b	INT4-MIXED	32	185.6	89.9	11.12347052
bloomz-7b1	INT4-MIXED	1024	1615.8	90.1	11.09877913
deepseek-r1-distill-llama-8b	INT4-MIXED	1135	1492.6	90.2	11.0864745
minicpm3-4b	INT8-CW	32	168.5	90.4	11.0619469
fara-7b	INT4-MIXED	1024	1776.8	90.5	11.04972376
afm-4.5b	INT8-CW	1024	910.3	90.7	11.02535832
qwen-7b-chat	INT4-MIXED	1024	1307.4	90.7	11.02535832
zephyr-7b-beta	INT4-MIXED	32	180.8	90.7	11.02535832
llama-3-8b-instruct	INT4-MIXED	1024	1317.4	91.3	10.95290252
qwen2-7b-instruct-gptq	INT4-MIXED	1024	1275.8	91.3	10.95290252
llama-3-8b-instruct	INT4-MIXED	1024	1328	91.4	10.94091904
llama-3.1-8b-instruct	INT4-MIXED	1024	1316.4	91.7	10.90512541
qwen3-8b	INT4-MIXED	1024	1236	91.9	10.88139282
minicpm-v-4_5	INT4-MIXED	217	973.5	92	10.86956522
minicpm4-8b	INT4-MIXED	1024	1284.6	92.2	10.84598698
minicpm4-8b	INT4-MIXED	32	194.9	92.3	10.83423619
bloomz-7b1	INT4-MIXED	1024	2020.8	92.4	10.82251082
llama-3-8b-instruct	INT4-MIXED	1024	1502.9	92.4	10.82251082
falcon-7b-instruct	INT4-MIXED	1024	1519.8	92.7	10.78748652
llava-next-video-7b-hf	INT4-MIXED	2945	566.8	93.8	10.66098081
qwen3-8b	INT4-MIXED	1024	1332.5	94.4	10.59322034
qwen3.5-9b	INT4-MIXED	83	286.1	94.6	10.57082452
minicpm4-8b	INT4-MIXED	1024	1418.3	95.1	10.51524711
qwen3-8b	INT4-MIXED	1024	1545.3	95.7	10.44932079
qwen3.5-9b	INT4-MIXED	1024	1852.6	95.9	10.42752868
phi-4-multimodal-instruct	INT8-CW	578	1170.5	96	10.41666667
qwen3-vl-4b-instruct	INT8-CW	4937	2719.7	96.1	10.40582726
qwen3-vl-4b-instruct	INT8-CW	4907	2622	96.2	10.3950104
zephyr-7b-beta	INT4-MIXED	1024	1373.1	96.3	10.38421599
baichuan2-7b-chat	INT4-MIXED	32	190.6	96.4	10.37344398
phi-4-multimodal-instruct	INT8-CW	786	738.8	96.9	10.31991744
gemma-7b-it	INT4-MIXED	32	177.2	97.6	10.24590164
minicpm4-8b	INT4-MIXED	1024	1626.6	98	10.20408163
gemma-2b-it	FP16	32	115.2	98.4	10.16260163
phi-4-multimodal-instruct	INT8-CW	1362	2787.8	98.9	10.11122346
gemma-2b-it	FP16	1024	646	99.4	10.06036217
phi-4-multimodal-instruct	INT8-CW	1570	1402	99.9	10.01001001

All models listed here were tested with the following parameters:

• Framework: PyTorch

• Beam: 1

• Batch size: 1

Get system descriptions [PDF]

Get the 7-368H table [CSV]

Get the 7-258V table [CSV]

Get the 7-255H table [CSV]

Get the 5-330 table [CSV]