Long context optimizations for LLM models#

Using LLM models with very long context and prompts might be challenging. The key goals are to get maximum throughput, minimal latency and reasonable memory consumption. It is very common for applications using RAG chains, document summarization, question answering, and many more. The optimizations below can significantly boost performance:

  • Prefix caching

  • KV cache compression

  • Max prompt length (NPU)

  • Cache interval multiplier for linear attention

  • Tuning max number of batched tokens

Prefix caching

Prefix caching in large language models (LLMs) is an optimization technique used to improve performance when processing repeated or static parts of input prompts. Instead of recomputing KV for the same prefix (e.g., a fixed instruction or context), it is cached after the first computation and stored after request is already processed and response is returned. When the same prefix is encountered again, the cached KV is reused, skipping redundant computations. This reduces latency and computational overhead, especially in scenarios like chatbots or applications with repetitive prompts. It is enabled by default.

KV cache compression: KV cache stores the intermediate key and value tensors generated by the model’s attention layers for each token in the input sequence. This cache allows the model to avoid recomputing attention for previous tokens when generating new tokens, greatly speeding up inference for long contexts. For very long contexts or high concurrency, the KV cache can consume a large amount of memory (RAM or VRAM). Compression reduces this memory usage, enabling longer prompts or more parallel requests without running out of memory. This parameter is applicable only to pipelines with continuous batching and paged attention. It is not used with NPU device.

Max prompt length Because NPU uses static memory allocation for prompt processing, a dedicated parameter for NPU device max_prompt_len has been introduced. The default value of 1024 should be adjusted to match expected request sizes.

Cache interval multiplier This parameter is dedicated for models with linear attention and prefix caching enabled. It adjusts the allocation size for state blocks internally in openvino.genai backend. For processing long inputs with low memory footprint, it is recommended to increase this parameter from default value 8 to higher like 64.

Max number of batched tokens This parameter influences the behavior of the continuous batching algorithm and the size of chunked prompts. The default value of 256 tokens is efficient when concurrent processing is expected. When typically only one client connects to the local model (especially with long prompts), increasing this value can improve first token latency.

Deployment#

Let’s demonstrate all the optimizations combined and test it with the real-life scenario of sending multiple various questions in the same context. It will illustrate the gain from the prefix caching on the first token latency, improved second token latency thanks to prompt lookup and moderate memory consumption despite very long prompts and parallel execution.

Prepare models directory:

mkdir models

docker run --user $(id -u):$(id -g) -d --rm -v $(pwd)/models:/models:rw -p 8000:8000 --device /dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) openvino/model_server:latest-gpu --rest_port 8000 --model_repository_path /models --source_model OpenVINO/gpt-oss-20b-int4-ov --kv_cache_precision u4 --target_device GPU --cache_size 5 --max_num_batched_tokens 8192

docker run --user $(id -u):$(id -g) -d --rm -v $(pwd)/models:/models:rw -p 8000:8000 --device /dev/accel --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) openvino/model_server:latest-gpu --rest_port 8000 --model_repository_path /models --source_model OpenVINO/Qwen3-8B-int4-cw-ov --max_prompt_len 8000 --tool_parser hermes3 --task text_generation --target_device NPU

Testing latency#

The vllm benchmark tool can measure model performance with various context lengths and prefix configurations. This allows you to verify performance gains from prefix caching under realistic conditions.

Key metrics used in the performance tables below:

  • TTFT (Time To First Token): Latency from sending a request to receiving the first output token (in milliseconds)

  • TPOT (Time Per Output Token): Average time to generate each subsequent token after the first (in milliseconds)

The command below generates synthetic load with a configurable prefix-repetition-prefix-len (5000) and prefix-repetition-output-len (20):

pip install vllm --extra-index-url https://wheels.vllm.ai/nightly/cpu
vllm bench serve --backend  openai --base-url http://localhost:8000/v3 --endpoint /completions --model  OpenVINO/gpt-oss-20b-int4-ov --tokenizer openai/gpt-oss-20b --prefix-repetition-prefix-len 50000 --prefix-repetition-suffix-len 10 --prefix-repetition-output-len 20 --prefix-repetition-num-prefixes 1  --num-prompts 1 --max_concurrency 1 --dataset-name prefix_repetition --num-warmups 1 --seed 1

Performance Comparison Table#

iGPU Platform: Intel(R) Core(TM) Ultra X7 368H

Context Length (tokens)

TTFT No Cache (ms)

TPOT No Cache (ms)

TTFT Prefix Cache (ms)

TPOT Prefix Cached (ms)

KV Cache Usage (GB)

1,000

943

30.48

175

28.88

0.02

12,000

6 496

38.19

328

37.80

0.2

25,000

21 381

39.87

408

48.50

0.3

50,000

88 159

69.46

945

82.42

0.6

100,000

248 777

119.20

1258

102.36

1.5

Those results confirm gain from prefix caching for repeated tokens and demonstrate low KV cache usage thanks to quantization even with long context.

NPU Platform: Intel(R) Core(TM) Ultra X7 368H

Context Length (tokens)

TTFT No Cache (ms)

TPOT No Cache (ms)

TTFT Prefix Cached (ms)

TPOT Prefix Cached (ms)

500

1 514

76.62

1 491

77.36

1,000

1 366

78.10

1 374

79.18

2,000

2 662

79.74

1 518

80.09

4,000

6 505

76.75

2 509

77.37

8,000

15 432

76.74

3 285

77.51

This table shows the gain from prefix caching on NPU device and flat latency for whole range of prompt length.

KV Cache Precision#

KV cache compression has minimal impact on accuracy and significantly reduces memory consumption and benchmark time. The default value is u8, but it’s possible to change it to u4, f16 or f32.

Context Length (tokens)

TTFT for precision u4 (ms)

Cache size for u4 (GB)

TTFT for precision u8 (ms)

Cache size for u8 (GB)

50,000

945

0.7

985

1.5

Lower precision in KV Cache reduces the memory consumption and can also improve latency.

Max prompt length for NPU#

Parameter --max_prompt_len has impact on the latency. It should be adjusted for expected input length to optimize performance.

Max prompt length

1k input TTFT (ms)

1k input TPOT (ms)

8K

1 637

55.88

1K

1 436

52.19

Recommendations#

General Strategy#

  • Leverage prefix caching to process repeated tokens faster (enabled by default)

  • Set KV cache size via --cache_size parameter based on available memory, expected concurrency, and context length

KV Cache Configuration#

  • Use u4 KV cache compression (default matches model precision). Increase precision only if memory pressure is low.

GPU Deployments#

  • Combine prefix caching with KV cache compression for optimal latency and memory usage

  • Adjust --cache_size based on available VRAM

NPU Deployments#

  • Set --max_prompt_len based on expected input length. Lower values reduce memory usage and latency.

  • NPU is recommended for short to medium context lengths (typically up to 8K tokens)

Models with Linear Attention (e.g., Qwen3.6-35B-A3B)#

  • Set --cache_interval_multiplier=64 to optimize memory usage with prefix caching

    • Default (adaptive): optimizes memory for long context

    • 64: recommended for medium context

    • 8: this minimal value optimize for performance, recommended with short inputs

Low-Concurrency / Long-Context Scenarios#

  • Increase --max_num_batched_tokens to 4096 or even the model’s max context length

  • This improves TTFT, especially for long inputs

  • Trade-off: impact on interactive usage under higher concurrency

Concurrency Control#

To limit concurrent requests on the server:

  • --rest_workers: Controls number of simultaneous connections (default: number of CPU cores)

  • --max_num_seqs: Controls maximum concurrent sequences in scheduling at the model level for finer control