Text generation demos#
- LLM demo
- VLM demo
- Agentic demo
- RAG
- Long Context
- Structured output
- LLM Models in Speculative Decoding Pipeline
- MTP (Multi-Token Prediction)
- ============ Serving Benchmark Result ============ Successful requests: 10 Failed requests: 0 Maximum request concurrency: 1 Benchmark duration (s): 48.47 Total input tokens: 7883 Total generated tokens: 2324 Request throughput (req/s): 0.21 Output token throughput (tok/s): 47.95 Peak output token throughput (tok/s): 67.00 Peak concurrent requests: 2.00 Total token throughput (tok/s): 210.58 —————Time to First Token—————- Mean TTFT (ms): 534.38 Median TTFT (ms): 462.31 P99 TTFT (ms): 865.17 —–Time per Output Token (excl. 1st token)—— Mean TPOT (ms): 18.57 Median TPOT (ms): 18.67 P99 TPOT (ms): 20.42 —————Inter-token Latency—————- Mean ITL (ms): 18.83 Median ITL (ms): 0.01 P99 ITL (ms): 58.57
- EAGLE3
- Setting Default Generation Parameters
- LLM on NPU
- Scaling on multi CPU and GPU
- Loading models in GGUF