Skip to main content
Back to top
Ctrl
+
K
Site Navigation
Installation
Product Page
Blog
Forum
Support
Performance
GitHub
Site Navigation
Installation
Product Page
Blog
Forum
Support
Performance
GitHub
Section Navigation
GET STARTED
Install OpenVINO
OpenVINO GenAI
OpenVINO Runtime on Linux
Use Archive
Use PyPI
Use APT
Use YUM
Use ZYPPER
Use Conda Forge
Use vcpkg
Use Homebrew
Use Docker
Use Conan
Use npm
OpenVINO Runtime on Windows
Use Archive
Use PyPI
Use Conda Forge
Use vcpkg
Use Docker
Use Conan
Use npm
Use WinGet
OpenVINO Runtime on macOS
Use Archive
Use Homebrew
Use PyPI
Use Conda Forge
Use vcpkg
Use Conan
Use npm
Create an OpenVINO Yocto Image
Additional Hardware Setup and Troubleshooting
For GPU
For NPU
GenAI Dependencies
Troubleshooting Guide for OpenVINO™ Installation & Configuration
Learn OpenVINO
Interactive Tutorials (Python)
Install Notebooks
Run Notebooks
Sample Applications (Python & C++)
Get Started with Samples
Hello Classification Sample
Hello NV12 Input Classification Sample
Hello Query Device Sample
Hello Reshape SSD Sample
Image Classification Async Sample
Model Creation Sample
Sync Benchmark Sample
Throughput Benchmark Sample
Bert Benchmark Python Sample
Benchmark Tool
System Requirements
HOW TO USE - GENERATIVE AI WORKFLOW
Generative Model Preparation
Inference with OpenVINO GenAI
NPU with OpenVINO GenAI
Inference with Optimum Intel
OpenVINO Tokenizers
HOW TO USE - CONVENTIONAL AI WORKFLOW
Conventional Model Preparation
Convert to OpenVINO Model
Convert from PyTorch
Convert from TensorFlow
Convert from ONNX
Convert from TensorFlow Lite
Convert from PaddlePaddle
Convert from JAX/Flax
Convert from Keras
Conversion Parameters
Setting Input Shapes
Model Optimization - NNCF
LLM Weight Compression
4-bit Weight Quantization
Microscaling (MX) Quantization
Post-training Quantization
Basic Quantization Flow
Quantizing with Accuracy Control
Training-time Optimization
Quantization-aware Training (QAT)
Running Inference
Model Representation in OpenVINO™ Runtime
Model Input/Output
Changing Input Shapes
Dynamic Shapes
When Dynamic Shapes API is Not Applicable
String Tensors
OpenVINO™ Inference Request
Stateful models and State API
Obtaining a Stateful OpenVINO Model
OpenVINO™ Python API Exclusives
OpenVINO™ Runtime Python API Advanced Inference
Inference Devices and Modes
CPU Device
Performance Hints and Thread Scheduling
GPU Device
Remote Tensor API of GPU Plugin
NPU Device
Remote Tensor API of NPU Plugin
NPU Plugin Batching
Automatic Device Selection
Debugging Auto-Device Plugin
Heterogeneous Execution
Automatic Batching
Query Device Properties - Configuration
Optimize Inference
General Optimizations
High-level Performance Hints
Precision Control
Optimizing for Latency
Model Caching Overview
Optimizing for Throughput
Advanced Throughput Options
Optimize Preprocessing
Preprocessing API - details
Use Case - Integrate and Save Preprocessing Steps Into IR
Layout API Overview
Torchvision preprocessing converter
Further Low-Level Implementation Details
Optimizing memory usage
Deployment on a Local System
Local Distribution Libraries
Optimize Binaries Size
Integrate OpenVINO with Ubuntu Snap
PyTorch Deployment via “torch.compile”
HOW TO USE - MODEL SERVING
QuickStart - classic models
QuickStart - LLM models
Prepare a Model Repository
Classic Models
Use Cloud Storage
Graphs Repository
Exporting GEN AI Models
OVMS Pull mode
Exporting models using script
OVMS Pull mode with optimum cli
Deploy Model Server
Deploying Model Server in Docker Container
Deploying Model Server on Baremetal
OpenVINO Model Server as service on Windows
Deploying Model Server in Kubernetes
Starting the Server
Model Server Parameters
Using AI Accelerators
Security Considerations
Write a Client Application
Generative AI Use Cases
Chat completion API
Completions API
Embeddings API
Tokenize API
Reranking API
Image generation API
Image generation API
Speech to text API
Text to speech API
KServe API
gRPC API
RESTful API
Examples
OpenVINO Model Server C-API
Model Server Features
GenAI Endpoints
Integration with mediapipe
gRPC Streaming API
Python Execution in OpenVINO Model Server
Support for Binary Encoded Image Input Data
Input Shape and Layout Considerations
Predict on Binary Inputs via KServe API
Convert TensorFlow Models to Accept Binary Inputs
Support for text data format
Batch, Shape and Layout
Dynamic Input Parameters
Dynamic Shape with dynamic IR/ONNX Model
Dynamic Shape with Binary Inputs
Dynamic Shape with a Custom Node
Online Configuration Updates
Model Version Policy
Idle Servable Management (Preview)
Metrics
OpenVINO Model Server C-API
Advanced Features
CPU Extensions
Model Cache
Custom Model Loader
Securing Model Server with NGINX
Legacy features
Directed Acyclic Graph (DAG) Scheduler
Demultiplexing in DAG
Custom Node Development Guide
Performance tuning
Demos
Text generation
LLM demo
VLM demo
Agentic demo
RAG
Long Context
Structured output
LLM Models in Speculative Decoding Pipeline
LLM on NPU
Scaling on multi CPU and GPU
Loading models in GGUF
Image generation
Audio
Text Embeddings models via OpenAI API
Documents Reranking via Cohere API
Classic models
Image classification
ONNX
TensorFlow
PaddlePaddle
Age gender classification
Face detection
Person Detection
Using input strings
Using output strings
MediaPipe
MediaPipe Object Detection Demo
Iris
Holistic
Realtime Stream Analysis
Python Node
OpenClip with Python Execution
Integrations
OpenWebUI
VSC Continue dev
Cline demo
GitHub Copilot demo
Troubleshooting
HOW TO USE - PHYSICAL AI
Getting Started
Installation
Quickstart
Run a Policy
Security Model
How-To Guides
Run a Policy on a Robot
Use Execution Modes
Add Runtime Callbacks
Share a Robot Across Processes
Upgrade from 0.1.x
Load an Exported Policy
Use a Manifest
Configure Preprocessing and Postprocessing
Write Runtime Config
Write Inference Config
Instantiate Components
CLI: Run
Explanation
Architecture
Inference
Runtime
Configuration
Manifests
CLI
Robots
Cameras
Reference
CLI Reference
Config Schema Reference
Manifest Schema Reference
Inference API Reference
Runtime API Reference
Robot API Reference
Camera API Reference
Benchmark API Reference
REFERENCE DOCUMENTATION
API Reference
OpenVINO Python API
openvino
openvino.preprocess
openvino.properties
openvino.properties.device
openvino.properties.hint
openvino.properties.intel_auto
openvino.properties.intel_cpu
openvino.properties.intel_gpu
openvino.properties.intel_gpu.hint
openvino.properties.log
openvino.properties.streams
openvino.op
openvino.op.util
openvino.opset1
openvino.opset2
openvino.opset3
openvino.opset4
openvino.opset5
openvino.opset6
openvino.opset7
openvino.opset8
openvino.opset9
openvino.opset10
openvino.opset11
openvino.opset12
openvino.opset13
openvino.opset14
openvino.opset15
openvino.opset16
openvino.opset17
openvino.passes
openvino.frontend
openvino.experimental
OpenVINO Node.js API
addon
element
resizeAlgorithm
CompiledModel
Core
CoreConstructor
InferRequest
InputInfo
InputModelInfo
InputTensorInfo
Model
Output
OutputInfo
OutputTensorInfo
OVAny
PartialShape
PartialShapeConstructor
PrePostProcessor
PrePostProcessorConstructor
PreProcessSteps
Tensor
TensorConstructor
Dimension
elementTypeString
SupportedTypedArray
GenAI Python API
openvino_genai
OpenVINO IR format and Operation Sets
Operation Sets in OpenVINO
Available Operation Sets
opset16
opset15
opset14
opset13
opset12
opset11
opset10
opset9
opset8
opset7
opset6
opset5
opset4
opset3
opset2
opset1
Operation Specifications
Abs-1
Acos-1
Acosh-3
AdaptiveAvgPool-8
AdaptiveMaxPool-8
Add-1
Asin-1
Asinh-3
Assign-3
Assign-6
Atan-1
Atanh-3
AUGRUCell
AUGRUSequence
AvgPool-1
AvgPool-14
AvgPool-16
BatchNormInference-1
BatchNormInference-5
BatchToSpace-2
BinaryConvolution-1
BitwiseAnd-13
BitwiseLeftShift-15
BitwiseNot-13
BitwiseOr-13
BitwiseRightShift-15
BitwiseXor-13
Broadcast-1
Broadcast-3
Bucketize-3
CTCGreedyDecoder-1
CTCGreedyDecoderSeqLen-6
Ceiling-1
Clamp-1
Col2Im-15
Concat-1
Constant-1
ConvertLike-1
ConvertPromoteTypes-14
Convert-1
ConvolutionBackpropData-1
Convolution-1
Cos-1
Cosh-1
CTCLoss-4
CumSum-3
DeformableConvolution-1
DeformableConvolution-8
DeformablePSROIPooling-1
DepthToSpace-1
DetectionOutput-1
DetectionOutput-8
DFT-7
Divide-1
Einsum-7
Elu-1
EmbeddingBagOffsetsSum-3
EmbeddingBagOffsets-15
EmbeddingBagPackedSum-3
EmbeddingBagPacked-15
EmbeddingSegmentsSum-3
Equal-1
Erf-1
Exp-1
ExperimentalDetectronDetectionOutput-6
ExperimentalDetectronGenerateProposalsSingleImage-6
ExperimentalDetectronPriorGridGenerator-6
ExperimentalDetectronROIFeatureExtractor-6
ExperimentalDetectronTopKROIs-6
ExtractImagePatches-3
Eye-9
FakeConvert-13
FakeQuantize-1
FloorMod-1
Floor-1
GridSample-9
GRN-1
GRUCell-3
GRUSequence-5
GatedDeltaNet
GatherTree-1
Gather-1
Gather-7
Gather-8
GatherElements-6
GatherND-5
GatherND-8
GELU-2
GELU-7
GenerateProposals-9
GreaterEqual-1
Greater-1
GroupConvolutionBackpropData-1
GroupConvolution-1
GroupNormalization-12
HardSigmoid-1
HSigmoid-5
HSwish-4
I420toBGR-8
I420toRGB-8
Identity-16
IDFT-7
IRDFT-9
ISTFT-16
If-8
Interpolate-1
Interpolate-4
Interpolate-11
Inverse-14
IsFinite-10
IsInf-10
IsNaN-10
LRN-1
LSTMCell-1
LSTMSequence-5
LessEqual-1
Less-1
Log-1
LogicalAnd-1
LogicalNot-1
LogicalOr-1
LogicalXor-1
LogSoftmax-5
Loop-5
MVN-1
MVN-6
MatMul-1
MatrixNms-8
MaxPool-1
MaxPool-8
MaxPool-14
Shape Calculation Rules for Pooling Operators
Maximum-1
Minimum-1
Mish-4
Mod-1
MulticlassNonMaxSuppression-8
MulticlassNonMaxSuppression-9
Multinomial-13
Multiply-1
Negative-1
NMSRotated-13
NonMaxSuppression-1
NonMaxSuppression-3
NonMaxSuppression-4
NonMaxSuppression-5
NonMaxSuppression-9
NonZero-3
NormalizeL2-1
NotEqual-1
NV12toBGR-8
NV12toRGB-8
OneHot-1
OneHot-16
PReLU-1
PSROIPooling-1
Pad-1
Pad-12
Parameter-1
Power-1
PriorBoxClustered-1
PriorBox-1
PriorBox-8
Proposal-1
Proposal-4
RandomUniform-8
Range-1
Range-4
RDFT-9
ReadValue-3
ReadValue-6
ReLU-1
ReduceL1-4
ReduceL2-4
ReduceLogicalAnd-1
ReduceLogicalOr-1
ReduceMax-1
ReduceMean-1
ReduceMin-1
ReduceProd-1
ReduceSum-1
RegionYolo-1
ReorgYolo-1
Reshape-1
Result-1
Reverse-1
ReverseSequence-1
RMS
RNNCell-3
RNNSequence-5
ROIAlign-3
ROIAlign-9
ROIAlignRotated-15
ROIPooling-1
Roll-7
Round-5
ScaledDotProductAttention-13
ScatterElementsUpdate-3
ScatterElementsUpdate-12
ScatterNDUpdate-3
ScatterNDUpdate-15
ScatterUpdate-3
SearchSorted-15
SegmentMax-16
Select-1
Selu-1
ShapeOf-1
ShapeOf-3
ShuffleChannels-1
Sigmoid-1
Sign-1
Sin-1
Sinh-1
Slice-8
SliceScatter-15
SoftMax-1
SoftMax-8
SoftPlus-4
SoftSign-9
SpaceToBatch-2
SpaceToDepth-1
SparseFillEmptyRows-16
Split-1
Sqrt-1
SquaredDifference-1
Squeeze-1
Squeeze-15
STFT-15
StridedSlice-1
StringTensorPack-15
StringTensorUnpack-15
Subtract-1
Swish-4
Tan-1
Tanh-1
TensorIterator-1
Tile-1
TopK-1
TopK-3
TopK-11
Transpose-1
Unique-10
Unsqueeze-1
VariadicSplit-1
Broadcast Rules For Elementwise Operations
Low Precision IR (INT8 Inference)
Compatibility and Support
Supported Devices
Supported Operations
AI Models verified for OpenVINO™
Legacy Features
OpenVINO Extensibility
Custom OpenVINO Operations
Frontend Extensions
How to Implement Custom GPU Operations
Overview of Transformations API
OpenVINO Model Pass
OpenVINO Matcher Pass
OpenVINO Graph Rewrite Pass
Transformation Patterns with OpenVINO API
OpenVINO Plugin Developer Guide
Implement Plugin Functionality
Implement Compiled Model Functionality
Implement Synchronous Inference Request
Implement Asynchronous Inference Request
Provide Plugin Specific Properties
Implement Remote Context
Implement Remote Tensor
Build Plugin Using CMake
Plugin Testing
Advanced Topics
Quantized models compute and restrictions
OpenVINO™ Low Precision Transformations
Quantization Scheme
Attributes
AvgPoolPrecisionPreserved
IntervalsAlignment
PrecisionPreserved
Precisions
QuantizationAlignment
QuantizationGranularity
Step 1. Prerequisites transformations
PullReshapeThroughDequantization
PullTransposeThroughDequantization
LinOpSequenceFusion
ConvertSubtractConstant
Step 2. Markup transformations
MarkupBias
MarkupCanBeQuantized
MarkupPrecisions
MarkupPerTensorQuantization
MarkupAvgPoolPrecisionPreserved
PropagatePrecisions
AlignQuantizationIntervals
AlignQuantizationParameters
CreateAttribute
CreatePrecisionsDependentAttribute
PropagateThroughPrecisionPreserved
PropagateToInput
PropagateSharedValue
UpdateSharedPrecisionPreserved
Step 3. Main transformations
AddTransformation
AvgPoolTransformation
BatchToSpaceTransformation
ClampTransformation
ConcatTransformation
ConvolutionTransformation
ConvolutionBackpropDataTransformation
DepthToSpaceTransformation
FakeQuantizeDecompositionTransformation
FakeQuantizeTransformation
InterpolateTransformation
GroupConvolutionTransformation
GatherTransformation
MatMulTransformation
MaxPoolTransformation
MultiplyPartialTransformation
MultiplyTransformation
MVNTransformation
NormalizeL2Transformation
PadTransformation
PReluTransformation
ReduceMaxTransformation
ReduceMeanTransformation
ReduceMinTransformation
ReduceSumTransformation
ReluTransformation
ReshapeTransformation
SpaceToBatchTransformation
SqueezeTransformation
ShuffleChannelsTransformation
SplitTransformation
StridedSliceTransformation
TransposeTransformation
UnsqueezeTransformation
VariadicSplitTransformation
Step 4. Cleanup transformations
EliminateFakeQuantizeTransformation
FoldConvertTransformation
FoldFakeQuantizeTransformation
FuseConvertTransformation
FuseMultiplyToFakeQuantizeTransformation
FuseSubtractToFakeQuantizeTransformation
MultiplyToGroupConvolutionTransformation
Plugin API Reference
OpenVINO Security
ABOUT OPENVINO
Key Features
Performance Benchmarks
Efficient LLMs for AI PC
Performance Information F.A.Q.
OpenVINO Accuracy
Getting Performance Numbers
OpenVINO Ecosystem
OpenVINO Integrations
OpenVINO Tools
OpenVINO™ Test Drive
Contribute to OpenVINO
Code Contribution Guide
Release Notes
System Requirements
Release Policy
OpenVINO™ Model Server
Demos
Python Node
Python Node
#
OpenClip with Python Execution
Download Docs
.pdf
.zip