Deploying Model Server on Baremetal#

It is possible to deploy Model Server outside of container. To deploy Model Server on baremetal, use pre-compiled binaries for Ubuntu22, Ubuntu24, RHEL9 or Windows 11.

Download precompiled package (without python support):

wget https://github.com/openvinotoolkit/model_server/releases/download/v2025.0/ovms_ubuntu22.tar.gz
tar -xzvf ovms_ubuntu22.tar.gz

or precompiled package (with python and LLM support):

wget https://github.com/openvinotoolkit/model_server/releases/download/v2025.0/ovms_ubuntu22_python_on.tar.gz
tar -xzvf ovms_ubuntu22_python_on.tar.gz

Install required libraries:

sudo apt update -y && apt install -y libxml2 curl

Set path to the libraries and add binary to the PATH

export LD_LIBRARY_PATH=${PWD}/ovms/lib
export PATH=$PATH:${PWD}/ovms/bin

In case of the version with python and LLM support run also:

export PYTHONPATH=${PWD}/ovms/lib/python
sudo apt -y install libpython3.10
pip3 install "Jinja2==3.1.6" "MarkupSafe==3.0.2"

Download precompiled package (without python support):

wget https://github.com/openvinotoolkit/model_server/releases/download/v2025.0/ovms_ubuntu24.tar.gz
tar -xzvf ovms_ubuntu24.tar.gz

or precompiled package (with python and LLM support):

wget https://github.com/openvinotoolkit/model_server/releases/download/v2025.0/ovms_ubuntu24_python_on.tar.gz
tar -xzvf ovms_ubuntu24_python_on.tar.gz

Install required libraries:

sudo apt update -y && apt install -y libxml2 curl

Set path to the libraries and add binary to the PATH

export LD_LIBRARY_PATH=${PWD}/ovms/lib
export PATH=$PATH:${PWD}/ovms/bin

In case of the version with python and LLM support run also:

export PYTHONPATH=${PWD}/ovms/lib/python
sudo apt -y install libpython3.12
pip3 install "Jinja2==3.1.6" "MarkupSafe==3.0.2"

Download precompiled package (without python support):

wget https://github.com/openvinotoolkit/model_server/releases/download/v2025.0/ovms_redhat.tar.gz
tar -xzvf ovms_redhat.tar.gz

or precompiled package (with python and LLM support):

wget https://github.com/openvinotoolkit/model_server/releases/download/v2025.0/ovms_redhat_python_on.tar.gz
tar -xzvf ovms_redhat_python_on.tar.gz

Install required libraries:

sudo yum install compat-openssl11.x86_64

Set path to the libraries and add binary to the PATH

export LD_LIBRARY_PATH=${PWD}/ovms/lib
export PATH=$PATH:${PWD}/ovms/bin

In case of the version with python and LLM support run also:

export PYTHONPATH=${PWD}/ovms/lib/python
sudo yum install -y python39-libs
pip3 install "Jinja2==3.1.6" "MarkupSafe==3.0.2"

Make sure you have Microsoft Visual C++ Redistributable installed before moving forward.

Download and unpack model server archive for Windows:

curl -L https://github.com/openvinotoolkit/model_server/releases/download/v2025.0/ovms_windows.zip -o ovms.zip
tar -xf ovms.zip

Run setupvars script to set required environment variables.

Windows Command Line

.\ovms\setupvars.bat

Windows PowerShell

.\ovms\setupvars.ps1

Note: Running this script changes Python settings for the shell that runs it. Environment variables are set only for the current shell so make sure you rerun the script before using model server in a new shell.

Note: When serving LLM models, OVMS uses Python’s Jinja package to apply chat template. Please ensure you have Windows “Beta Unicode UTF-8 for worldwide language support” enabled. Instruction

You can also build model server from source by following the developer guide.

Test the Deployment#

Download ResNet50 model:

curl --create-dirs -k https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.1/models_bin/2/resnet50-binary-0001/FP32-INT1/resnet50-binary-0001.xml -o models/resnet50/1/model.xml
curl --create-dirs -k https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.1/models_bin/2/resnet50-binary-0001/FP32-INT1/resnet50-binary-0001.bin -o models/resnet50/1/model.bin

For linux run:

chmod -R 755 models

Start the server:

ovms --port 9000 --model_name resnet --model_path models/resnet50

or start as a background process, daemon initiated by systemctl/initd or a Windows service depending on the operating system and specific hosting requirements.

Most of the Model Server documentation demonstrate containers usage, but the same can be achieved with just the binary package. Learn more about model server starting parameters.

NOTE: When serving models on AI accelerators, some additional steps may be required to install device drivers and dependencies. Learn more in the Additional Configurations for Hardware documentation.

Next Steps#

Additional Resources#