Starting the Server#

There are three methods for passing model information and configuration to the model server:

via CLI parameters, for a single model or pipeline
via a config file in JSON format, for any number of classic models
via graph.pbtxt in the model folder, for all MediaPipe pipelines including generative models

Note that changing configuration at runtime while serving is possible only with the config file. When deploying models with a configuration file, you can add or delete models, as well as update their configurations at runtime, without restarting the server. Updating graph.pbtxt runtime parameters takes effect after the model is unloaded and reloaded (removed from config.json and added again, or after the service is restarted).

Serving Multiple Models#

To serve multiple models and pipelines from the same container, you need an additional JSON configuration file that defines each model. The model_config_list array includes a collection of config objects for each served model. The name and base_path values are required for each config object.

{
   "model_config_list":[
      {
         "config":{
            "name":"model_name1",
            "base_path":"/opt/ml/models/model1",
            "batch_size": "16",
            "model_version_policy": {"all": {}}
         }
      },
      {
         "config":{
            "name":"model_name3",
            "base_path":"gs://bucket/models/model3",
            "model_version_policy": {"specific": { "versions":[1, 3] }}
         }
      },
      {
         "config":{
             "name":"model_name4",
             "base_path":"s3://bucket/models/model4",
             "shape": {
                "input1": "(1,3,200,200)",
                "input2": "(1,3,50,50)"
             },
             "plugin_config": {"PERFORMANCE_HINT": "THROUGHPUT"}
         }
      },
      {
         "config":{
            "name":"Phi-3-mini-FastDraft-50M-int8-ov",
            "base_path":"/models/OpenVINO/Phi-3-mini-FastDraft-50M-int8-ov/"
         }
      }
   ]
}

How to run with config.json:

With Docker

Required: Docker Engine installed

docker run --user $(id -u):$(id -g) --rm -v <models_repository>:/models:ro -p 9000:9000 -p 8000:8000 openvino/model_server:latest \
--config_path /models/config.json --port 9000 --rest_port 8000

On Bare-metal Host

Required: OpenVINO Model Server package - see deployment instructions for details.

ovms --config_path /models/config.json --port 9000 --rest_port 8000

Config management#

List models#

Assuming you have a prepared model repository, use the following command to check which models/graphs are servable from the specified repository:

With Docker

Required: Docker Engine installed :sync: docker

docker run -d --rm -v <model_repository_path>:/models openvino/model_server:latest \
--model_repository_path /models --list_models

On Bare-metal Host

Required: OpenVINO Model Server package - see deployment instructions for details.

ovms --model_repository_path <model_repository_path> --list_models

For the following directory structure:

/models
├── meta
│   ├── llama4
│   │   └── graph.pbtxt
│   ├── llama3.1
│   │   └── graph.pbtxt
├── llama3.2
│   └── graph.pbtxt
└── resnet
    └── 1
        └── saved_model.pb

The output would be:

meta/llama4
meta/llama3.1
llama3.2
resnet

Enable model#

To add a model to an OVMS configuration file, you can either do it manually or use:

With Docker

Required: Docker Engine installed

docker run -d --rm -v <model_repository_path>:/models openvino/model_server:latest \
--model_repository_path /models/ --add_to_config --config_path <config_file_path> --model_name <name>

On Bare-metal Host

Required: OpenVINO Model Server package - see deployment instructions for details. Note: the OVMS_MODEL_REPOSITORY_PATH environment variable can determine default values for --model_repository_path and --config_path, where --config_path defaults to config.json inside the model repository path.

export OVMS_MODEL_REPOSITORY_PATH=/models
ovms --add_to_config --model_name <name>

Use this when the model is directly inside the model repository.

Note: If you want to add a model with a specific path, you can use the --model_path parameter:

docker run -d --rm -v <model_repository_path>:/models openvino/model_server:latest \
--add_to_config --config_path <config_file_path> --model_name <name> --model_path <model_path>

Note: Use relative or absolute paths. config_path is relative to the current folder. model_path is relative to the config file. With model_repository_path, model_name represents a folder relative to the model repository path. For example:

ovms --add_to_config --config_path models/config.json --model_name OpenVINO/DeepSeek-R1-Distill-Qwen-1.5B-int4-ov --model_repository_path models
or
ovms --add_to_config --config_path models/config.json --model_name OpenVINO/DeepSeek-R1-Distill-Qwen-1.5B-int4-ov --model_path OpenVINO/DeepSeek-R1-Distill-Qwen-1.5B-int4-ov

Adding classic models to config.json is also possible with extra runtime parameters:

ovms --add_to_config --config_path config.json --model_name resnet --model_path models/resnet/resnet.xml --batch_size 1 --target_device CPU

Disable model#

If you want to remove a model from the configuration file, you can do it manually or use the command below:

With Docker

Required: Docker Engine installed

docker run -d --rm -v <model_repository_path>:/models openvino/model_server:latest \
--remove_from_config --config_path <config_file_path> --model_name <name>

On Bare-metal Host

Required: OpenVINO Model Server package - see deployment instructions for details.

ovms --remove_from_config --config_path <config_file_path> --model_name <name>

Starting the Server#

Serving a Single Classic Model, MediaPipe, or GenAI Model#

Starting with prepared models#

Starting the GenAI model from Hugging Face directly#

Serving Multiple Models#

Config management#

List models#

Enable model#

Disable model#