Bare Metal and Virtual Hosts

OpenVINO™ Model Server includes a C++ implementation of gRPC and RESTful API interfaces defined by TensorFlow Serving. In the backend, it uses OpenVINO Runtime libraries from OpenVINO toolkit, which speeds up the execution on CPU, and enables it on iGPU and Movidius devices.

OpenVINO Model Server can be hosted on a bare metal server, virtual machine, or inside a docker container. It is also suitable for landing in the Kubernetes environment.

Before you start:

OpenVINO Model Server execution on baremetal is tested on Ubuntu 20.04.x. For other operating systems we recommend using OVMS docker containers.

For supported hardware, refer to supported configurations.

Always verify if your model is supported by the VPU Plugins and convert it to the OpenVINO format, using OpenVINO Model Optimizer.

Installing Model Server

  • Clone model server git repository.

  • Navigate to the model server directory.

  • To install Model Server, you can either use a precompiled binary or build it on your own, in a Docker container.

  • Navigate to the folder containing the binary package and unpack the included tar.gz file.

Here is an example of this process:

git clone https://github.com/openvinotoolkit/model_server

cd model_server

# automatically build a container from source
# it will also place a copy of the binary package in the `dist` subfolder in the Model Server root directory
make docker_build

# unpack the `tar.gz` file
cd dist/ubuntu && tar -xzvf ovms.tar.gz

Running the Server

The server can be started in two ways:

  • using the ./ovms/bin/ovms --help command in the folder, where OVMS was is installed

  • in the interactive mode - as a background process or a daemon initiated by systemctl/initd depending on the Linux distribution and specific hosting requirements

Refer to Running Model Server using Docker Container to get more details on the OpenVINO Model Server parameters and configuration.

Note

When AI accelerators are used for inference execution, additional steps may be required to install their drivers and dependencies. Learn more about it Learn more about it on OpenVINO installation guide.