Prediction Example with an ONNX Model

Steps are similar to when you work with IR model format. Model Server accepts ONNX models as well with no differences in versioning. Locate ONNX model file in separate model version directory. Below is a complete functional use case using Python 3.6 or higher. For this example let’s use a public ONNX ResNet model - resnet50-caffe2-v1-9.onnx model.

This model requires additional preprocessing function. Preprocessing can be performed in the client by manipulating data before sending the request. Preprocessing can be also delegated to the server by creating a DAG and using a custom processing node. Both methods will be explained below.

Option 1: Adding preprocessing to the client side

Option 2: Adding preprocessing to the server side (building DAG)

Option 1: Adding preprocessing to the client side

Clone the repository and enter using_onnx_model directory

git clone https://github.com/openvinotoolkit/model_server.git
cd model_server/demos/using_onnx_model/python

Prepare workspace with the model by running:

make client_preprocessing

You should see workspace directory created with the following content:

workspace/
└── resnet50-onnx
    └── 1
        └── resnet50-caffe2-v1-9.onnx

Start the OVMS container with single model instance:

docker run -d -u $(id -u):$(id -g) -v $(pwd)/workspace:/workspace -p 9001:9001 openvino/model_server:latest \
--model_path /workspace/resnet50-onnx --model_name resnet --port 9001

Install python client dependencies:

pip3 install -r requirements.txt

The onnx_model_demo.py script can run inference both with and without performing preprocessing. Since in this variant we want to run preprocessing on the client side let’s set --run_preprocessing flag.

Run the client with preprocessing:

python3 onnx_model_demo.py --service_url localhost:9001 --run_preprocessing
Running with preprocessing on client side
../../common/static/images/bee.jpeg (1, 3, 224, 224) ; data range: -2.117904 : 2.64
Class is with highest score: 309
Detected class name: bee

Option 2: Adding preprocessing to the server side (building a DAG)

Prepare workspace with the model, preprocessing node library and configuration file by running:

make server_preprocessing

You should see workspace directory created with the following content:

workspace/
├── config.json
├── lib
│   └── libcustom_node_image_transformation.so
└── resnet50-onnx
    └── 1
        └── resnet50-caffe2-v1-9.onnx

Start the OVMS container with a configuration file option:

docker run -d -u $(id -u):$(id -g) -v $(pwd)/workspace:/workspace -p 9001:9001 openvino/model_server:latest \
--config_path /workspace/config.json --port 9001

The onnx_model_demo.py script can run inference both with and without performing preprocessing. Since in this variant preprocessing is done by the model server (via custom node), there’s no need to perform any image preprocessing on the client side. In that case, run without --run_preprocessing option. See preprocessing function run in the client.

Run the client without preprocessing:

python3 onnx_model_demo.py --service_url localhost:9001
Running without preprocessing on client side
Class is with highest score: 309
Detected class name: bee

Node parameters explanation

Additional preprocessing step applies a division and an subtraction to each pixel value in the image. This calculation is configured by passing two parameters to image transformation custom node in config.json :

"params": {
  ...
  "mean_values": "[123.675,116.28,103.53]",
  "scale_values": "[58.395,57.12,57.375]",
  ...
}

For each pixel, the custom node subtractes 123.675 from blue value, 116.28 from green value and 103.53 from red value. Next, it divides in the same color order using 58.395, 57.12, 57.375 values. This way we match the image data to the input required by onnx model.