Image Classification Demo (C++)#

This demo provides 2 clients:

  • classification_client_sync - simple client using synchronous gRPC API, testing accuracy of classification models

  • classification_client_async_benchmark - client using asynchronous gRPC API, testing accuracy and performance with real image data

To build the clients, run make command in this directory. It will build docker image named ovms_cpp_image_classification with all dependencies. The example clients image also contains test images required for accuracy measurements. It is also possible to use custom images.

git clone https://github.com/openvinotoolkit/model_server.git
cd model_server/demos/image_classification/cpp
make

Note: In this directory you can only see the code specific to the benchmark client. The code shared with other C++ demos as well as all building utilities are placed in the common C++ directory.

Prepare classification model#

Start OVMS with resnet50-binary model:

curl -L --create-dir https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.1/models_bin/2/resnet50-binary-0001/FP32-INT1/resnet50-binary-0001.bin -o resnet50-binary/1/model.bin https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.1/models_bin/2/resnet50-binary-0001/FP32-INT1/resnet50-binary-0001.xml -o resnet50-binary/1/model.xml

Client requesting prediction synchronously#

The client sends requests synchronously and displays latency for each request. You can specify number of iterations and layout: nchw, nhwc or binary. Each request contains image in selected format. The client also tests server responses for accuracy.

Prepare the server#

docker run -d -u $(id -u):$(id -g) -v $(pwd)/resnet50-binary:/model -p 9001:9001 openvino/model_server:latest \
--model_path /model --model_name resnet --port 9001 --layout NHWC:NCHW

Start the client:#

docker run --rm --network host -e "no_proxy=localhost" -v ${PWD}/input_images.txt:/clients/input_images.txt ovms_cpp_image_classification ./classification_client_sync --grpc_port=9001 --iterations=10 --layout="binary"

Address: localhost
Images list path: input_images.txt
Layout: binary
call predict ok
call predict time: 6ms
outputs size is 1
call predict ok
call predict time: 5ms
outputs size is 1
call predict ok
call predict time: 4ms
outputs size is 1
call predict ok
call predict time: 4ms
outputs size is 1
call predict ok
call predict time: 4ms
outputs size is 1
call predict ok
call predict time: 4ms
outputs size is 1
call predict ok
call predict time: 5ms
outputs size is 1
call predict ok
call predict time: 4ms
outputs size is 1
call predict ok
call predict time: 4ms
outputs size is 1
call predict ok
call predict time: 5ms
outputs size is 1
Overall accuracy: 100%
Total time divided by number of requests: 5ms

Clients requesting predictions asynchronously#

The client sends requests asynchronously to mimic parallel clients scenario. There are plenty of parameters to configure those clients.

name

description

default

available with synthetic data

grpc_address

url to grpc service

localhost

yes

grpc_port

port to grpc service

9000

yes

model_name

model name to request

resnet

yes

input_name

input tensor name with image

0

no, deduced automatically

output_name

output tensor name with classification result

1463

no

iterations

number of requests to be send by each producer thread

10

yes

batch_size

batch size of each iteration

1

no, deduced automatically

images_list

path to a file with a list of labeled images

input_images.txt

no

layout

binary, nhwc or nchw

nchw

no, deduced automatically

producers

number of threads asynchronously scheduling prediction

1

yes

consumers

number of threads receiving responses

8

yes

max_parallel_requests

maximum number of parallel inference requests; 0=no limit

100

yes

benchmark_mode

1 removes pre/post-processing and logging; 0 enables accuracy measurement

0

no

Async client with real image data#

Prepare the server#

docker run -d -u $(id -u):$(id -g) -v $(pwd)/resnet50-binary:/model -p 9001:9001 openvino/model_server:latest \
--model_path /model --model_name resnet --port 9001 --layout NCHW

Start the client:#

docker run --rm --network host -e "no_proxy=localhost"  -v ${PWD}/input_images.txt:/clients/input_images.txt ovms_cpp_image_classification ./classification_client_async_benchmark --grpc_port=9001 --layout="nchw" --iterations=2000 --batch_size=1 --max_parallel_requests=100 --consumers=8 --producers=1 --benchmark_mode=1

Address: localhost:9001
Model name: resnet
Images list path: input_images.txt

Running the workload...
========================
        Summary
========================
Benchmark mode: True
Accuracy: N/A
Total time: 1976ms
Total iterations: 2000
Layout: nchw
Batch size: 1
Producer threads: 1
Consumer threads: 8
Max parallel requests: 100
Avg FPS: 1012.15