Image Classification Demo (C++)¶
This demo provides 2 clients:
classification_client_sync - simple client using synchronous gRPC API, testing accuracy of classification models
classification_client_async_benchmark - client using asynchronous gRPC API, testing accuracy and performance with real image data
To build the clients, run make command in this directory. It will build docker image named ovms_cpp_image_classification with all dependencies.
The example clients image also contains test images required for accuracy measurements. It is also possible to use custom images.
git clone https://github.com/openvinotoolkit/model_server.git
cd model_server/demos/image_classification/cpp
make
Note: In this directory you can only see the code specific to the benchmark client. The code shared with other C++ demos as well as all building utilities are placed in the common C++ directory.
Prepare classification model¶
Start OVMS with resnet50-binary model:
curl -L --create-dir https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.1/models_bin/2/resnet50-binary-0001/FP32-INT1/resnet50-binary-0001.bin -o resnet50-binary/1/model.bin https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.1/models_bin/2/resnet50-binary-0001/FP32-INT1/resnet50-binary-0001.xml -o resnet50-binary/1/model.xml
Client requesting prediction synchronously¶
The client sends requests synchronously and displays latency for each request.
You can specify number of iterations and layout: nchw, nhwc or binary.
Each request contains image in selected format.
The client also tests server responses for accuracy.
Prepare the server¶
docker run -d -u $(id -u):$(id -g) -v $(pwd)/resnet50-binary:/model -p 9001:9001 openvino/model_server:latest \
--model_path /model --model_name resnet --port 9001 --layout NHWC:NCHW
Start the client:¶
docker run --rm --network host -e "no_proxy=localhost" -v ${PWD}/input_images.txt:/clients/input_images.txt ovms_cpp_image_classification ./classification_client_sync --grpc_port=9001 --iterations=10 --layout="binary"
Address: localhost
Images list path: input_images.txt
Layout: binary
call predict ok
call predict time: 6ms
outputs size is 1
call predict ok
call predict time: 5ms
outputs size is 1
call predict ok
call predict time: 4ms
outputs size is 1
call predict ok
call predict time: 4ms
outputs size is 1
call predict ok
call predict time: 4ms
outputs size is 1
call predict ok
call predict time: 4ms
outputs size is 1
call predict ok
call predict time: 5ms
outputs size is 1
call predict ok
call predict time: 4ms
outputs size is 1
call predict ok
call predict time: 4ms
outputs size is 1
call predict ok
call predict time: 5ms
outputs size is 1
Overall accuracy: 100%
Total time divided by number of requests: 5ms
Clients requesting predictions asynchronously¶
The client sends requests asynchronously to mimic parallel clients scenario. There are plenty of parameters to configure those clients.
name |
description |
default |
available with synthetic data |
|---|---|---|---|
grpc_address |
url to grpc service |
localhost |
yes |
grpc_port |
port to grpc service |
9000 |
yes |
model_name |
model name to request |
resnet |
yes |
input_name |
input tensor name with image |
0 |
no, deduced automatically |
output_name |
output tensor name with classification result |
1463 |
no |
iterations |
number of requests to be send by each producer thread |
10 |
yes |
batch_size |
batch size of each iteration |
1 |
no, deduced automatically |
images_list |
path to a file with a list of labeled images |
input_images.txt |
no |
layout |
binary, nhwc or nchw |
nchw |
no, deduced automatically |
producers |
number of threads asynchronously scheduling prediction |
1 |
yes |
consumers |
number of threads receiving responses |
8 |
yes |
max_parallel_requests |
maximum number of parallel inference requests; 0=no limit |
100 |
yes |
benchmark_mode |
1 removes pre/post-processing and logging; 0 enables accuracy measurement |
0 |
no |
Async client with real image data¶
Prepare the server¶
docker run -d -u $(id -u):$(id -g) -v $(pwd)/resnet50-binary:/model -p 9001:9001 openvino/model_server:latest \
--model_path /model --model_name resnet --port 9001 --layout NCHW
Start the client:¶
docker run --rm --network host -e "no_proxy=localhost" -v ${PWD}/input_images.txt:/clients/input_images.txt ovms_cpp_image_classification ./classification_client_async_benchmark --grpc_port=9001 --layout="nchw" --iterations=2000 --batch_size=1 --max_parallel_requests=100 --consumers=8 --producers=1 --benchmark_mode=1
Address: localhost:9001
Model name: resnet
Images list path: input_images.txt
Running the workload...
========================
Summary
========================
Benchmark mode: True
Accuracy: N/A
Total time: 1976ms
Total iterations: 2000
Layout: nchw
Batch size: 1
Producer threads: 1
Consumer threads: 8
Max parallel requests: 100
Avg FPS: 1012.15