String output model demo#


This demo demonstrates example deployment of a model with output precision ov::element::string. The output text is serialized into corresponding fields in gRPC proto/REST body. This allows the client to consume the text directly and avoid the process of label mapping or detokenization.

Download and prepare MobileNet model using from TensorFlow#

The script below is downloading a public MobileNet model trained on the ImageNet data. The original model accepts on input the image array in the range of 0-1 and returns probabilities for all the trained classes. We are adding to the model preprocessing function changing the input data range to 0-255 and also postprocessing function which is retrieving the most likely label name as a string. This is a very handy functionality because it allows us to export the model with the included pre/post processing functions as the model layers. The client just receives the string data with the label name for the classified image.

pip install -r requirements.txt
rm model/1/fingerprint.pb

tree model
└── 1
    ├── assets
    ├── saved_model.pb
    └── variables
        └── variables.index

Start the OVMS container:#

docker run -d -u $(id -u):$(id -g) -v $(pwd):/workspace -p 8000:8000 openvino/model_server:latest \
--model_path /workspace/model --model_name mobile_net --rest_port 8000

Send request#

Use example client to send requests containing images via KServ REST API:


Request may be sent also using other APIs (KServ GRPC, TFS). In this sections you can find short code samples how to do this:

Expected output#

Start processing:
        Model name: mobile_net
../common/static/images/airliner.jpeg classified  as airliner
Iteration 0; Processing time: 31.09 ms; speed 32.16 fps

../common/static/images/arctic-fox.jpeg classified  as Arctic fox
Iteration 0; Processing time: 5.27 ms; speed 189.75 fps

../common/static/images/bee.jpeg classified  as bee
Iteration 0; Processing time: 3.02 ms; speed 331.46 fps

../common/static/images/golden_retriever.jpeg classified  as clumber
Iteration 0; Processing time: 3.12 ms; speed 320.82 fps

../common/static/images/gorilla.jpeg classified  as gorilla
Iteration 0; Processing time: 3.04 ms; speed 329.06 fps

../common/static/images/magnetic_compass.jpeg classified  as magnetic compass
Iteration 0; Processing time: 3.10 ms; speed 323.00 fps

../common/static/images/peacock.jpeg classified  as peacock
Iteration 0; Processing time: 3.24 ms; speed 308.17 fps

../common/static/images/pelican.jpeg classified  as pelican
Iteration 0; Processing time: 3.17 ms; speed 315.36 fps

../common/static/images/snail.jpeg classified  as snail
Iteration 0; Processing time: 3.06 ms; speed 327.33 fps

../common/static/images/zebra.jpeg classified  as zebra
Iteration 0; Processing time: 4.19 ms; speed 238.72 fps