Model Server Parameters#
Model Configuration Options#
Option |
Value format |
Description |
---|---|---|
|
|
Model name exposed over gRPC and REST API.(use |
|
|
If using a Google Cloud Storage, Azure Storage or S3 path, see cloud storage guide. The path may look as follows: |
|
|
|
|
|
Optional. By default, the batch size is derived from the model, defined through the OpenVINO Model Optimizer. |
|
|
|
|
|
Optional. The model version policy lets you decide which versions of a model that the OpenVINO Model Server is to serve. By default, the server serves the latest version. One reason to use this argument is to control the server memory consumption.The accepted format is in json or string. Examples: |
|
|
List of device plugin parameters. For full list refer to OpenVINO documentation and performance tuning guide. Example: |
|
|
The size of internal request queue. When set to 0 or no value is set value is calculated automatically based on available resources. |
|
|
Device name to be used to execute inference operations. Accepted values are: |
|
|
If set to true, model is loaded as stateful. |
|
|
If set to true, model will be subject to periodic sequence cleaner scans. See idle sequence cleanup. |
|
|
Determines how many sequences can be handled concurrently by a model instance. |
|
|
If set to true, model server will apply low latency transformation on model load. |
|
|
Flag enabling metrics endpoint on rest_port. |
|
|
Comma separated list of metrics. If unset, only default metrics will be enabled. |
Note : Specifying config_path is mutually exclusive with putting model parameters in the CLI (serving multiple models).
Option |
Value format |
Description |
---|---|---|
|
|
Absolute path to json configuration file |
Server configuration options#
Configuration options for the server are defined only via command-line options and determine configuration common for all served models.
Option |
Value format |
Description |
---|---|---|
|
|
Number of the port used by gRPC sever. |
|
|
Number of the port used by HTTP server (if not provided or set to 0, HTTP server will not be launched). |
|
|
Network interface address or a hostname, to which gRPC server will bind to. Default: all interfaces: 0.0.0.0 |
|
|
Network interface address or a hostname, to which REST server will bind to. Default: all interfaces: 0.0.0.0 |
|
|
Number of the gRPC server instances (must be from 1 to CPU core count). Default value is 1 and it’s optimal for most use cases. Consider setting higher value while expecting heavy load. |
|
|
Number of HTTP server threads. Effective when |
|
|
Time interval between config and model versions changes detection in seconds. Default value is 1. Zero value disables changes monitoring. |
|
|
Time interval (in minutes) between next sequence cleaner scans. Sequences of the models that are subjects to idle sequence cleanup that have been inactive since the last scan are removed. Zero value disables sequence cleaner. See idle sequence cleanup. It also sets the schedule for releasing free memory from the heap. |
|
|
Time interval (in seconds) between two consecutive resources cleanup scans. Default is 1. Must be greater than 0. See custom node development. |
|
|
Optional path to a library with custom layers implementation. |
|
|
Serving logging level |
|
|
Optional path to the log file. |
|
|
Path to the model cache storage. Caching will be enabled if this parameter is defined or the default path /opt/cache exists |
|
|
A comma separated list of arguments to be passed to the grpc server. (e.g. grpc.max_connection_age_ms=2000) |
|
|
Maximum number of threads which can be used by the grpc server. Default value depends on number of CPUs. |
|
|
GRPC server buffer memory quota. Default value set to 2147483648 (2GB). |
|
|
Shows help message and exit |
|
|
Shows binary version |