Online Configuration Updates

Updating Configuration File

OpenVINO Model Server monitors changes to the configuration file and applies required modifications during runtime using two different methods:

  1. Automatically, with an interval defined by the parameter --file_system_poll_wait_seconds. (introduced in version 2021.1)

  2. On demand, using the Config Reload API. (introduced in version 2021.3)

Configuration reload triggers the following operations:

  • new model(s) or DAG(s) are added to the configuration file, loaded and served.

  • changes to the configured model storage (e.g. new model version is added) are applied.

  • changes to the configuration of deployed models and DAGs are applied.

  • all model versions will be reloaded when there is a change to the model configuration.

  • when a deployed model or DAG is deleted from config.json, it will be unloaded completely from the server after already running inference operations have been completed.

  • DAGs that depend on changed or removed models are reloaded.

  • changes to custom loaders and custom node library configs are applied.

Model Server behavior in case of errors during configuration reloading:

  • if a new config.json is not compliant with JSON schema, no changes are applied to the served models.

  • if the new model, DAG or custom loader has an invalid configuration, it will be ignored until the next configuration reload. Configurations may be invalid due to incorrect paths (leading to non-existent directories), forbidden values in the config, invalid DAG structure (e.g. cycle found in a graph), etc.

  • an error occurs when a model, DAG or custom loader is reloading but does not prevent the reload of the remaining updated models.

  • errors from configuration reload are triggered internally and saved in the logs. If Config Reload API is used, the response will also contain an error message.