Configuration file declares validation process. Every model has to have entry in models
list. Each entry has to contain distinct name
, launchers
and datasets
sections.
Example:
Also there are composite models which consist of several parts (models) and the accuracy measurement requires building the pipeline from these parts. Thus, the evaluation is performed by sequentially executing a set of models and impossible to evaluate them independently. Each composite model has to have entry in evaluations
list. Each entry should contain distinct name
, module
and module_config
. module_config
has to consist of network_info
,launchers
and datasets
fields. Custom evaluators are used for such models. More information about defining and using your own evaluator or an existing one can be found in Custom Evaluators Guide
Example:
Predefined configuration file accuracy-check.yml
for each Open Model Zoo model can be found in the model directory.
<model_name>.yml
file, which is located in current configs
folder, is a link to accuracy-check.yml
for <model_name>
model.
Example:
alexnet.yml is a link for configuration file accuracy-check.yml for alexnet model.
-c, --config
command line.-m, --models
command line option.-d, --definitions
. In order, if you want use definitions file in quantization via Post Training Optimization Toolkit, you should use environment variable DEFINITIONS_FILE
for specifying path to definitions.data_source
is specified can be provided via -s, --source
command line. If you want to evaluate models using well-known datasets, you need to organize folders with validation datasets in a certain way. More detailed information about dataset preparation you can find in Dataset Preparation Guide. In order, if you want use data source in quantization via Post Training Optimization Toolkit, you should use environment variable DATA_DIR
for specifying path to root of directories with datasets.annotation
and dataset_meta
are specified can be provided via -a, --annotations
command line. Annotation and dataset_meta (if required) will be stored to this directory after annotation conversion step if they do not exist and can be used for the next running to skip annotation conversion. Detailed information about annotation conversion you can find in Annotation Conversion Guide.--model_attributes
command line option (usually, it is the model directory).-td, --target_devices
command line option. Several devices should be separated by spaces (e.g. -td CPU GPU).-tf, --target_framework
command line option. Otherwise, if the option is not provided evaluation will be launched with all frameworks mentioned in the configuration file.See how to evaluate model with using predefined configuration file for densenet-121-tf model.
OMZ_ROOT
- root of Open Model Zoo projectDATASET_DIR
- root directory with datasetMODEL_DIR
- root directory with modelOPENVINO_DIR
- root directory with installed the OpenVINO™ toolkit