Custom Evaluators for Accuracy Checker

Standard Accuracy Checker validation pipeline: Annotation Reading -> Data Reading -> Preprocessing -> Inference -> Postprocessing -> Metrics. In some cases it can be unsuitable (e.g. if you have sequence of models). You are able to customize validation pipeline using own evaluator. Suggested approach based on writing python module which will describe validation approach


Adding new evaluator process similar with adding any other entities in the tool. Custom evaluator is the class which should be inherited from BaseEvaluator and overwrite all abstract methods.

The most important methods for overwriting:


Each custom evaluation config should start with keyword evaluation and contain: