Directed Acyclic Graph (DAG) Scheduler¶

Introduction¶

The Directed Acyclic Graph (DAG) Scheduler makes it possible to create a pipeline of models for execution in a single client request. The pipeline is a Directed Acyclic Graph with different nodes which define how to process each step of predict request. By using a pipeline, there is no need to return intermediate results of every model to the client. This allows avoiding the network overhead by minimizing the number of requests sent to the Model Server. Each model output can be mapped to another model input. Since intermediate results are kept in the server’s RAM these can be reused by subsequent inferences which reduce overall latency.

This guide gives information about:

Node Types
Configuration file
Using the pipelines
Pipelines examples
Current Limitations

Node Types¶

Auxiliary Node Types¶

There are two special kinds of nodes - Request and Response node. Both of them are predefined and included in every pipeline definition you create:

Request node
- This node defines which inputs are required to be sent via gRPC/REST request for pipeline usage. You can refer to it by node name: request.
Response node
- This node defines which outputs will be fetched from the final pipeline state and packed into gRPC/REST response. You cannot refer to it in your pipeline configuration since it is the pipeline final stage. To define final outputs fill outputs field.

Deep Learning node type¶

DL model - this node contains underlying OpenVINO model and performs inference on the selected target device. This can be defined in the configuration file. Each model input needs to be mapped to some node’s data_item - input from gRPC/REST request or another DL model output. Outputs of the node may be mapped to another node’s inputs or the response node, meaning it will be exposed in gRPC/REST response.

Custom node type¶

custom - that node can be used to implement all operations on the data which can not be handled by the neural network model. It is represented by a C++ dynamic library implementing OVMS API defined in custom_node_interface.h. Custom nodes can run the data processing using OpenCV, which is included in OVMS, or include other third-party components. Custom node libraries are loaded into OVMS by adding their definition to the pipeline configuration. The configuration includes a path to the compiled binary with the .so extension. Custom nodes are not versioned, meaning one custom node library is bound to one name. To load another version, another name needs to be used.

OpenVINO Model Server docker image comes with prebuilt custom nodes that you can use out-of-the-box in your pipeline. See the list of built-in custom nodes and learn more about developing custom nodes yourself in the custom node developer guide.

Demultiplexing data¶

During the pipeline execution, it is possible to split a request with multiple batches into a set of branches with a single batch. That way a model configured with a batch size 1, can process requests with arbitrary batch size. Internally, OVMS demultiplexer will divide the data, process them in parallel and combine the results.

De-multiplication of the node output is enabled in the configuration file by adding demultiply_count. It assumes the batches are combined on the first dimension which is dropped after splitting. For example:

a node returns output with shape [8,1,3,224,224]
demuliplexer creates 8 requests with shape [1,3,224,224]
next model processes in parallel 8 requests with output shape [1,1001] each.
results are combined into a single output with shape [8,1,1001]

Learn more about demuliplexing

Configuration file¶

Pipelines configuration is to be placed in the same json file like the models config file. While models are defined in section model_config_list, pipelines are configured in the pipeline_config_list section. Nodes in the pipelines can reference only the models configured in model_config_list section.

Basic pipeline section template is depicted below:

{
    "model_config_list": [...],
    "custom_node_library_config_list": [
        {
            "name": "custom_node_lib",
            "base_path": "/libs/libcustom_node.so"
        }
    ],
    "pipeline_config_list": [
        {
            "name": "<pipeline name>",
            "inputs": ["<input1>",...],
            "nodes": [
                {
                    "name": "<node name>",
                    "model_name": <reference to the model>,
                    "type": "DL model",
                    "inputs": [
                        {"input": {"node_name": "request",  # reference to pipeline input
                                   "data_item": "<input1>"}}  # input name from the request
                    ],
                    "outputs": [  # mapping the model output name to node output name
                        {"data_item": "<model output>",
                         "alias": "<node output name>"}
                    ]
                },
                {
                    "name": "custon_node_name",
                    "library_name": "custom_node_lib",
                    "type": "custom",
                    "params": {
                        "param1": "value1",
                        "param2": "value2",
                    },
                    "inputs": [
                        {"input": {"node_name": "request",  # reference to pipeline input
                                   "data_item": "<input1>"}}  # input name from the request
                    ],
                    "outputs": [
                        {"data_item": "<library_output>",
                            "alias": "<node_output>"},
                    ]
                }
            ],
            "outputs": [      # pipeline outputs
                {"label": {"node_name": "<node to return results>",
                           "data_item": "<node output name to return results>"}}
            ]
        }
    ]
}

Pipeline configuration options explained¶

Option	Type	Description	Required
`"name"`	string	Pipeline identifier related to name field specified in gRPC/REST request	Yes
`"inputs"`	array	Defines input names required to be present in gRPC/REST request	Yes
`"outputs"`	array	Defines outputs (data items) to be retrieved from intermediate results (nodes) after pipeline execution completed for final gRPC/REST response to the client	Yes
`"nodes"`	array	Declares nodes used in pipeline and its connections	Yes

Node Options¶

Option	Type	Description	Required
`"name"`	string	Node name so you can refer to it from other nodes	Yes
`"model_name"`	string	You can specify underlying model (needs to be defined in `model_config_list` ), available only for `DL model` nodes	required for `DL model` nodes
`"version"`	integer	You can specify a model version for inference, available only for `DL model` nodes	No
`"type"`	string	Node kind, currently there are 2 types available: `DL model` and `custom`	Yes
`"demultiply_count"`	integer	Splits node outputs to desired chunks and branches pipeline execution	No
`"gather_from_node"`	string	Setups node to converge pipeline and collect results into one input before execution	No
`"inputs"`	array	Defines the list of input/output mappings between this and dependency nodes, IMPORTANT : Please note that output shape, precision, and layout of previous node/request needs to match input of current node’s model	Yes
`"outputs"`	array	Defines model output name alias mapping - you can rename model output names for easier use in subsequent nodes	Yes

Node Input Options¶

Option	Type	Description	Required
`"node_name"`	string	Defines which node we refer to	Yes
`"data_item"`	string	Defines which resource of the node we point to	Yes

Node Output Options¶

Option	Type	Description	Required
`"data_item"`	string	Is the name of resource exposed by node - for `DL model` nodes it means model output	Yes
`"alias"`	string	Is a name assigned to a data item, makes it easier to refer to results of this node in subsequent nodes	Yes

Custom Node Options¶

In case the pipeline definition includes the custom node, the configuration file must include custom_node_library_config_list section. It includes:

Option	Type	Description	Required
`"name"`	string	The name of the custom node library - it will be used as a reference in the custom node pipeline definition	Yes
`"base_path"`	string	Path the dynamic library with the custom node implementation	Yes

Custom node definition in a pipeline configuration is similar to a model node. Node inputs and outputs are configurable in the same way. Custom node functions are just like a standard node in that respect. The differences are in the extra parameters:

Option	Type	Description	Required
`"library_name"`	string	Name of the custom node library defined in `custom_node_library_config_list`	Yes
`"type"`	string	Must be set to `custom`	Yes
`"params"`	json object with string values	a list of parameters and their values which could be used in the custom node implementation	No

Using Pipelines¶

Pipelines can use the same API as the models. There are exactly the same calls for running the predictions. The request format must match the pipeline definition inputs.

The pipeline configuration can be queried using gRPC GetModelMetadata calls and REST Metadata. It returns the definition of the pipelines inputs and outputs.

Similarly, pipelines can be queried for their state using the calls GetModelStatus and REST Model Status

The only difference in using the pipelines and individual models is in version management. In all calls to the pipelines, the version parameter is ignored. Pipelines are not versioned. Though, they can reference a particular version of the models in the graph.

Pipelines Examples¶

Single face analysis with combined models

Multiple vehicles analysis using demultiplexer with model_zoo_object_detection example custom node

Optical Character Recognition pipeline with east_ocr example custom node

Horizontal Text Detection pipeline with horizontal_ocr example custom node

Current limitations¶

Models with “auto” batch size or shape cannot be referenced in pipeline
Connected inputs and output for subsequent node models need to match each other in terms of data shape, precision and layout - there is no automatic conversion between input/output model precisions or layouts. This limitation can be addressed with --shape and --layout model configuration or with a custom node to transform the data as required to match the expected data format.
REST requests with no named format (JSON body with one unnamed input) are not supported