Directed Acyclic Graph (DAG) Scheduler#
Introduction#
The Directed Acyclic Graph (DAG) Scheduler makes it possible to create a pipeline of models for execution in a single client request. The pipeline is a Directed Acyclic Graph with different nodes which define how to process each step of predict request. By using a pipeline, there is no need to return intermediate results of every model to the client. This allows avoiding the network overhead by minimizing the number of requests sent to the Model Server. Each model output can be mapped to another model input. Since intermediate results are kept in the server’s RAM these can be reused by subsequent inferences which reduce overall latency.
This guide gives information about:
Node Types#
Auxiliary Node Types#
There are two special kinds of nodes - Request and Response node. Both of them are predefined and included in every pipeline definition you create:
Request node
This node defines which inputs are required to be sent via gRPC/REST request for pipeline usage. You can refer to it by node name:
request
.
Response node
This node defines which outputs will be fetched from the final pipeline state and packed into gRPC/REST response. You cannot refer to it in your pipeline configuration since it is the pipeline final stage. To define final outputs fill
outputs
field.
Deep Learning node type#
DL model - this node contains underlying OpenVINO™ model and performs inference on the selected target device. This can be defined in the configuration file. Each model input needs to be mapped to some node’s
data_item
- input from gRPC/RESTrequest
or anotherDL model
output. Outputs of the node may be mapped to another node’s inputs or theresponse
node, meaning it will be exposed in gRPC/REST response.
Custom node type#
custom - that node can be used to implement all operations on the data which can not be handled by the neural network model. It is represented by a C++ dynamic library implementing OVMS API defined in custom_node_interface.h. Custom nodes can run the data processing using OpenCV, which is included in OVMS, or include other third-party components. Custom node libraries are loaded into OVMS by adding their definition to the pipeline configuration. The configuration includes a path to the compiled binary with the
.so
extension. Custom nodes are not versioned, meaning one custom node library is bound to one name. To load another version, another name needs to be used.OpenVINO Model Server docker image comes with prebuilt custom nodes that you can use out-of-the-box in your pipeline. See the list of built-in custom nodes and learn more about developing custom nodes yourself in the custom node developer guide.
Demultiplexing data#
During the pipeline execution, it is possible to split a request with multiple batches into a set of branches with a single batch. That way a model configured with a batch size 1, can process requests with arbitrary batch size. Internally, OVMS demultiplexer will divide the data, process them in parallel and combine the results.
De-multiplication of the node output is enabled in the configuration file by adding demultiply_count
.
It assumes the batches are combined on the first dimension which is dropped after splitting. For example:
a node returns output with shape
[8,1,3,224,224]
demultiplexer creates 8 requests with shape
[1,3,224,224]
next model processes in parallel 8 requests with output shape
[1,1001]
each.results are combined into a single output with shape
[8,1,1001]
Configuration file#
Pipelines configuration is to be placed in the same json file like the
models config file.
While models are defined in section model_config_list
, pipelines are configured in
the pipeline_config_list
section.
Nodes in the pipelines can reference only the models configured in model_config_list section.
Basic pipeline section template is depicted below:
{
"model_config_list": [...],
"custom_node_library_config_list": [
{
"name": "custom_node_lib",
"base_path": "/libs/libcustom_node.so"
}
],
"pipeline_config_list": [
{
"name": "<pipeline name>",
"inputs": ["<input1>",...],
"nodes": [
{
"name": "<node name>",
"model_name": <reference to the model>,
"type": "DL model",
"inputs": [
{"input": {"node_name": "request", # reference to pipeline input
"data_item": "<input1>"}} # input name from the request
],
"outputs": [ # mapping the model output name to node output name
{"data_item": "<model output>",
"alias": "<node output name>"}
]
},
{
"name": "custom_node_name",
"library_name": "custom_node_lib",
"type": "custom",
"params": {
"param1": "value1",
"param2": "value2",
},
"inputs": [
{"input": {"node_name": "request", # reference to pipeline input
"data_item": "<input1>"}} # input name from the request
],
"outputs": [
{"data_item": "<library_output>",
"alias": "<node_output>"},
]
}
],
"outputs": [ # pipeline outputs
{"label": {"node_name": "<node to return results>",
"data_item": "<node output name to return results>"}}
]
}
]
}
Pipeline configuration options explained#
Option |
Type |
Description |
Required |
---|---|---|---|
|
string |
Pipeline identifier related to name field specified in gRPC/REST request |
Yes |
|
array |
Defines input names required to be present in gRPC/REST request |
Yes |
|
array |
Defines outputs (data items) to be retrieved from intermediate results (nodes) after pipeline execution completed for final gRPC/REST response to the client |
Yes |
|
array |
Declares nodes used in pipeline and its connections |
Yes |
Node Options#
Option |
Type |
Description |
Required |
---|---|---|---|
|
string |
Node name so you can refer to it from other nodes |
Yes |
|
string |
You can specify underlying model (needs to be defined in |
required for |
|
integer |
You can specify a model version for inference, available only for |
No |
|
string |
Node kind, currently there are 2 types available: |
Yes |
|
integer |
Splits node outputs to desired chunks and branches pipeline execution |
No |
|
string |
Setups node to converge pipeline and collect results into one input before execution |
No |
|
array |
Defines the list of input/output mappings between this and dependency nodes, IMPORTANT: Please note that output shape, precision, and layout of previous node/request needs to match input of current node’s model |
Yes |
|
array |
Defines model output name alias mapping - you can rename model output names for easier use in subsequent nodes |
Yes |
Node Input Options#
Option |
Type |
Description |
Required |
---|---|---|---|
|
string |
Defines which node we refer to |
Yes |
|
string |
Defines which resource of the node we point to |
Yes |
Node Output Options#
Option |
Type |
Description |
Required |
---|---|---|---|
|
string |
Is the name of resource exposed by node - for |
Yes |
|
string |
Is a name assigned to a data item, makes it easier to refer to results of this node in subsequent nodes |
Yes |
Custom Node Options#
In case the pipeline definition includes the custom node, the configuration file must include custom_node_library_config_list
section. It includes:
Option |
Type |
Description |
Required |
---|---|---|---|
|
string |
The name of the custom node library - it will be used as a reference in the custom node pipeline definition |
Yes |
|
string |
Path the dynamic library with the custom node implementation |
Yes |
Custom node definition in a pipeline configuration is similar to a model node. Node inputs and outputs are configurable in the same way. Custom node functions are just like a standard node in that respect. The differences are in the extra parameters:
Option |
Type |
Description |
Required |
---|---|---|---|
|
string |
Name of the custom node library defined in |
Yes |
|
string |
Must be set to |
Yes |
|
json object with string values |
a list of parameters and their values which could be used in the custom node implementation |
No |
Using Pipelines#
Pipelines can use the same API as the models. There are exactly the same calls for running the predictions. The request format must match the pipeline definition inputs.
The pipeline configuration can be queried using gRPC GetModelMetadata calls and REST Metadata. It returns the definition of the pipelines inputs and outputs.
Similarly, pipelines can be queried for their state using the calls GetModelStatus and REST Model Status
The only difference in using the pipelines and individual models is in version management. In all calls to the pipelines, the version parameter is ignored. Pipelines are not versioned. Though, they can reference a particular version of the models in the graph.
Pipelines Examples#
Single face analysis with combined models
Multiple vehicles analysis using demultiplexer with model_zoo_object_detection example custom node
Optical Character Recognition pipeline with east_ocr example custom node
Horizontal Text Detection pipeline with horizontal_ocr example custom node
Current limitations#
Models with “auto” batch size or shape cannot be referenced in pipeline
Connected inputs and output for subsequent node models need to match each other in terms of data shape, precision and layout - there is no automatic conversion between input/output model precisions or layouts. This limitation can be addressed with
--shape
and--layout
model configuration or with a custom node to transform the data as required to match the expected data format.REST requests with no named format (JSON body with one unnamed input) are not supported