The Inference Engine workflow involves the creation of custom kernels and either custom or existing layers.
A Layer is a convolutional neural network (CNN) building block implemented in the training framework, for example, Convolution
in Caffe*. A Kernel is defined as the corresponding implementation in Inference Engine.
Please refer to the Custom Layers in the Model Optimizer section for the details of how a mapping between framework layers and Inference Engine kernels is registered.
In short, you can plug your own kernel implementations into the Inference Engine and map them to the layers in the original framework.
The rest of the section covers custom kernels and how do you integrate them into the Inference Engine.
Every sample uses the Inference Engine API to load custom kernels depending on the device type. Specifically, for the CPU, it is a shared library that exports certain interface that registers the kernels. For GPU or MYRIAD, it is an .xml
file that lists the kernels along with parameters that the kernels accept and how these map to the specific Intermediate Representation (IR) values.
You can find the examples of CPU-targeted kernels in the <INSTALL_DIR>/deployment_tools/inference_engine/src/extension
directory. You can also use as an example global GPU kernels delivered with the OpenVINO toolkit.
Several GPU-targeted kernels are also added to the binaries upon samples compilation so that the samples application can easy load them. Refer to the cldnn_global_custom_kernels
folder in GPU plugin installation directory.
The GPU codepath abstracts many details about OpenCL™. You need to provide the kernel code in the OpenCL C and the configuration file that connects the kernel and its parameters to the parameters of the layer.
There are two options of using custom layer configuration file:
cldnn_global_custom_kernels/cldnn_global_custom_kernels.xml
file (hosted in the <INSTALL_DIR> /deployment_tools/inference_engine/bin/intel64/{Debug/Release}
folder)IInferencePlugin::SetConfig()
method from your application with the PluginConfigParams::KEY_CONFIG_FILE
key and the configuration file name as the value before loading the network that uses custom layers to the plugin: All Inference Engine samples (except trivial hello_classification
) feature a dedicated command-line option -c
to load custom kernels. For example, to load custom layers for the classification sample:
The configuration file is expected to follow the .xml
file structure with a node of type CustomLayer
for every custom layer you provide.
The following definitions will use the notations:
CustomLayer Node and Sub-node Structure
CustomLayer
node contains the entire configuration for a single custom layer.
Attribute Name | # | Description |
---|---|---|
name |
(1) | The name of the layer type to be used. This name should be identical to the type used in the IR. |
type |
(1) | Must be SimpleGPU |
version |
(1) | Must be 1 |
Sub-nodes: Kernel
(1), Buffers
(1), CompilerOptions
(0+), WorkSizes
(0/1)
Kernel Node and Sub-node Structure
Kernel
node contains all kernel source code configuration. No kernel node structure exists.
Sub-nodes: Source
(1+), Define
(0+)
Source Node and Sub-node Structure
Source
node points to a single OpenCL source file.
Attribute Name | # | Description |
---|---|---|
filename |
(1) | Name of the file containing OpenCL source code. Notice that path is relative to your executable. Multiple source nodes will have their sources concatenated in order. |
Sub-nodes: None
Define Node and Sub-node Structure
Define
node configures a single #define
instruction to be added to the sources during compilation (JIT).
Attribute Name | # | Description |
---|---|---|
name |
(1) | The name of the defined JIT. For static constants, this can include the value as well (taken as a string). |
param |
(0/1) | This parameter value will be used as the value of this JIT definition. |
type |
(0/1) | The parameter type. Accepted values: int , float , and int[] , float[] for arrays. |
default |
(0/1) | The default value to be used if the specified parameters is missing from the layer in the IR. |
Sub-nodes: None
The resulting JIT will be of the form: #define [name] [type] [value/default]
.
Buffers Node and Sub-node Structure
Buffers
node configures all input/output buffers for the OpenCL entry function. No buffers node structure exists.
Sub-nodes: Data
(0+), Tensor
(1+)
Data Node and Sub-node Structure
Data
node configures a single input with static data (for example, weight or biases).
Attribute Name | # | Description |
---|---|---|
name |
(1) | Name of a blob attached to a layer in the IR |
arg-index |
(1) | 0-based index in the entry function arguments to be bound to |
Sub-nodes: None
Tensor Node and Sub-node Structure
Tensor
node configures a single input or output tensor.
Attribute Name | # | Description |
---|---|---|
arg-index |
(1) | 0-based index in the entry function arguments to be bound to. |
type |
(1) |
input or output |
port-index |
(1) | 0-based index in the layer’s input/output ports in the IR |
format |
(0/1) | Data layout declaration for the tensor. Accepted values: BFYX , BYXF , YXFB , FYXB (also in all lowercase). Default value: BFYX |
CompilerOptions Node and Sub-node Structure
CompilerOptions
node configures the compilation flags for the OpenCL sources.
Attribute Name | # | Description |
---|---|---|
options |
(1) | Options string to be passed to the OpenCL compiler |
Sub-nodes: None
WorkSizes Node and Sub-node Structure
WorkSizes
node configures the global/local work sizes to be used when queuing the OpenCL program for execution.
Attribute Name | # | Description |
---|---|---|
global local |
(0/1) (0/1) |
An array of up to 3 integers (or formulas) for defining the OpenCL work-sizes to be used during execution. The formulas can use the values of the B,F,Y,X dimensions and contain the operators: +,-,/,*,% (all evaluated in integer arithmetic). Default value: global=”B*F*Y*X” local=”” |
dim |
(0/1) | A tensor to take the work size from. Accepted values: input N , output , where N is an index of input tensor starting with 0. Default value: output |
Sub-nodes: None
The following code sample provides an example configuration file (in .xml
format). For information on configuration file structure, see Configuration File Format.
The following table includes definitions that will be attached before the user sources, where <TENSOR>
is the actual input and output, (for example, INPUT0
or OUTPUT0
).
For an example, see Example Kernel.
Name | Value |
---|---|
NUM_INPUTS |
Number of the input tensors bound to this kernel |
GLOBAL_WORKSIZE |
An array of global work sizes used to execute this kernel |
GLOBAL_WORKSIZE_SIZE |
The size of the GLOBAL_WORKSIZE array |
LOCAL_WORKSIZE |
An array of local work sizes used to execute this kernel |
LOCAL_WORKSIZE_SIZE |
The size of the LOCAL_WORKSIZE array |
<TENSOR>_DIMS |
An array of the tensor dimension sizes. Always ordered as BFYX |
<TENSOR>_DIMS_SIZE |
The size of the <TENSOR>_DIMS array. |
<TENSOR>_TYPE |
The data-type of the tensor: float , half or char |
<TENSOR>_FORMAT_ |
The format of the tensor, BFYX, BYXF, YXFB , FYXB or ANY. The format will be concatenated to the defined name. You can use the tensor format to define codepaths in your code with #ifdef/#endif . |
<TENSOR>_LOWER_PADDING |
An array of padding elements used for the tensor dimensions before they start. Always ordered as BFYX. |
<TENSOR>_ LOWER_PADDING_SIZE |
The size of the <TENSOR>_LOWER_PADDING array |
<TENSOR>_UPPER_PADDING |
An array of padding elements used for the tensor dimensions after they end. Always ordered as BFYX. |
<TENSOR>_UPPER_PADDING_SIZE |
The size of the <TENSOR>_UPPER_PADDING array |
<TENSOR>_PITCHES |
The number of elements between adjacent elements in each dimension. Always ordered as BFYX. |
<TENSOR>_PITCHES_SIZE |
The size of the <TENSOR>_PITCHES array |
<TENSOR>_OFFSET |
The number of elements from the start of the tensor to the first valid element (bypassing the lower padding) |
All <TENSOR>
values will be automatically defined for every tensor bound to this layer (INPUT0
, INPUT1
, OUTPUT0
, and so on), as shown in the following for example:
NOTE: As described in the previous section, all the things like
INPUT0_TYPE
are actually defined as OpenCL (pre-)compiler inputs by the Inference Engine for efficiency reasons. See Debugging Tips for information on debugging the results.
clDNN_program0.cl
, clDNN_program1.cl
. There are as many files as distinct sets of parameters for your custom kernel (different input tensor sizes, and kernel parameters).printf
in your kernels. However, you should be careful: for instance, do not output excessively as it would generate too much data. Since the printf
output is typical, your output can be truncated to fit the buffer. Also, because of buffering, you actually get an entire buffer of output when the execution ends.Since the primary vehicle for the performance of the CPU codepath in the Inference Engine is the Intel® Math Kernel Library for Deep Neural Networks (Intel® MKL-DNN), new CPU kernels extend the Inference Engine plugin for the Intel MKL-DNN. Implementing the InferenceEngine::ILayerImplFactory
defines a general CPU-side extension. There are no Intel MKL-DNN specifics in the way you need to implement a kernel.
All Inference Engine samples (except trivial hello_classification
) feature a dedicated command-line option -l
to CPU load custom kernels. Use the following command-line code to execute the Classification Sample with custom CPU kernels:
Consider simple CustomLayerFactory
class that registers example kernels which make multiplication by two of its input data, but and does not change the dimensions:
CustomLayerFactory
class: InferenceEngine::ILayerImplFactory
getShapes
and getImplementations
of the InferenceEngine::ILayerImplFactory
class: CustomLayerImpl
class: execute
method to change data, inherit it from the abstract class InferenceEngine::ILayerExecImpl
, overload and implement the abstract methods of this class: getSupportedConfigurations
virtual method, which returns all supported configuration formats (input/output tensor layouts) for your implementation. To specify formats of data, use InferenceEngine::TensorDesc
. Refer to the Memory Primitives section for instructions on how to do it. init
method to get a runtime-selected configuration from a vector that populated in the previous step and check the parameters: execute
method, which accepts and processes the actual tenors as input/output blobs: InferenceEngine::IExtension
, which defines the functions that you need to implement:
Unload
, Release
, SetLogCallback
: GetVersion
: AddExtension
method of the general plugin interface to load your primitives: clc
compiler with the following command: Write a configuration file with a kernel parameter description and bindings. For example, given the following OpenCL kernel signature:
Configuration file for this kernel might be the following:
Each custom layer is described with CustomLayer
node. It has the following required nodes and attributes:
CustomLayer
must contain the following attributes:name
, which is the name of the Inference Engine layer to bind kernel withtype
and version
. Set them to MVCL
and 1
.Kernel
must contain the following attributes:entry
, which is a name of your kernel function as you defined it in a source file (in the example above, it is reorg_nhwc
)Source
must contain the following attributes:filename
, which is a path to a compiled binary relative to the .xml
binding fileParameters
, which describes parameters bindingsWorkSizes
, which describes local, global work group sizes and source for dimension deduction as a pair direction,port
. In the example above, the work group is described relatively to the dimension of input tensor that comes thought port 0 in IR. global
and local
work group configurations support any simple math expressions with +,-,*,/, and () from B
(batch), Y
(height), X
(width) and F
(channels).Parameter description supports Tensor
and Scalar
nodes and has the following format:
Tensor
node must contain the following attribute:arg-name
, which is a name of a kernel parameter in the kernel signaturetype
, which is input
or output
as in the IRport-index
, which is a number of input/output port as in the IRformat
, which specifies channel order in tensor. Optional repacks are generated if custom layer format is not compatible with formats of neighboring layers.Scalar
node must contain the following attributes:arg-name
, which is a name of a kernel parameter in the kernel signaturetype
, which is int
or float
. It is used for correct argument extraction from IR parameters.source
, which contains the name of the parameter in IR file or input/output (I
/O
, In
/On
, where n
is a port number) followed by dimension B
(batch), Y
(height), X
(width) or F
(channels).IInferencePlugin::SetConfig()
method with the PluginConfigParams::KEY_CONFIG_FILE
key and the configuration file name as the value: VPU_CUSTOM_LAYERS
and /path/to/your/customLayers.xml
as a network configuration: NOTE: If both native and custom layer implementations are present, custom kernel has a priority over native code.