TensorIterator¶
Versioned name: TensorIterator-1
Category: Infrastructure
Short description: TensorIterator layer performs recurrent execution of the network, which is described in the body, iterating through the data.
TensorIterator attributes:
Body:
bodyis a network that will be recurrently executed. The network is described layer by layer as a typical IR network.Body attributes:
No attributes available.
Port map:
port_map is a set of rules to map input or output data tensors of
TensorIteratorlayer ontobodydata tensors. Theport_mapentries can beinputandoutput. Each entry describes a corresponding mapping rule.Port map attributes:
external_port_id
Description: external_port_id is a port ID of the
TensorIteratorlayer.Range of values: indexes of the TensorIterator outputs
Type:
intDefault value: None
Required: yes
internal_layer_id
Description: internal_layer_id is a Parameter or Result layer ID inside the
bodynetwork to map to.Range of values: IDs of the Parameter layers inside in the TensorIterator layer
Type:
intDefault value: None
Required: yes
axis
Description: axis is an axis to iterate through. It triggers the slicing of this tensor. Only if it is specified, the corresponding
inputoroutputis divided into pieces and start, end and stride attributes define how slicing is performed.Range of values: an integer
Type:
intDefault value: None
Required: no
start
Description: start is an index where the iteration starts from. Negative value means counting indexes from the end. Applies only when the attribute
axisis specified.Range of values: an integer
Type:
intDefault value: 0
Required: no
end
Description: end is an index where iteration ends. Negative value means counting indexes from the end. Applies only when the attribute
axisis specified.Range of values: an integer
Type:
intDefault value: -1
Required: no
stride
Description: stride is a step of iteration. Negative value means backward iteration. Applies only when the attribute
axisis specified.Range of values: an integer
Type:
intDefault value: 1
Required: no
Back edges:
back_edges is a set of rules to transfer tensor values from
bodyoutputs at one iteration tobodyparameters at the next iteration. Back edge connects some Result layer inbodyto Parameter layer in the samebody.Back edge attributes:
from-layer
Description: from-layer is a Result layer ID inside the
bodynetwork.Range of values: IDs of the Result layers inside the TensorIterator
Type:
intDefault value: None
Required: yes
to-layer
Description: to-layer is a Parameter layer ID inside the
bodynetwork to end mapping.Range of values: IDs of the Parameter layers inside the TensorIterator
Type:
intDefault value: None
Required: yes
Inputs
Multiple inputs: Tensors of any type and shape supported type.
Outputs
Multiple outputs: Results of execution of the
body. Tensors of any type and shape.
Detailed description
Similar to other layers, TensorIterator has regular sections: input and output. It allows connecting TensorIterator to the rest of the IR.
TensorIterator also has several special sections: body, port_map, back_edges. The principles of their work are described below.
How body is iterated:
At the first iteration: TensorIterator slices input tensors by a specified axis and iterates over all parts in a specified order. It process input tensors with arbitrary network specified as an IR network in the body section. IR is executed as no back-edges are present. Edges from port map are used to connect input ports of TensorIterator to Parameters in body.
[inputs] - Port map edges -> [Parameters:body:Results]
Parameter and Result layers are part of the body. Parameters are stable entry points in the body. The results of the execution of the body are presented as stable Result layers. Stable means that these nodes cannot be fused.
Next iterations:
Back edges define which data is copied back to Parameters layers from Results layers between IR iterations in TensorIterator body. That means they pass data from source layer back to target layer. Each layer that is a target for back-edge has also an incoming port map edge as an input. The values from back-edges are used instead of corresponding edges from port map. After each iteration of the network, all back edges are executed.
Iterations can be considered as statically unrolled sequence: all edges that flow between two neighbor iterations are back-edges. So in the unrolled loop, each back-edge is transformed to regular edge.
… -> [Parameters:body:Results] - back-edges -> [Parameters:body:Results] - back-edges -> [Parameters:body:Results] - back-edges -> …
Calculation of results:
If output entry in the Port map doesn’t have partitioning (axis, begin, end, strides) attributes, then the final value of output of TensorIterator is the value of Result node from the last iteration. Otherwise the final value of output of TensorIterator is a concatenation of tensors in the Result node for all body iterations. Concatenation order is specified by stride attribute.
The last iteration:
[Parameters:body:Results] - Port map edges -> [outputs], if partitioning attributes are not set.
if there are partitioning attributes, then an output tensor is a concatenation of tensors from all body iterations. If stride > 0:
output = Concat(S[0], S[1], ..., S[N-1])
where Si is value of Result operation at i-th iteration in the tensor iterator body that corresponds to this output port. If stride < 0, then output is concatenated in a reverse order:
output = Concat(S[N-1], S[N-2], ..., S[0])
Examples
Example 1: a typical TensorIterator structure
<layer type="TensorIterator" ... >
<input> ... </input>
<output> ... </output>
<port_map>
<input external_port_id="0" internal_layer_id="0" axis="1" start="-1" end="0" stride="-1"/>
<input external_port_id="1" internal_layer_id="1"/>
...
<output external_port_id="3" internal_layer_id="2" axis="1" start="-1" end="0" stride="-1"/>
...
</port_map>
<back_edges>
<edge from-layer="1" to-layer="1"/>
...
</back_edges>
<body>
<layers> ... </layers>
<edges> ... </edges>
</body>
</layer>
Example 2: a full TensorIterator layer
<layer type="TensorIterator" ...>
<input>
<port id="0">
<dim>1</dim>
<dim>25</dim>
<dim>512</dim>
</port>
<port id="1">
<dim>1</dim>
<dim>256</dim>
</port>
<port id="2">
<dim>1</dim>
<dim>256</dim>
</port>
</input>
<output>
<port id="3" precision="FP32">
<dim>1</dim>
<dim>25</dim>
<dim>256</dim>
</port>
</output>
<port_map>
<input axis="1" external_port_id="0" internal_layer_id="0" start="0"/>
<input external_port_id="1" internal_layer_id="3"/>
<input external_port_id="2" internal_layer_id="4"/>
<output axis="1" external_port_id="3" internal_layer_id="12"/>
</port_map>
<back_edges>
<edge from-layer="8" to-layer="4"/>
<edge from-layer="9" to-layer="3"/>
</back_edges>
<body>
<layers>
<layer id="0" type="Parameter" ...>
<output>
<port id="0" precision="FP32">
<dim>1</dim>
<dim>1</dim>
<dim>512</dim>
</port>
</output>
</layer>
<layer id="1" type="Const" ...>
<data offset="0" size="16"/>
<output>
<port id="1" precision="I64">
<dim>2</dim>
</port>
</output>
</layer>
<layer id="2" type="Reshape" ...>
<input>
<port id="0">
<dim>1</dim>
<dim>1</dim>
<dim>512</dim>
</port>
<port id="1">
<dim>2</dim>
</port>
</input>
<output>
<port id="2" precision="FP32">
<dim>1</dim>
<dim>512</dim>
</port>
</output>
</layer>
<layer id="3" type="Parameter" ...>
<output>
<port id="0" precision="FP32">
<dim>1</dim>
<dim>256</dim>
</port>
</output>
</layer>
<layer id="4" type="Parameter" ...>
<output>
<port id="0" precision="FP32">
<dim>1</dim>
<dim>256</dim>
</port>
</output>
</layer>
<layer id="5" type="Const" ...>
<data offset="16" size="3145728"/>
<output>
<port id="1" precision="FP32">
<dim>1024</dim>
<dim>768</dim>
</port>
</output>
</layer>
<layer id="6" type="Const" ...>
<data offset="3145744" size="4096"/>
<output>
<port id="1" precision="FP32">
<dim>1024</dim>
</port>
</output>
</layer>
<layer id="7" type="LSTMCell" ...>
<data hidden_size="256"/>
<input>
<port id="0">
<dim>1</dim>
<dim>512</dim>
</port>
<port id="1">
<dim>1</dim>
<dim>256</dim>
</port>
<port id="2">
<dim>1</dim>
<dim>256</dim>
</port>
<port id="3">
<dim>1024</dim>
<dim>768</dim>
</port>
<port id="4">
<dim>1024</dim>
</port>
</input>
<output>
<port id="5" precision="FP32">
<dim>1</dim>
<dim>256</dim>
</port>
<port id="6" precision="FP32">
<dim>1</dim>
<dim>256</dim>
</port>
</output>
</layer>
<layer id="8" type="Result" ...>
<input>
<port id="0">
<dim>1</dim>
<dim>256</dim>
</port>
</input>
</layer>
<layer id="9" type="Result" ...>
<input>
<port id="0">
<dim>1</dim>
<dim>256</dim>
</port>
</input>
</layer>
<layer id="10" type="Const" ...>
<data offset="3149840" size="24"/>
<output>
<port id="1" precision="I64">
<dim>3</dim>
</port>
</output>
</layer>
<layer id="11" type="Reshape" ...>
<input>
<port id="0">
<dim>1</dim>
<dim>256</dim>
</port>
<port id="1">
<dim>3</dim>
</port>
</input>
<output>
<port id="2" precision="FP32">
<dim>1</dim>
<dim>1</dim>
<dim>256</dim>
</port>
</output>
</layer>
<layer id="12" type="Result" ...>
<input>
<port id="0">
<dim>1</dim>
<dim>1</dim>
<dim>256</dim>
</port>
</input>
</layer>
</layers>
<edges>
<edge from-layer="0" from-port="0" to-layer="2" to-port="0"/>
<edge from-layer="1" from-port="1" to-layer="2" to-port="1"/>
<edge from-layer="2" from-port="2" to-layer="7" to-port="0"/>
<edge from-layer="3" from-port="0" to-layer="7" to-port="1"/>
<edge from-layer="4" from-port="0" to-layer="7" to-port="2"/>
<edge from-layer="5" from-port="1" to-layer="7" to-port="3"/>
<edge from-layer="6" from-port="1" to-layer="7" to-port="4"/>
<edge from-layer="7" from-port="6" to-layer="8" to-port="0"/>
<edge from-layer="7" from-port="5" to-layer="9" to-port="0"/>
<edge from-layer="7" from-port="5" to-layer="11" to-port="0"/>
<edge from-layer="10" from-port="1" to-layer="11" to-port="1"/>
<edge from-layer="11" from-port="2" to-layer="12" to-port="0"/>
</edges>
</body>
</layer>