MVN#
Versioned name: MVN-1
Category: Normalization
Short description: Calculates mean-variance normalization of the input tensor. Supports two normalization techniques: Instance/Contrast Normalization and Layer Normalization.
Detailed description
Based on across_channels
attribute mean value is calculated using one of formulas below:
If
true
mean value is calculated using Layer Normalization:\[\mu_{n} = \frac{\sum_{c}^{C}\sum_{h}^{H}\sum_{w}^{W} i_{nchw}}{C * H * W}\]If
false
mean value is calculated using Instance/Contrast Normalization:\[\mu_{nc} = \frac{\sum_{h}^{H}\sum_{w}^{W} i_{nchw}}{H * W}\]
where \(i_{nchw}\) is an input tensor parametrized by \(n\) batches, math:c channels and math:h,w spatial dimensions.
If reduction_axes
attribute is provided mean value is calculated based on formula:
Afterwards MVN subtracts mean value from the input blob.
If normalize_variance is set to true
, the output blob is divided by variance:
where \(\sigma_{k}^2\) is the variance calculated based on mean value, \(\epsilon\) is a value added to the variance for numerical stability and corresponds to epsilon
attribute.
Attributes
across_channels
Description: across_channels is a flag that specifies whether mean values are shared across channels. If
true
mean values and variance are calculated for each sample across all channels and spatial dimensions (Layer Normalization), otherwise calculation is done for each sample and for each channel across spatial dimensions (Instance/Contrast Normalization).Range of values:
false
- do not share mean values across channelstrue
- share mean values across channels
Type:
boolean
Required: yes
reduction_axes
Description: 1D tensor of unique elements and type T_IND which specifies indices of dimensions in
data
that define normalization slices. Negative value means counting dimensions from the back.Range of values: allowed range of axes is
[-r; r-1]
wherer = rank(data)
, the order cannot be sortedType:
int
Required: yes
normalize_variance
Description: normalize_variance is a flag that specifies whether to perform variance normalization.
Range of values:
false
- do not normalize variancetrue
- normalize variance
Type:
boolean
Required: yes
eps
Description: eps is the number to be added to the variance to avoid division by zero when normalizing the value. For example, epsilon equal to 0.001 means that 0.001 is added to the variance.
Range of values: a positive floating-point number
Type:
double
Required: yes
Important
It is necessary to use only one of
across_channels
orreduction_axes
attributes, they cannot be defined together.
Inputs
1:
data
- input tensor of type T and arbitrary shape. Required.
Outputs
1: normalized tensor of type T and shape as input tensor.
Types
T: any floating point type.
T_IND:
int64
orint32
.
Examples
Example: with across_channels
attribute
<layer ... type="MVN">
<data across_channels="true" eps="1e-9" normalize_variance="true"/>
<input>
<port id="0">
<dim>6</dim>
<dim>12</dim>
<dim>10</dim>
<dim>24</dim>
</port>
</input>
<output>
<port id="2">
<dim>6</dim>
<dim>12</dim>
<dim>10</dim>
<dim>24</dim>
</port>
</output>
</layer>
Example: with reduction_axes
attribute
<layer ... type="MVN">
<data reduction_axes="2,3" eps="1e-9" normalize_variance="true"/>
<input>
<port id="0">
<dim>6</dim>
<dim>12</dim>
<dim>10</dim>
<dim>24</dim>
</port>
</input>
<output>
<port id="2">
<dim>6</dim>
<dim>12</dim>
<dim>10</dim>
<dim>24</dim>
</port>
</output>
</layer>