AUGRUCell

Versioned name: AUAUGRUCell

Category: Sequence processing

Short description: AUGRUCell represents a single AUGRU Cell (GRU with attentional update gate).

Detailed description: The main difference between AUGRUCell and GRUCell is the additional attention score input A, which is a multiplier for the update gate. The AUGRU formula is based on the paper arXiv:1809.03672.

AUGRU formula:
  *  - matrix multiplication
 (.) - Hadamard product (element-wise)

 f, g - activation functions
 z - update gate, r - reset gate, h - hidden gate
 a - attention score

  rt = f(Xt*(Wr^T) + Ht-1*(Rr^T) + Wbr + Rbr)
  zt = f(Xt*(Wz^T) + Ht-1*(Rz^T) + Wbz + Rbz)
  ht = g(Xt*(Wh^T) + (rt (.) Ht-1)*(Rh^T) + Rbh + Wbh)  # 'linear_before_reset' is False

  zt' = (1 - at) (.) zt  # multiplication by attention score

  Ht = (1 - zt') (.) ht + zt' (.) Ht-1

Attributes

• hidden_size

  • Description: hidden_size specifies hidden state size.

  • Range of values: a positive integer

  • Type: int

  • Required: yes

• activations

  • Description: activation functions for gates

  • Range of values: sigmoid, tanh

  • Type: a list of strings

  • Default value: sigmoid for f, tanh for g

  • Required: no

• activations_alpha, activations_beta

  • Description: activations_alpha, activations_beta attributes of functions; applicability and meaning of these attributes depends on chosen activation functions

  • Range of values: []

  • Type: float[]

  • Default value: []

  • Required: no

• clip

  • Description: clip specifies bound values [-C, C] for tensor clipping. Clipping is performed before activations.

  • Range of values: 0.

  • Type: float

  • Default value: 0. that means the clipping is not applied

  • Required: no

• linear_before_reset

  • Description: linear_before_reset flag denotes, if the output of hidden gate is multiplied by the reset gate before or after linear transformation.

  • Range of values: False

  • Type: boolean

  • Default value: False

  • Required: no.

Inputs

• 1: X - 2D tensor of type T and shape [batch_size, input_size], input data. Required.

• 2: H_t - 2D tensor of type T and shape [batch_size, hidden_size]. Input with initial hidden state data. Required.

• 3: W - 2D tensor of type T and shape [3 * hidden_size, input_size]. The weights for matrix multiplication, gate order: zrh. Required.

• 4: R - 2D tensor of type T and shape [3 * hidden_size, hidden_size]. The recurrence weights for matrix multiplication, gate order: zrh. Required.

• 5: B - 2D tensor of type T. The biases. If linear_before_reset is set to False, then the shape is [3 * hidden_size], gate order: zrh. Otherwise the shape is [4 * hidden_size] - the sum of biases for z and r gates (weights and recurrence weights), the biases for h gate are placed separately. Required.

• 6: A - 2D tensor of type T and shape [batch_size, 1], the attention score. Required.

Outputs

• 1: Ho - 2D tensor of type T [batch_size, hidden_size], the last output value of hidden state.

Types

• T: any supported floating-point type.

Example

<layer ... type="AUGRUCell" ...>
    <data hidden_size="128"/>
     <input>
        <port id="0"> <!-- `X` input data -->
            <dim>1</dim>
            <dim>16</dim>
        </port>
        <port id="1"> <!-- `H_t` input -->
            <dim>1</dim>
            <dim>128</dim>
        </port>
         <port id="3"> <!-- `W` weights input -->
            <dim>384</dim>
            <dim>16</dim>
        </port>
         <port id="4"> <!-- `R` recurrence weights input -->
            <dim>384</dim>
            <dim>128</dim>
        </port>
         <port id="5"> <!-- `B` bias input -->
            <dim>384</dim>
        </port>
        <port id="6"> <!-- `A` attention score input -->
            <dim>1</dim>
            <dim>1</dim>
        </port>
    </input>
    <output>
        <port id="7"> <!-- `Y` output -->
            <dim>1</dim>
            <dim>4</dim>
            <dim>128</dim>
        </port>
        <port id="8"> <!-- `Ho` output -->
            <dim>1</dim>
            <dim>128</dim>
        </port>
    </output>
</layer>