Category: Sequence processing
Short description: CTCLoss computes the CTC (Connectionist Temporal Classification) Loss.
Detailed description:
CTCLoss operation is presented in Connectionist Temporal Classification - Labeling Unsegmented Sequence Data with Recurrent Neural Networks: Graves et al., 2016
CTCLoss estimates likelihood that a target labels[i,:]
can occur (or is real) for given input sequence of logits logits[i,:,:]
. Briefly, CTCLoss operation finds all sequences aligned with a target labels[i,:]
, computes log-probabilities of the aligned sequences using logits[i,:,:]
and computes a negative sum of these log-probabilies.
Input sequences of logits logits
can have different lengths. The length of each sequence logits[i,:,:]
equals logit_length[i]
. A length of target sequence labels[i,:]
equals label_length[i]
. The length of the target sequence must not be greater than the length of corresponding input sequence logits[i,:,:]
. Otherwise, the operation behaviour is undefined.
CTCLoss calculation scheme:
j
-th character at time step t
for i
-th input sequence from logits
using softmax formula: \[ p_{i,t,j} = \frac{\exp(logits[i,t,j])}{\sum^{K}_{k=0}{\exp(logits[i,t,k])}} \]
i
-th target from labels[i,:]
find all aligned paths. A path S = (c1,c2,...,cT)
is aligned with a target G=(g1,g2,...,gT)
if both chains are equal after decoding. The decoding extracts substring of length label_length[i]
from a target G
, merges repeated characters in G
in case preprocess_collapse_repeated equal to true and finds unique elements in the order of character occurrence in case unique equal to true. The decoding merges repeated characters in S
in case ctc_merge_repeated equal to true and removes blank characters represented by blank_index
. By default, blank_index
is equal to C-1
, where C
is a number of classes including the blank. For example, in case default ctc_merge_repeated, preprocess_collapse_repeated, unique and blank_index
a target sequence G=(0,3,2,2,2,2,2,4,3)
of a length label_length[i]=4
is processed to (0,3,2,2)
and a path S=(0,0,4,3,2,2,4,2,4)
of a length logit_length[i]=9
is also processed to (0,3,2,2)
, where C=5
. There exist other paths that are also aligned with G
, for instance, 0,4,3,3,2,4,2,2,2
. Paths checked for alignment with a target label[:,i]
must be of length logit_length[i] = L_i
. Compute probabilities of these aligned paths (alignments) as follows: \[ p(S) = \prod_{t=1}^{L_i} p_{i,t,ct} \]
\[ CTCLoss = - \ln \sum_{S} p(S) \]
Note 1: This calculation scheme does not provide steps for optimal implementation and primarily serves for better explanation.
Note 2: This is recommended to compute a log-probability \( \ln p(S)\) for an aligned path as a sum of log-softmax of input logits. It helps to avoid underflow and overflow during calculation. Having log-probabilities for aligned paths, log of summed up probabilities for these paths can be computed as follows:
\[ \ln(a + b) = \ln(a) + \ln(1 + \exp(\ln(b) - \ln(a))) \]
Attributes
labels[i,:]
passed to the loss are merged into single labels.boolean
boolean
labels[i,:]
before matching with potential alignments. Unique elements in the processed labels[i,:]
are sorted in the order of their occurrence in original labels[i,:]
. For example, the processed sequence for labels[i,:]=(0,1,1,0,1,3,3,2,2,3)
of length label_length[i]=10
will be (0,1,3,2)
in case unique equal to true.boolean
Inputs
logits
- Input tensor with a batch of sequences of logits. Type of elements is T_F. Shape of the tensor is [N, T, C]
, where N
is the batch size, T
is the maximum sequence length and C
is the number of classes including the blank. Required.logit_length
- 1D input tensor of type T1 and of a shape [N]
. The tensor must consist of non-negative values not greater than T
. Lengths of input sequences of logits logits[i,:,:]
. Required.labels
- 2D tensor with shape [N, T]
of type T2. A length of a target sequence labels[i,:]
is equal to label_length[i]
and must contain of integers from a range [0; C-1]
except blank_index
. Required.label_length
- 1D tensor of type T1 and of a shape [N]
. The tensor must consist of non-negative values not greater than T
and label_length[i] <= logit_length[i]
for all possible i
. Required.blank_index
- Scalar of type T2. Set the class index to use for the blank label. Default value is C-1
. Optional.Output
[N]
, negative sum of log-probabilities of alignments. Type of elements is T_F.Types
int32
or int64
.Example