# MatrixNonMaxSuppression#

Versioned name: MatrixNonMaxSuppression-8

Category: Sorting and maximization

Short description: MatrixNonMaxSuppression performs matrix non-maximum suppression (NMS) of the boxes with predicted scores.

Detailed description: The operation performs the following:

1. Selects candidate bounding boxes with scores higher than `score_threshold`.

2. For each class, selects at most `nms_top_k` candidate boxes.

3. Decays scores of the candidate boxes according to the Matrix NMS algorithm Wang et al. This algorithm is applied independently to each class and each batch element. Boxes of `background_class` are skipped and thus eliminated during the process.

4. Selects boxes with the decayed scores higher than `post_threshold`, and selects at most `keep_top_k` scoring candidate boxes per batch element.

The Matrix NMS algorithm is described below:

1. Sort descending the candidate boxes by score, and compute `n*n` pairwise IOU (IntersectionOverUnion) matrix `X` for the top `n` boxes. Suppose `n` is the number of candidate boxes.

2. Set the lower triangle and diagonal of `X` to 0. Therefore get the upper triangular matrix `X`.

3. Take the column-wise max of `X` to compute a vector `K` of maximum IOU for each candidate box.

4. Repeat element value of `K` along axis 1. Suppose this gets a matrix `X_cmax`.

5. Compute the decay factor: `decay_factor = exp((X_cmax**2 - X**2) * gaussian_sigma)` if `decay_function` is `gaussian`, else `decay_factor = (1 - X) / (1 - X_cmax)`.

6. Take the column-wise min of `decay_factor`, and element-wise multiply with scores to decay them.

Attributes:

• sort_result

• Description: sort_result specifies the order of output elements.

• Range of values: `class`, `score`, `none`

• class - sort selected boxes by class id (ascending).

• score - sort selected boxes by score (descending).

• none - do not guarantee the order.

• Type: `string`

• Default value: `none`

• Required: no

• sort_result_across_batch

• Description: sort_result_across_batch is a flag that specifies whenever it is necessary to sort selected boxes across batches or not.

• Range of values: true or false

• true - sort selected boxes across batches.

• false - do not sort selected boxes across batches (boxes are sorted per batch element).

• Type: boolean

• Default value: false

• Required: no

• output_type

• Description: the tensor type of outputs `selected_indices` and `valid_outputs`.

• Range of values: `i64` or `i32`

• Type: `string`

• Default value: `i64`

• Required: no

• score_threshold

• Description: minimum score to consider box for the processing.

• Range of values: a floating-point number

• Type: `float`

• Default value: `0`

• Required: no

• nms_top_k

• Description: maximum number of boxes to be selected per class.

• Range of values: an integer

• Type: `int`

• Default value: `-1` meaning to keep all boxes

• Required: no

• keep_top_k

• Description: maximum number of boxes to be selected per batch element.

• Range of values: an integer

• Type: `int`

• Default value: `-1` meaning to keep all boxes

• Required: no

• background_class

• Description: the background class id.

• Range of values: an integer

• Type: `int`

• Default value: `-1` meaning to keep all classes

• Required: no

• normalized

• Description: normalized is a flag that indicates whether `boxes` are normalized or not.

• Range of values: true or false

• true - the box coordinates are normalized.

• false - the box coordinates are not normalized.

• Type: boolean

• Default value: True

• Required: no

• decay_function

• Description: decay function used to decay scores.

• Range of values: `gaussian`, `linear`

• Type: `string`

• Default value: `linear`

• Required: no

• gaussian_sigma

• Description: gaussian_sigma parameter for gaussian decay_function.

• Range of values: a floating-point number

• Type: `float`

• Default value: `2.0`

• Required: no

• post_threshold

• Description: threshold to filter out boxes with low confidence score after decaying.

• Range of values: a floating-point number

• Type: `float`

• Default value: `0`

• Required: no

Inputs:

• 1: `boxes` - tensor of type T and shape `[num_batches, num_boxes, 4]` with box coordinates. The box coordinates are layout as `[xmin, ymin, xmax, ymax]`. Required.

• 2: `scores` - tensor of type T and shape `[num_batches, num_classes, num_boxes]` with box scores. The tensor type should be same with `boxes`. Required.

Outputs:

• 1: `selected_outputs` - tensor of type T which should be same with `boxes` and shape `[number of selected boxes, 6]` containing the selected boxes with score and class as tuples `[class_id, box_score, xmin, ymin, xmax, ymax]`.

• 2: `selected_indices` - tensor of type T_IND and shape `[number of selected boxes, 1]` the selected indices in the flattened input `boxes`, which are absolute values cross batches. Therefore possible valid values are in the range `[0, num_batches * num_boxes - 1]`.

• 3: `selected_num` - 1D tensor of type T_IND and shape `[num_batches]` representing the number of selected boxes for each batch element.

When there is no box selected, `selected_num` is filled with `0`. `selected_outputs` is an empty tensor of shape `[0, 6]`, and `selected_indices` is an empty tensor of shape `[0, 1]`.

Types

• T: floating-point type.

• T_IND: `int64` or `int32`.

Example

```<layer ... type="MatrixNonMaxSuppression" ... >
<data decay_function="gaussian" sort_result="score" output_type="i64"/>
<input>
<port id="0">
<dim>3</dim>
<dim>100</dim>
<dim>4</dim>
</port>
<port id="1">
<dim>3</dim>
<dim>5</dim>
<dim>100</dim>
</port>
</input>
<output>
<port id="5" precision="FP32">
<dim>-1</dim> <!-- "-1" means a undefined dimension calculated during the model inference -->
<dim>6</dim>
</port>
<port id="6" precision="I64">
<dim>-1</dim>
<dim>1</dim>
</port>
<port id="7" precision="I64">
<dim>3</dim>
</port>
</output>
</layer>
```