Neural Network Compression Framework

This document describes the Neural Network Compression Framework (NNCF) which is distributed as a separate tool but is highly aligned with OpenVINO in terms of the supported optimization features and models. It is open-sourced and available on GitHub.

Introduction

Neural Network Compression Framework (NNCF) is aimed at optimizing Deep Neural Network (DNN) by applying optimization methods, such as quantization, pruning, etc., to the original framework model. It provides in-training optimization capabilities which means that optimization methods require model fine-tuning or even re-training. The diagram below shows the model optimization workflow using NNCF.

_images/nncf_workflow.png

Features

  • Support optimization of PyTorch and TensorFlow 2.x models.

  • Support of various optimization algorithms, applied during a model fine-tuning process to achieve a better performance-accuracy trade-off:

Compression algorithm

PyTorch

TensorFlow 2.x

8- bit quantization

Supported

Supported

Filter pruning

Supported

Supported

Sparsity

Supported

Supported

Mixed-precision quantization

Supported

Not supported

Binarization

Supported

Not supported

  • Stacking of optimization methods. For example: 8-bit quaNtization + Filter Pruning.

  • Support for Accuracy-Aware model training pipelines via the Adaptive Compression Level Training and Early Exit Training.

  • Automatic, configurable model graph transformation to obtain the compressed model.

    Note

    Limited support for TensorFlow models. Only the models created, using Sequential or Keras Functional API, are supported.

    • GPU-accelerated layers for the faster compressed model fine-tuning.

  • Distributed training support.

  • Configuration file examples for each supported compression algorithm.

  • Exporting PyTorch compressed models to ONNX* checkpoints and TensorFlow compressed models to SavedModel or Frozen Graph format, ready to use with OpenVINO toolkit.

  • Git patches for prominent third-party repositories (huggingface-transformers) demonstrating the process of integrating NNCF into custom training pipelines

Get started

Installation

NNCF provides the packages available for installation through the PyPI repository. To install the latest version via pip manager run the following command:

pip install nncf

Usage examples

NNCF provides various examples and tutorials that demonstrate usage of optimization methods.