Quantization vs Pruning vs Distillation: Optimizing NNs for Inference

Summary

The video explains four ways to reduce neural-network inference time and model size. The benefits of each method depend on the technique as well as the hardware and software used to run the model.

1. Quantization

Quantization stores model values at lower precision. For example, changing FP32 weights to INT8 can reduce their storage requirements by about 4×. The video describes zero-point quantization, which maps values linearly into a smaller integer range.

• Weight quantization reduces model size, but may not make inference faster if weights are converted back to FP32 for computation.
• Activation quantization enables computation at lower precision, which can improve speed on hardware with fast INT8 support. It requires selecting appropriate activation scales; calibration data can help, while poor calibration may cause values to be clipped.
• Hardware capabilities matter. The video cites the A10 GPU as having substantially higher INT8 than FP32 performance, while some older GPUs may benefit less.
• Outliers can make quantization less effective, particularly in large language models. The video mentions a mixed-decomposition approach that handles outliers separately.

2. Pruning

Pruning sets selected network weights to zero, producing a sparse model. In magnitude pruning, the smallest weights by absolute value are removed. Further training with those weights held at zero can help recover accuracy.

Zeroing weights alone does not necessarily reduce storage or speed up inference. A sparse execution engine must be able to skip zero values. The video distinguishes between:

• Unstructured pruning: Removes weights without imposing a pattern.
• Structured pruning: Follows a pattern that hardware can exploit. One example is 2:4 sparsity, in which only two values in each group of four may remain nonzero. Some NVIDIA Tensor Core GPUs support this pattern.

The pruning pattern should be chosen with the target hardware in mind.

3. Knowledge distillation

Distillation trains a smaller student model to imitate a trained teacher. The teacher’s probability distribution over possible outputs can provide richer learning signals than a single ground-truth label for each example.

A key advantage is that the student can use a different, smaller architecture—for example, one with fewer transformer layers. As a result, distillation can produce larger speedups than quantization or pruning. However, preparing training data and generating teacher outputs can be costly and complex.

The video cites a distilled BERT example that reportedly reduced model size by 40% while retaining 97% of its accuracy. It contrasts the example’s approximately 700 GPU-hours of training with about 24,000 GPU-hours for a cited RoBERTa training run.

4. Engineering optimizations

Engineering optimizations improve execution and can be combined with the other techniques. Examples include:

• CPU vector instructions such as AVX2 and AVX-512, along with hardware support for reduced-precision or sparse computation.
• Fused kernels, which combine sequential operations to reduce execution overhead.
• FlashAttention, which fuses attention operations and uses GPU-aware tiling and memory-hierarchy optimizations.

The video also recommends using software designed for inference rather than assuming a training framework is ideal for deployment. It names ONNX Runtime and TensorFlow Lite as examples.

Unlike quantization or pruning, engineering optimizations should preserve the model’s outputs.

Overall takeaway

Quantization and pruning can reduce inference costs, but may affect accuracy and require compatible hardware or execution engines. Distillation can enable a substantially smaller architecture, but is more involved and computationally costly to train. Engineering optimizations are complementary and should generally preserve accuracy. The best approach balances development effort, inference cost, and model quality.

Source and content type

• Content type: An educational overview and guide to inference-optimization techniques; it does not review a specific product.
• Main speaker/source: Bi, a machine-learning engineer and PhD in natural language processing, presenting on the Efficient NLP channel.