The Impact of Model Compression on Convolutional Neural Networks

The Impact of Model Compression on Convolutional Neural Networks

Table of Contents:

1. Introduction
2. The Effects of Model Compression on Convolutional Neural Networks 2.1 Motivation for Model Compression 2.2 Background on Model Compression Techniques 2.2.1 Pruning 2.2.1.1 Structured vs. Unstructured Pruning 2.2.1.2 Global vs. Local Pruning 2.2.2 Quantization 2.2.2.1 Post-training Quantization 2.2.2.2 Quantization-aware Training
3. Experimental Setup 3.1 Networks and Datasets 3.2 Model Compression Techniques
4. General Results of Model Compression 4.1 Overall Accuracy 4.2 Robustness Analysis
5. Changes in Class Confusion and Class Accuracy
6. Changes in Model Confidence
7. Changes in Saliency Maps
8. Conclusions and Future Directions

The Effects of Model Compression on Convolutional Neural Networks

Model compression techniques have gained popularity due to their ability to reduce the size and inference time of convolutional neural networks (CNNs) without significantly sacrificing accuracy. However, it is important to understand how model compression impacts the underlying behavior of the networks. This article investigates the effects of model compression on CNNs beyond test accuracy, exploring how compression changes the predictive quality and attention of the models.

1. Introduction

The motivation for this study Stems from the need to deploy CNNs to low-power devices while maintaining accuracy. Model compression, specifically techniques like pruning and quantization, offer potential solutions to reduce the size and improve the efficiency of CNNs. However, mere test accuracy is not sufficient to assess the impact of model compression on the overall behavior and performance of the networks. This study aims to analyze how model compression ALTERS CNNs under the hood and understand the changes in their predictive quality and attention mechanisms.

2. The Effects of Model Compression on Convolutional Neural Networks

2.1 Motivation for Model Compression

While model compression offers benefits like reduced model size and inference time, it is important to consider the changes it introduces to CNNs. Accuracy alone does not provide a comprehensive understanding of the network's behavior. This study aims to analyze the impact of model compression on a class example level and assess changes in attention and relevance within the models.

2.2 Background on Model Compression Techniques

To investigate the effects of model compression, two techniques are considered: pruning and quantization.

2.2.1 Pruning

Pruning induces sparsity in CNNs by removing neurons or SYNAPSE connections. There are two types of pruning: structured and unstructured. Structured pruning involves removing entire groups of neurons, such as channels or layers, while unstructured pruning focuses on removing individual atomic elements like neurons themselves. Pruning can be done on a global or local Scale, affecting the entire network or specific groups of neurons independently.

2.2.2 Quantization

Quantization aims to reduce the number of bits required to represent model parameters. The precision is lowered from floating-point (32 or 16 bits) to integer precision (e.g., 8-bit). Post-training quantization modifies the weights and activations after the training phase, while quantization-aware training adjusts the training pipeline to consider quantization during the training process.

3. Experimental Setup

The experimental setup involves three different networks: Alvinet5, SqueezeNet, and ResNet-18. Two datasets are considered: CIFAR-10 and the German Traffic Sign Recognition Benchmark. Four model compression techniques are applied: global unstructured pruning, post-training quantization with 8-bit precision, post-training quantization with 4-bit precision, and a combination of global unstructured pruning and 8-bit post-training quantization.

4. General Results of Model Compression

4.1 Overall Accuracy

The overall accuracy of the compressed models is analyzed to understand the impact of model compression. The results show a decrease in accuracy of less than one percentage point, with test accuracies mostly within the range of 0.2 to 0.3 percentage points lower compared to the original uncompressed models. However, some experiments with severe quantization caused significant degradation in model performance.

4.2 Robustness Analysis

Various robustness metrics are evaluated, including expected calibration error, corruption, and out-of-distribution detection. The analysis shows no significant differences in overall robustness between the compressed and uncompressed models. The researchers did not find any systematic differences in the impact of corruptions or biases on the models' performance. Additionally, the imbalanced nature of the German Traffic Sign Recognition Benchmark dataset did not significantly affect the performance of the compressed models or introduce any noticeable differences compared to the CIFAR-10 dataset.

5. Changes in Class Confusion and Class Accuracy

This section explores the changes that occur in the class confusion and accuracy of the compressed models. The analysis reveals that pruning and quantization can introduce significant changes at the class level, with up to 7.5 percent of the classes being differently classified after compression. However, despite these changes, the overall accuracy remains relatively similar, with only a slight difference of about 0.2 to 0.3 percentage points.

6. Changes in Model Confidence

The study investigates how model compression affects the confidence of the predictions. The analysis shows significant changes in the prediction confidences, with some samples exhibiting completely different confidences for the compressed and uncompressed models. These changes in confidence can lead to misclassifications and variations in the attention and saliency maps.

7. Changes in Saliency Maps

The saliency maps of the compressed and uncompressed models are compared to understand how compression affects the attention mechanisms. The analysis reveals significant differences in the saliency maps between the two models, even when the class accuracy remains consistent. This indicates that the compressed models can exhibit different attention Patterns and highlight the need for further investigation into the expressiveness and reliability of saliency maps.

8. Conclusions and Future Directions

In conclusion, model compression introduces significant changes in the behavior of CNNs, beyond simple accuracy differences. The analysis highlights the importance of considering these changes in safety assurance and the development of trustable AI systems. Future directions include investigating other model compression techniques, developing methods for systematic analysis of machine learning models, and incorporating continuous safety assurance as an integral part of the development process.

For additional information and resources, please refer to the following:

• Winning HAND: A Pruning-Quantization Method for Neural Network Compression. Link
• Repulsive Deep Ensembles for More Accurate and Stable Uncertainty Estimation. Link

FAQ:

Q: How does model compression impact the overall accuracy of CNNs? A: Model compression techniques can lead to a decrease in overall accuracy of less than one percentage point.

Q: Are there significant differences in robustness between compressed and uncompressed models? A: No significant differences in overall robustness were observed between the two.

Q: Can model compression techniques improve the diversity of ensembles? A: While compression techniques may affect model behavior, it is not guaranteed to improve ensemble diversity.

Q: How do changes in model confidence impact predictions? A: Model compression can significantly alter the prediction confidences, leading to different classifications for the same samples.

Resources:

• Matt Dwyer's Plenary on Distributional Test Adequacy