Methods for Pruning Deep Neural Networks
A survey of methods for pruning deep neural networks is presented, categorising over 150 studies based on the underlying approach used and focusing on three categories: methods that use magnitude based pruning, methods that utilise clustering to identify redundancy, and methods that using sensitivity analysis to assess the effect of pruning.
Abstract
This paper presents a survey of methods for pruning deep neural networks. It begins by \ncategorising over 150 studies based on the underlying approach used and then focuses on three categories: \nmethods that use magnitude based pruning, methods that utilise clustering to identify redundancy, and \nmethods that use sensitivity analysis to assess the effect of pruning. Some of the key influencing studies \nwithin these categories are presented to highlight the underlying approaches and results achieved. Most \nstudies present results which are distributed in the literature as new architectures, algorithms and data \nsets have developed with time, making comparison across different studied difficult. The paper therefore \nprovides a resource for the community that can be used to quickly compare the results from many different \nmethods on a variety of data sets, and a range of architectures, including AlexNet, ResNet, DenseNet and \nVGG. The resource is illustrated by comparing the results published for pruning AlexNet and ResNet50 on \nImageNet and ResNet56 and VGG16 on the CIFAR10 data to reveal which pruning methods work well in \nterms of retaining accuracy whilst achieving good compression rates. The paper concludes by identifying \nsome research gaps and promising directions for future research.