Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?

It is formally proved that these networks with random Gaussian weights perform a distance-preserving embedding of the data, with a special treatment for in-class and out-of-class data.

Abstract

Three important properties of a classification machinery are: (i) the system\npreserves the core information of the input data; (ii) the training examples\nconvey information about unseen data; and (iii) the system is able to treat\ndifferently points from different classes. In this work we show that these\nfundamental properties are satisfied by the architecture of deep neural\nnetworks. We formally prove that these networks with random Gaussian weights\nperform a distance-preserving embedding of the data, with a special treatment\nfor in-class and out-of-class data. Similar points at the input of the network\nare likely to have a similar output. The theoretical analysis of deep networks\nhere presented exploits tools used in the compressed sensing and dictionary\nlearning literature, thereby making a formal connection between these important\ntopics. The derived results allow drawing conclusions on the metric learning\nproperties of the network and their relation to its structure, as well as\nproviding bounds on the required size of the training set such that the\ntraining examples would represent faithfully the unseen data. The results are\nvalidated with state-of-the-art trained networks.\n