Dualistic geometry of the manifold of higher-order neurons

An information geometrical method, which can be applied to more general neural network manifolds, is proposed and the accuracy of statistical estimation is shown in terms of the dimensionality of a model and the number of examples.

Abstract

A set of neural networks, in particular the set of all the neurons of higher-order, forms a geometrical manifold. Specifically, the sets Nk(k = 1, 2, …) of the kth order neurons constitute a hierarchy of manifolds Nn ⊃ Nn−1 ⊃ … ⊃ N1, where n is the number of inputs. A natural geometry is introduced to Nk and characteristics of higher-order neurons are studied therefrom. A Riemannian metric is defined in Nk and a dual pair of affine connections are introduced in these manifolds. A higher-order neuron realizes a transformation from vector inputs to a scalar output. Given a transformation, the approximation problem searches for its best approximation by a higher-order neuron. The best approximation is proved to be obtained by the dual geodesic projection (Projection Theorem). Moreover, the approximation error is decomposed into a sum of contributions corresponding to various orders of higher-order interactions (Decomposition Theorem). The accuracy of statistical estimation is also shown in terms of the dimensionality of a model and the number of examples (Estimation Theorem). This paper proposes an information geometrical method, which can be applied to more general neural network manifolds.