Approximation by Ridge Functions and Neural Networks

The theorems obtained show that this form of ridge approximation has the same efficiency of approximation as other more traditional methods of multivariate approximation such as polynomials, splines, or wavelets.

Abstract

We investigate the efficiency of approximation by linear combinations of ridge functions in the metric of L2 (Bd ) with Bd the unit ball in Rd . If Xn is an n-dimensional linear space of univariate functions in L2 (I), I=[-1,1], and $\Omega $ is a subset of the unit sphere Sd-1 in Rd of cardinality m, then the space Yn:={span}\{r({\bf x}\cdot\xi):r\in X_n,\omega\in\Omega\}$ is a linear space of ridge functions of dimension $\le mn$. We show that if Xn provides order of approximation O(n-r ) for univariate functions with r derivatives in L2 (I), and $\Omega$ are properly chosen sets of cardinality O(nd-1 ), then Yn will provide approximation of order O(n-r-d/2+1/2 ) for every function $f\in L2 (Bd ) with smoothness of order r+d/2-1/2 in L2 (Bd ). Thus, the theorems we obtain show that this form of ridge approximation has the same efficiency of approximation as other more traditional methods of multivariate approximation such as polynomials, splines, or wavelets. The theorems we obtain can be applied to show that a feed-forward neural network with one hidden layer of computational nodes given by certain sigmoidal function $\sigma$ will also have this approximation efficiency. Minimal requirements are made of the sigmoidal functions and in particular our results hold for the unit-impulse function $\sigma=\chi_{\raisebox{-.6ex}{$\scriptstyle [0, \infty)$}}$.