Gradient Directed Regularization for Linear Regression and Classi…cation

The gradient descent path…nding paradigm can be readily generalized to include a wide variety of loss criteria, leading to robust methods for regression and classi…cation, as well as to apply user de…ned constraints on the parameter values, all with highly e¢ cient computational implementations.

Abstract

Regularization in linear modeling is viewed as a two–stage process. First a set of candidate models is de…ned by a path through the space of joint parameter values, and then a point on this path is chosen to be the …nal model. Various path…nding strategies for the …rst stage of this process are examined, based on the notion of generalized gradient descent. Several of these strategies are seen to produce paths that closely correspond to those induced by commonly used penalization methods. Others give rise to new regularization techniques that are shown to be advantageous in some situations. In all cases, the gradient descent path…nding paradigm can be readily generalized to include the use of a wide variety of loss criteria, leading to robust methods for regression and classi…cation, as well as to apply user de…ned constraints on the parameter values, all with highly e¢ cient computational implementations.