Sparse Additive Models
Journal of the Royal Statistical Society Series B (Statistical Methodology)Published 19 October 2009Open access
Pradeep Ravikumar, John Lafferty, Han Liu, Larry Wasserman
Citations553
SJR quartileQ1
SJR score3.31
SNIP2.48
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
An algorithm for fitting the models is derived that is practical and effective even when the number of covariates is larger than the sample size, and empirical results show that they can be effective in fitting sparse non‐parametric models in high dimensional data.
Abstract
Summary We present a new class of methods for high dimensional non-parametric regression and classification called sparse additive models. Our methods combine ideas from sparse linear modelling and additive non-parametric regression. We derive an algorithm for fitting the models that is practical and effective even when the number of covariates is larger than the sample size. Sparse additive models are essentially a functional version of the grouped lasso of Yuan and Lin. They are also closely related to the COSSO model of Lin and Zhang but decouple smoothing and sparsity, enabling the use of arbitrary non-parametric smoothers. We give an analysis of the theoretical properties of sparse additive models and present empirical results on synthetic and real data, showing that they can be effective in fitting sparse non-parametric models in high dimensional data.
Keywords
Mathematics
Journal of the Royal Statistical Society Series B (Statistical Methodology)Regression Shrinkage and Selection Via the Lasso
51,790 Citations1996Robert Tibshirani
A new method for estimation in linear models called the lasso, which minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant, is proposed.
The Elements of Statistical Learning: Data Mining, Inference, and Prediction
19,345 Citations2013Trevor Hastie, Robert Tibshirani +1 more
Journal of the American Statistical AssociationVariable Selection via Nonconcave Penalized Likelihood and its Oracle Properties
9,196 Citations2001Jianqing Fan, Runze Li
In this article, penalized likelihood approaches are proposed to handle variable selection problems, and it is shown that the newly proposed estimators perform as well as the oracle procedure in variable selection; namely, they work as well if the correct submodel were known.
Journal of the American Statistical AssociationThe Adaptive Lasso and Its Oracle Properties
7,633 Citations2006Hui Zou
A new version of the lasso is proposed, called the adaptive lasso, where adaptive weights are used for penalizing different coefficients in the ℓ1 penalty, and the nonnegative garotte is shown to be consistent for variable selection.
Journal of the Royal Statistical Society Series B (Statistical Methodology)Model Selection and Estimation in Regression with Grouped Variables
7,467 Citations2005Ming Yuan, Yi Lin
NatureEmergence of simple-cell receptive field properties by learning a sparse code for natural images
5,840 Citations1996Bruno A. Olshausen, David J. Field
It is shown that a learning algorithm that attempts to find sparse linear codes for natural scenes will develop a complete family of localized, oriented, bandpass receptive fields, similar to those found in the primary visual cortex.
Communications on Pure and Applied MathematicsAn iterative thresholding algorithm for linear inverse problems with a sparsity constraint
4,957 Citations2004Ingrid Daubechies, Michel Defrise +1 more
It is proved that replacing the usual quadratic regularizing penalties by weighted 𝓁p‐penalized penalties on the coefficients of such expansions, with 1 ≤ p ≤ 2, still regularizes the problem.
The Annals of StatisticsHigh-dimensional graphs and variable selection with the Lasso
2,406 Citations2006Nicolai Meinshausen, Peter Bühlmann
It is shown that neighborhood selection with the Lasso is a computationally attractive alternative to standard covariance selection for sparse high-dimensional graphs and is hence equivalent to variable selection for Gaussian linear models.
On Model Selection Consistency of Lasso
2,015 Citations2006Peng Zhao, Bin Yu
It is proved that a single condition, which is called the Irrepresentable Condition, is almost necessary and sufficient for Lasso to select the true model both in the classical fixed p setting and in the large p setting as the sample size n gets large.
Probability in Banach Spaces
1,414 Citations1991Michel Ledoux, Michel Talagrand
Probability in Banach Spaces: Isoperimetry and Processes
1,276 Citations1991Michel Ledoux, Michel Talagrand
Springer series in statisticsNonparametric and Semiparametric Models
1,058 Citations2004Wolfgang Karl Härdle, Axel Werwatz +2 more
Journal of Machine Learning ResearchOn Model Selection Consistency of Lasso
1,010 Citations2006ZhaoPeng, Yubin Yubin
The Annals of StatisticsLinear Smoothers and Additive Models
991 Citations1989Andreas Buja, Trevor Hastie +1 more
It is shown that backfitting is the Gauss-Seidel iterative method for solving a set of normal equations associated with the additive model and conditions for consistency and nondegeneracy are provided and convergence is proved for the backfitting and related algorithms for a class of smoothers that includes cubic spline smoothers.
UC BerkeleyLasso-type recovery of sparse representations for high-dimensional data
751 Citations2006Meinshausen, Nicolai, Yu, Bin
The Annals of StatisticsComponent selection and smoothing in multivariate nonparametric regression
566 Citations2006Yi Lin, Hao Helen Zhang
A detailed analysis reveals that the COSSO does model selection by applying a novel soft thresholding type operation to the function components, which leads naturally to an iterative algorithm.
Journal of the American Statistical AssociationRegularization of Wavelet Approximations
488 Citations2001Anestis Antoniadis, Jianqing Fan
Sparsity oracle inequalities for the Lasso
421 Citations2012Florentina Bunea, Alexandre B. Tsybakov +1 more
BernoulliPersistence in high-dimensional linear predictor selection and the virtue of overparametrization
359 Citations2004Eitan Greenshtein, Ya’acov Ritov
Under various sparsity assumptions on the optimal predictor there is “asymptotically no harm” in introducing many more explanatory variables than observations, and such practice can be beneficial in comparison with a procedure that screens in advance a small subset of explanatory variables.
Electronic Journal of StatisticsSparsity oracle inequalities for the Lasso
240 Citations2007Florentina Bunea, Alexandre Tsybakov +1 more
It is shown that the penalized least squares estimator satisfies sparsity oracle inequalities, i.e., bounds in terms of the number of non-zero components of the oracle vector, in nonparametric regression setting with random design.
Neural Information Processing SystemsHigh-Dimensional Graphical Model Selection Using ell_1-Regularized Logistic Regression
178 Citations2006Martin J. Wainwright, John Lafferty +1 more
Under certain mutual incoherence conditions analogous to those imposed in previous work on linear regression, it is proved that consistent neighborhood selection can be obtained as long as the number of observations n grows more quickly than 6d6 log d + 2d5 log p, thereby establishing that logarithmic growth in thenumber of samples n relative to graph size p is sufficient to achieve neighborhood consistency.
The Annals of StatisticsFunctional aggregation for nonparametric regression
165 Citations2000Anatoli Juditsky, Arkadi Nemirovski
This paper addresses the following aggregation problem: given M functions f 1, ..., f M, and proposes algorithms which provide approximations of f * with expected L 2 accuracy O(N -1/4 In 1/4 M), and shows that this approximation rate cannot be significantly improved.
Journal of the American Statistical AssociationNonparametric Inferences for Additive Models
154 Citations2005Jianqing Fan, Jiancheng Jiang
The generalized likelihood ratio (GLR) tests are extended to additive models, using the backfitting estimator, and it is proved that the GLR tests are asymptotically optimal in terms of rates of convergence for nonparametric hypothesis testing.
Neural Information Processing SystemsSpAM: Sparse Additive Models
153 Citations2007Han Liu, Larry Wasserman +2 more
A statistical analysis of the properties of SpAM and empirical results on synthetic and real data show that SpAM can be effective in fitting sparse nonparametric models in high dimensional data.
arXiv (Cornell University)Sharp thresholds for high-dimensional and noisy recovery of sparsity
152 Citations2006Martin J. Wainwright
This work analyzes the behavior of $\ell_1$-constrained quadratic programming (QP), also referred to as the Lasso, for recovering the sparsity pattern and establishes a sharp relation between the problem dimension $\mdim$ and the number of observations that are required for reliable recovery.
Generalized Additive Models
57 Citations2000Stefan Sperlich, Jiří Zelinka
eScholarship (California Digital Library)Variable Selection via Penalized Likelihood
47 Citations1999Jianqing Fan, Runze Liu
The sieve likelihood ratio statistics are shown to be general and powerful for nonparametric inferences based on function estimation and can be adaptively optimal in the sense of Spokoiny (1996) by using a simple choice of adaptive smoothing parameter.
Lasso-type recovery of sparse representations for high-dimensional data
41 Citations2006Nicolai Meinshausen, Bin Yu
Nonnegative Garrote Component Selection in Functional ANOVA models
34 Citations2007Ming Yuan
It is shown that the whole solution path of the proposed nonparametric extension of the nonnegative garrote can be efficiently computed, which facilitates the selection of the tuning parameter.
arXiv (Cornell University)Multi-Stage Variable Selection: Screen and Clean
5 Citations2007Larry Wasserman, Kathryn Roeder