The Group Lasso for Logistic Regression
Journal of the Royal Statistical Society Series B (Statistical Methodology)Published 4 January 2008
Lukas Meier, Sara van de Geer, Peter Bühlmann
Citations1,720
SJR quartileQ1
SJR score3.31
SNIP2.48
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
An efficient algorithm is presented, that is especially suitable for high dimensional problems, which can also be applied to generalized linear models to solve the corresponding convex optimization problem.
Abstract
Summary The group lasso is an extension of the lasso to do variable selection on (predefined) groups of variables in linear regression models. The estimates have the attractive property of being invariant under groupwise orthogonal reparameterizations. We extend the group lasso to logistic regression models and present an efficient algorithm, that is especially suitable for high dimensional problems, which can also be applied to generalized linear models to solve the corresponding convex optimization problem. The group lasso estimator for logistic regression is shown to be statistically consistent even if the number of predictors is much larger than sample size but with sparse true underlying structure. We further use a two-stage procedure which aims for sparser models than the group lasso, leading to improved prediction performance for some cases. Moreover, owing to the two-stage nature, the estimates can be constructed to be hierarchical. The methods are used on simulated and real data sets about splice site detection in DNA sequences.
Keywords
Computer ScienceMathematics
Journal of the Royal Statistical Society Series B (Statistical Methodology)Regression Shrinkage and Selection Via the Lasso
51,790 Citations1996Robert Tibshirani
A new method for estimation in linear models called the lasso, which minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant, is proposed.
PubMedRegularization Paths for Generalized Linear Models via Coordinate Descent.
13,977 Citations2010Jerome H. Friedman, Trevor Hastie +1 more
The Annals of StatisticsLeast angle regression
9,493 Citations2004Bradley Efron, Trevor Hastie +2 more
Journal of the Royal Statistical Society Series B (Statistical Methodology)Model Selection and Estimation in Regression with Grouped Variables
7,467 Citations2005Ming Yuan, Yi Lin
Journal of Molecular BiologyPrediction of complete gene structures in human genomic DNA
4,264 Citations1997Chris Burge, Samuel Karlin
A general probabilistic model of the gene structure of human genomic sequences which incorporates descriptions of the basic transcriptional, translational and splicing signals, as well as length distributions and compositional features of exons, introns and intergenic regions is introduced.
Statistics in MedicineTHE LASSO METHOD FOR VARIABLE SELECTION IN THE COX MODEL
4,211 Citations1997Robert Tibshirani
Simulations indicate that the lasso can be more accurate than stepwise selection in this setting and reduce the estimation variance while providing an interpretable final model in Cox's proportional hazards model.
Journal of Computational BiologyMaximum Entropy Modeling of Short Sequence Motifs with Applications to RNA Splicing Signals
2,131 Citations2004G Yeo, Christopher B. Burge
Journal of Optimization Theory and ApplicationsConvergence of a Block Coordinate Descent Method for Nondifferentiable Minimization
1,923 Citations2001P. Tseng
IEEE Transactions on Pattern Analysis and Machine IntelligenceSparse multinomial logistic regression: fast algorithms and generalization bounds
884 Citations2005Balaji Krishnapuram, Lawrence Carin +2 more
This paper introduces a true multiclass formulation based on multinomial logistic regression and derives fast exact algorithms for learning sparse multiclass classifiers that scale favorably in both the number of training samples and the feature dimensionality, making them applicable even to large data sets in high-dimensional feature spaces.
Journal of the Royal Statistical Society Series B (Statistical Methodology)<i>L</i>1-Regularization Path Algorithm for Generalized Linear Models
862 Citations2007Mee Young Park, Trevor Hastie
A path following algorithm for L1‐regularized generalized linear models that efficiently computes solutions along the entire regularization path by using the predictor–corrector method of convex optimization.
IMA Journal of Numerical AnalysisA new approach to variable selection in least squares problems
847 Citations2000M. R. Osborne
A compact descent method for solving the constrained problem for a particular value of κ is formulated, and a homotopy method, in which the constraint bound κ becomes the Homotopy parameter, is developed to completely describe the possible selection regimes.
TechnometricsLarge-Scale Bayesian Logistic Regression for Text Categorization
813 Citations2007Alexander Genkin, David Lewis +1 more
This work presents a simple Bayesian logistic regression approach that uses a Laplace prior to avoid overfitting and produces sparse predictive models for text data and applies this approach to a range of document classification problems and shows that it produces compact predictive models at least as effective as those produced by support vector machine classifiers or ridgelogistic regression combined with feature selection.
Mathematical ProgrammingA coordinate gradient descent method for nonsmooth separable minimization
784 Citations2007Paul Tseng, Sangwoon Yun
A (block) coordinate gradient descent method for solving this class of nonsmooth separable problems and establishes global convergence and, under a local Lipschitzian error bound assumption, linear convergence for this method.
Journal of the American Statistical AssociationRegularization of Wavelet Approximations
488 Citations2001Anestis Antoniadis, Jianqing Fan
Computational Statistics & Data AnalysisRelaxed Lasso
462 Citations2006Nicolai Meinshausen
It is shown that the contradicting demands of an efficient computational procedure and fast convergence rates of the `2-loss can be overcome by a two-stage procedure, termed the relaxed Lasso.
BioinformaticsA simple and efficient algorithm for gene selection using sparse logistic regression
407 Citations2003Shirish Shevade, S. Sathiya Keerthi
The Annals of StatisticsHigh-dimensional generalized linear models and the lasso
396 Citations2008Sara A. van de Geer
A nonasymptotic oracle inequality is proved for the empirical risk minimizer with Lasso penalty for high-dimensional generalized linear models with Lipschitz loss functions, and the penalty is based on the coefficients in the linear predictor, after normalization with the empirical norm.
IEEE Transactions on Neural NetworksThe Generalized LASSO
308 Citations2004Volker Röth
This paper presents a different class of kernel regressors that effectively overcome the above problems, and presents a highly efficient algorithm with guaranteed global convergence that defies a unique framework for sparse regression models in the very rich class of IRLS models.
ℓ1-Penalization for Mixture Regression Models
180 Citations2016Nicolas Städler
ANU Open Research (Australian National University)Adaptive regression and model selection in data mining problems
176 Citations1999Sergey Bakin
Nonlinear Programming
108 Citations2004Harvir Singh Kasana, Krishna Dev Kumar
Stagewise Lasso
90 Citations2007Peng Zhao, Bin Yu
The BLasso algorithm is proposed that ties the FSF (e-Boosting) algorithm with the Lasso method that minimizes the L1 penalized L2 loss and provides a class of simple and easy-to-implement algorithms for tracing the regularization or solution paths of penalized minimization problems.
New comprehensive biochemistryModeling dependencies in pre-mRNA splicing signals
85 Citations1998Christopher B. Burge
This chapter discusses the compositional properties of pre-mRNA splicing signals and discusses some caveats concerning the dangers of constructing models with too many parameters and overinterpreting observed correlations in sequence data.
Journal of Jiamusi UniversityL1 Regularization Path Algorithm for Generalized Linear Models
61 Citations2008Yang Jing
An estimation algorithm of coefficient to select variables for L1 regularized generalized linear models to efficiently compute solutions along the entire regularization path using the predictor-corrector method of convex-optimization.
Columbia Academic Commons (Columbia University)Algorithms for Sparse Linear Classifiers in the Massive Data Setting
51 Citations2008Suhrid Balakrishnan, David Madigan +1 more
BernoulliClassifiers of support vector machine type with \ell1 complexity regularization
45 Citations2006Bernadetta Tarigan, Sara A. van de Geer
Under certain conditions on the base functions, hinge loss with this complexity penalty is shown to lead to an oracle inequality involving both model complexity and margin.
Following Curved Regularized Optimization Solution Paths
39 Citations2004Saharon Rosset
A general path-following algorithm based on second-order approximations is suggested, it is proved that under mild conditions it remains "very close" to the path of optimal solutions and illustrated it with examples.
Regularization Path Algorithms for Detecting Gene Interactions
19 Citations2006Mee Young Park, Trevor Hastie
This study considers several regularization path algorithms with grouped variable selection for modeling gene-interactions and proposes a path-following algorithm for the group-Lasso method applied to generalized linear models.
Journal of Machine Learning ResearchAlgorithms for Sparse Linear Classifiers in the Massive Data Setting
19 Citations2008BalakrishnanSuhrid, MadiganDavid
This paper proposes online and multi-pass algorithms for training sparse linear classifiers for high dimensional data that have computational complexity and memory requirements that make learning on massive data sets feasible.
Fast Algorithms and Generalization Bounds
3 Citations2005Balaji Krishnapuram, Lawrence Carin +2 more
This paper introduces a true multiclass formulation based on multinomial logistic regression and derives fast exact algorithms for learning sparse multiclass classifiers that scale favorably in both the number of training samples and the feature dimensionality, making them applicable even to large data sets in high-dimensional feature spaces.