A Direct Estimation Approach to Sparse Linear Discriminant Analysis
Journal of the American Statistical AssociationPublished 1 December 2011Open access
Tommaso Cai, Weidong Liu
Citations267
SJR quartileQ1
SJR score4.10
SNIP3.08
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A simple and effective classifier is introduced by estimating the product Ωδ directly through constrained ℓ1 minimization and it has superior finite sample performance and significant computational advantages over the existing methods that require separate estimation of Ω and δ.
Abstract
Abstract This article considers sparse linear discriminant analysis of high-dimensional data. In contrast to the existing methods which are based on separate estimation of the precision matrix Ω and the difference δ of the mean vectors, we introduce a simple and effective classifier by estimating the product Ωδ directly through constrained ℓ1 minimization. The estimator can be implemented efficiently using linear programming and the resulting classifier is called the linear programming discriminant (LPD) rule. The LPD rule is shown to have desirable theoretical and numerical properties. It exploits the approximate sparsity of Ωδ and as a consequence allows cases where it can still perform well even when Ω and/or δ cannot be estimated consistently. Asymptotic properties of the LPD rule are investigated and consistency and rate of convergence results are given. The LPD classifier has superior finite sample performance and significant computational advantages over the existing methods that require separate estimation of Ω and δ. The LPD rule is also applied to analyze real datasets from lung cancer and leukemia studies. The classifier performs favorably in comparison to existing methods. Keywords: : ClassificationConstrained l1 minimizationFisher’s ruleNaive Bayes ruleSparsity
Keywords
Computer ScienceBiochemistry, Genetics and Molecular BiologyEngineering
Cambridge University Press eBooksConvex Optimization
31,266 Citations2004Stephen Boyd, Lieven Vandenberghe
ScienceMolecular Classification of Cancer: Class Discovery and Class Prediction by Gene Expression Monitoring
11,622 Citations1999Todd R. Golub, Donna K. Slonim +10 more
A generic approach to cancer classification based on gene expression monitoring by DNA microarrays is described and applied to human acute leukemias as a test case and suggests a general strategy for discovering and predicting cancer classes for other types of cancer, independent of previous biological knowledge.
Journal of Business and Economic StatisticsAn Introduction to Multivariate Statistical Analysis
9,232 Citations1986Robb J. Muirhead, T. W. Anderson
Proceedings of the National Academy of SciencesDiagnosis of multiple cancer types by shrunken centroids of gene expression
2,947 Citations2002Robert Tibshirani, Trevor Hastie +2 more
The method of “nearest shrunken centroids” identifies subsets of genes that best characterize each class, which was highly efficient in finding genes for classifying small round blue cell tumors and leukemias.
Journal of the American Statistical AssociationRegularized Discriminant Analysis
2,238 Citations1989Jerome H. Friedman
Alternatives to the usual maximum likelihood estimates for the covariance matrices are proposed, characterized by two parameters, the values of which are customized to individual situations by jointly minimizing a sample-based estimate of future misclassification risk.
IEEE Transactions on Information TheoryStable recovery of sparse overcomplete representations in the presence of noise
2,209 Citations2005David L. Donoho, Michael Elad +1 more
This paper establishes the possibility of stable recovery under a combination of sufficient sparsity and favorable structure of the overcomplete system and shows that similar stability is also available using the basis and the matching pursuit algorithms.
arXiv (Cornell University)The Dantzig selector: Statistical estimation when $p$ is much larger than $n$
1,725 Citations2005Candes, Emmanuel, Tao, Terence
Is it possible to estimate β reliably based on the noisy data y?
Journal of the American Statistical AssociationA Constrained<i>ℓ</i><sub>1</sub>Minimization Approach to Sparse Precision Matrix Estimation
1,019 Citations2011Tommaso Cai, Weidong Liu +1 more
A constrained ℓ1 minimization method for estimating a sparse inverse covariance matrix based on a sample of n iid p-variate random variables and is applied to analyze a breast cancer dataset and is found to perform favorably compared with existing methods.
PubMedTranslation of microarray data into clinically relevant cancer diagnostic tests using gene expression ratios in lung cancer and mesothelioma.
880 Citations2002Gavin J. Gordon, Roderick V. Jensen +7 more
It is proposed that using gene expression ratios is an accurate and inexpensive technique with direct clinical applicability for distinguishing between MPM and lung cancer and evidence is provided suggesting that this technique can be equally accurate in other clinical scenarios.
Journal of the American Statistical AssociationRegularized Discriminant Analysis
867 Citations1989Jerome H. Friedman
Sparse Permutation Invariant Covariance Estimation
764 Citations2007Adam J. Rothman, Peter J. Bickel +1 more
Mathematical Programming ComputationTemplates for convex cone problems with applications to sparse signal recovery
632 Citations2011Stephen R. Becker, Emmanuel J. Candès +1 more
A general framework for solving a variety of convex cone problems that frequently arise in signal processing, machine learning, statistics, and other fields, and results showing that the smooth and unsmoothed problems are sometimes formally equivalent are applied.
Journal of the American Statistical AssociationAdaptive Thresholding for Sparse Covariance Matrix Estimation
627 Citations2011Tommaso Cai, Weidong Liu
BiostatisticsRegularized linear discriminant analysis and its application in microarrays
617 Citations2006Yi Guo, Trevor Hastie +1 more
Through both simulated data and real life data, it is shown that this method performs very well in multivariate classification problems, often outperforms the PAM method and can be as competitive as the support vector machines classifiers.
BernoulliSome theory for Fisher's linear discriminant function, `naive Bayes', and some alternatives when there are many more variables than observations
564 Citations2004Peter J. Bickel, Elizaveta Levina
Electronic Journal of StatisticsSparse permutation invariant covariance estimation
537 Citations2008Adam J. Rothman, Peter J. Bickel +2 more
A method for constructing a sparse estimator for the inverse covariance (concentration) matrix in high-dimensional settings using a penalized normal likelihood approach and forces sparsity by using a lasso-type penalty is proposed.
The Annals of StatisticsHigh-dimensional classification using features annealed independence rules
478 Citations2008Jianqing Fan, Yingying Fan
The conditions under which all the important features can be selected by the two-sample t-statistic are established and the choice of the optimal number of features, or equivalently, the threshold value of the test statistics are proposed based on an upper bound of the classification error.
Journal of Machine Learning ResearchHigh Dimensional Inverse Covariance Matrix Estimation via Linear Programming
350 Citations2010Ming Yuan
ScholarlyCommons (University of Pennsylvania)Optimal Rates of Convergence for Sparse Covariance Matrix Estimation
250 Citations2012Cai, T. Tony, Zhou, Harrison H
Sparse linear discriminant analysis by thresholding for high dimensional data
220 Citations2012Jun Shao, Yazhen Wang +2 more
Journal of the Royal Statistical Society Series B (Statistical Methodology)Covariance-Regularized Regression and Classification for high Dimensional Problems
216 Citations2009Daniela Witten, Robert Tibshirani
It is shown that ridge regression, the lasso and the elastic net are special cases of covariance‐regularized regression, and it is demonstrated that certain previously unexplored forms of covariant regularized regression can outperform existing methods in a range of situations.
The Annals of StatisticsSparse linear discriminant analysis by thresholding for high dimensional data
151 Citations2011Jun Shao, Yazhen Wang +2 more
When and why the linear discriminant analysis (LDA) has poor performance is explored and a sparse LDA is proposed that is asymptotically optimal under some sparsity conditions on the unknown parameters.
BioinformaticsSparse linear discriminant analysis for simultaneous testing for the significance of a gene set/pathway and gene selection
114 Citations2009Michael C. Wu, Lingsong Zhang +3 more
The results show that sLDA-based testing provides a powerful approach to test for the significance of a differentially expressed pathway and gene selection.
arXiv (Cornell University)High-dimensional covariance estimation by minimizing $\ell_1$-penalized log-determinant divergence
47 Citations2008Pradeep Ravikumar, Martin J. Wainwright +2 more
Journal of the American Statistical AssociationMedian-Based Classifiers for High-Dimensional Data
31 Citations2009Peter Hall, D. M. Titterington +1 more
Using componentwise medians to construct a robust classifier that is relatively insensitive to the difficulties caused by heavy-tailed data and entails straightforward computation is suggested.