Understanding variable importances in forests of randomized trees
Open Repository and Bibliography (University of Liège)Published 5 December 2013Open access
Gilles Louppe, Louis Wehenkel, Antonio Sutera, Pierre Geurts
Citations806
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This work characterize the Mean Decrease Impurity (MDI) variable importances as measured by an ensemble of totally randomized trees in asymptotic sample and ensemble size conditions and shows that this MDI importance of a variable is equal to zero if and only if the variable is irrelevant.
Abstract
peer reviewed
Keywords
Computer ScienceEnvironmental Science
Machine LearningRandom Forests
126,192 Citations2001Leo Breiman
Internal estimates monitor error, strength, and correlation and these are used to show the response to increasing the number of features used in the forest, and are also applicable to regression.
arXiv (Cornell University)Scikit-learn: Machine Learning in Python
63,264 Citations2012Fabián Pedregosa, Gaël Varoquaux +17 more
Classification and Regression by randomForest
18,390 Citations2007Andy Liaw, Matthew C. Wiener
random forests are proposed, which add an additional layer of randomness to bagging and are robust against overfitting, and the randomForest package provides an R interface to the Fortran programs by Breiman and Cutler.
Artificial IntelligenceWrappers for feature subset selection
8,925 Citations1997Ron Kohavi, George H. John
The wrapper method searches for an optimal feature subset tailored to a particular algorithm and a domain and compares the wrapper approach to induction without feature subset selection and to Relief, a filter approach tofeature subset selection.
Machine LearningExtremely randomized trees
8,669 Citations2006Pierre Geurts, Damien Ernst +1 more
A new tree-based ensemble method for supervised classification and regression problems that consists of randomizing strongly both attribute and cut-point choice while splitting a tree node and builds totally randomized trees whose structures are independent of the output values of the learning sample.
IEEE Transactions on Pattern Analysis and Machine IntelligenceThe random subspace method for constructing decision forests
6,737 Citations1998Tin Kam Ho
A method to construct a decision tree based classifier is proposed that maintains highest accuracy on training data and improves on generalization accuracy as it grows in complexity.
BMC BioinformaticsBias in random forest variable importance measures: Illustrations, sources and a solution
3,612 Citations2007Carolin Strobl, Anne‐Laure Boulesteix +2 more
An alternative implementation of random forests is proposed, that provides unbiased variable selection in the individual classification trees, that can be used reliably for variable selection even in situations where the potential predictor variables vary in their scale of measurement or their number of categories.
BMC BioinformaticsConditional variable importance for random forests
3,244 Citations2008Carolin Strobl, Anne‐Laure Boulesteix +3 more
Pattern Recognition LettersVariable selection using random forests
2,580 Citations2010Robin Genuer, Jean‐Michel Poggi +1 more
This paper proposes, focusing on random forests, the increasingly used statistical method for classification and regression problems introduced by Leo Breiman in 2001, to investigate two classical issues of variable selection, and proposes a strategy involving a ranking of explanatory variables using the random forests score of importance and a stepwise ascending variable introduction strategy.
Quantile Regression Forests
1,503 Citations2006Nicolai Meinshausen
It is shown here that random forests provide information about the full conditional distribution of the response variable, not only about the conditional mean, in order to be competitive in terms of predictive power.
Journal of the American Statistical AssociationRandom Forests and Adaptive Nearest Neighbors
523 Citations2006Yi Lin, Yongho Jeon
It is shown that random forests with adaptive splitting schemes assign weights to k-PNNs in a desirable way: for the estimation at a given target point, these random forests assign voting weights to the k- PNNs of the target point according to the local importance of different input variables.
Consistency of Random Forests and Other Averaging Classifiers
467 Citations2007Gérard Biau, Luc Devroye +1 more
A number of theorems are given that establish the universal consistency of averaging rules, and it is shown that some popular classifiers, including one suggested by Breiman, are not universally consistent.
Variable importance in binary regression trees and forests
396 Citations2012Hemant Ishwaran
CONSISTENCY FOR A SIMPLE MODEL OF RANDOM FORESTS
164 Citations2004Leo Breiman
A heuristic analysis is presented in this paper based on a simplified version of RF denoted RF0 that supports the empirical results from RF and illuminates why RF is able to handle large numbers of input variables and what the role of mtry is.
Machine LearningTechnical Note: Bias in Information-Based Measures in Decision Tree Induction
120 Citations1994A. P. White, Wei Zhong Liu
A fresh look is taken at the problem of bias in information-based attribute selection measures, used in the induction of decision trees and it is concluded that approaches which utilise the chi-square distribution are preferable because they compensate automatically for differences between attributes in the number of levels they take.
Utah State Research and Scholarship (Utah State University)A New Perspective on Classification
8 Citations2021Guohua Zhao