Improving Random Forests

Lecture notes in computer sciencePublished 1 January 2004

Marko Robnik‐Šikonja

DOI10.1007/978-3-540-30115-8_34

Source URL

Citations231

SJR quartileQ2

SJR score0.35

SNIP0.55

TL;DR

Using several attribute evaluation measures instead of just one gives promising results and replacement of ordinary voting with voting weighted with margin achieved on most similar instances gives improvements which are statistically highly significant over several data sets.

Abstract

Random forests are one of the most successful ensemble methods which exhibits performance on the level of boosting and support vector machines. The method is fast, robust to noise, does not overfit and offers possibilities for explanation and visualization of its output. We investigate some possibilities to increase strength or decrease correlation of individual trees in the forest. Using several attribute evaluation measures instead of just one gives promising results. On the other hand replacement of ordinary voting with voting weighted with margin achieved on most similar instances gives improvements which are statistically highly significant over several data sets.

Keywords

Computer Science

Machine LearningRandom Forests

126,192 Citations2001Leo Breiman

Internal estimates monitor error, strength, and correlation and these are used to show the response to increasing the number of features used in the forest, and are also applicable to regression.

View PDF

BiometricsClassification and Regression Trees.

23,841 Citations1984Alexander Gordon, Leo Breiman +3 more

C4.5: Programs for Machine Learning

23,665 Citations1992J. R. Quinlan

A complete guide to the C4.5 system as implemented in C for the UNIX environment, which starts from simple core learning methods and shows how they can be elaborated and extended to deal with typical problems such as missing data and over hitting.

Machine LearningBagging Predictors

16,973 Citations1996Leo Breiman

View PDF

Experiments with a new boosting algorithm

7,585 Citations1996Yoav Freund, Robert E. Schapire

Programs for Machine Learning

5,794 Citations1994Steven L. Salzberg, Alberto M. Segre

In his new book, C4.5: Programs for Machine Learning, Quinlan has put together a definitive, much needed description of his complete system, including the latest developments, which will be a welcome addition to the library of many researchers and students.

Machine LearningTheoretical and Empirical Analysis of ReliefF and RReliefF

2,986 Citations2003Marko Robnik‐Šikonja, Igor Kononenko

How and why Relief algorithms work, their theoretical and practical properties, their parameters, what kind of dependencies they detect, how do they scale up to large number of examples and features, how to sample data for them, how robust are they regarding the noise, how irrelevant and redundant attributes influence their output and how different metrics influences them.

View PDF

Lecture notes in computer scienceEstimating attributes: Analysis and extensions of RELIEF

2,950 Citations1994Igor Kononenko

In the context of machine learning from examples this paper deals with the problem of estimating the quality of attributes with and without dependencies among them and is analysed and extended to deal with noisy, incomplete, and multi-class data sets.

Machine LearningA Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems

2,264 Citations2001David J. Hand, Robert John Till

This work extends the definition of the area under the ROC curve to the case of more than two classes by averaging pairwise comparisons and proposes an alternative definition of proportion correct based on pairwise comparison of classes for a simple artificial case.

View PDF

NeurocomputingThe support vector machine under test

773 Citations2003David Meyer, Friedrich Leisch +1 more

A popular SVM implementation is compared to 16 classification methods and 9 regression methods accessible through the software R by the means of standard performance measures and bias-variance decompositions which showed mostly good performances both on classification and regression tasks, but other methods proved to be very competitive.