Noise detection and elimination in data preprocessing: Experiments in medical domains
Applied Artificial IntelligencePublished 1 February 2000Open access
Dragan Gamberger, Nada Lavrač, Sašo Džeroski
Citations135
SJR quartileQ2
SJR score0.78
SNIP1.41
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A simple compression measure can be used to detect noisy training examples, where noise is due to random classification errors and a hypothesis is then built from the set of remaining examples.
Abstract
Compression measures used in inductive learners, such as measures based on the minimum description length principle, can be used as a basis for grading candidate hypotheses. Compression ± based induction is suited also for handling noisy data. This paper shows that a simple compression measure can be used to detect noisy training examples, where noise is due to randomclassication errors. A technique is proposed in which noisy examples are detected and eliminated from the training set, and a hypothesis is then built from the set of remaining examples. This noise elimination method was applied to preprocess data for four machine ± learning algorithms, and evaluated on selected medical domains.
Keywords
Computer Science
C4.5: Programs for Machine Learning
23,665 Citations1992J. R. Quinlan
A complete guide to the C4.5 system as implemented in C for the UNIX environment, which starts from simple core learning methods and shows how they can be elaborated and extended to deal with typical problems such as missing data and over hitting.
AutomaticaModeling by shortest data description
5,944 Citations1978J. Rissanen
The number of digits it takes to write down an observed sequence x1,...,xN of a time series depends on the model with its parameters that one assumes to have generated the observed data.
International Journal of Man-Machine StudiesSimplifying decision trees
2,502 Citations1987J. R. Quinlan
Techniques for simplifying decision trees while retaining their accuracy are discussed, described, illustrated, and compared on a test-bed of decision trees from a variety of domains.
Machine LearningThe CN2 Induction Algorithm
2,015 Citations1989Peter Clark, Tim Niblett
A description and empirical evaluation of a new induction system, CN2, designed for the efficient induction of simple, comprehensible production rules in domains where problems of poor description language and/or noise may be present.
Lecture notes in computer scienceRule induction with CN2: Some recent improvements
866 Citations1991Peter Clark, Robin Boswell
Improvements to the CN2 algorithm are described, including the use of the Laplacian error estimate as an alternative evaluation function and it is shown how unordered as well as ordered rules can be generated.
Machine LearningOn the Handling of Continuous-Valued Attributes in Decision Tree Generation
680 Citations1992Usama M. Fayyad, Keki B. Irani
The result serves to give a better understanding of the entropy measure, to point out that the behavior of the information entropy heuristic possesses desirable properties that justify its usage in a formal sense, and to improve the efficiency of evaluating continuous-valued attributes for cut value selection.
Machine LearningAn Empirical Comparison of Pruning Methods for Decision Tree Induction
631 Citations1989John Mingers
This paper compares five methods for pruning decision trees, developed from sets of examples, and shows that three methods—critical value, error complexity and reduced error—perform well, while the other two may cause problems.
Machine LearningAn Empirical Comparison of Selection Measures for Decision-Tree Induction
464 Citations1989John Mingers
The paper considers a number of different measures and experimentally examines their behavior in four domains and shows that the choice of measure affects the size of a tree but not its accuracy, which remains the same even when attributes are selected randomly.
Identifying and eliminating mislabeled training instances
223 Citations1996Carla E. Brodley, M. A. Friedl
Empirical results suggest that the ensemble filter approach is an effective method for identifying labeling errors, and further, that the approach will significantly benefit ongoing research to develop accurate and robust remote sensing-based methods to map land cover at global scales.
Lecture notes in computer scienceOn estimating probabilities in tree pruning
158 Citations1991Bojan Cestnik, Ivan Bratko
The resulting pruning method improves on the original Niblett-Bratko pruning in the following respects: apriori probabilities can be incorporated into error estimation, several trees pruned to various degrees can be generated, and the degree of pruning is not affected by the number of classes.
A study of distance-based machine learning algorithms
124 Citations1994Thomas G. Dietterich, Dietrich Wettschereck
It is shown that the k-nearest neighbor algorithm (kNN) outperforms the first nearest neighbor algorithm only under certain conditions, and methods for choosing the value of k for kNN are investigated, and two methods for learning feature weights for a weighted Euclidean distance metric are proposed.
Lecture notes in computer scienceNoise elimination in inductive concept learning: A case study in medical diagnosis
73 Citations1996Dragan Gamberger, Nada Lavrač +1 more
This paper shows that a simple compression measure can be used to detect noisy examples and propose a technique in which noisy examples are detected and eliminated from the training set, and a hypothesis is then built from the set of remaining examples.
Hrčak Portal of scientific journals of Croatia (University Computing Centre)Using the m -estimate in rule induction
63 Citations1993Sašo Džeroski, Bojan Cestnik +1 more
This work replaces the Laplace estimate in the rule induction system CN2 with a general Bayesian probability estimate, the m- estimate, which does not rely on the Laplacian assumption of equally likely classes and allows for adapting to the learning domain.
Lecture notes in computer scienceConditions for Occam's razor applicability and noise elimination
51 Citations1997Dragan Gamberger, Nada Lavrač
This work gives a theoretical framework for the applicability of Occam's razor, developed into a procedure for eliminating noise from a training set and the results of empirical evaluation show the usefulness of the presented approach to noise elimination.
Lecture notes in computer scienceA minimization approach to propositional inductive learning
25 Citations1995Dragan Gamberger
An approach to the problem of propositional inductive learning of if-then-else rules, different from the commonly used ones, is presented and how iterative application of the literal selection process can solve theproblem of learning from noisy domains by appropriate exclusion of some learning examples is presented.
PubMedRule induction and instance-based learning applied in medical diagnosis.
20 Citations1996Sašo Džeroski, Nada Lavrač
Lecture notes in computer scienceNoise detection and elimination applied to noise handling in a KRK chess endgame
10 Citations1997Dragan Gamberger, Nada Lavrač
This paper shows that a simple compression measure can be used to detect noisy examples and propose a technique in which noisy examples are detected and eliminated from the training set, and a hypothesis is then built from the set of remaining examples.
Noise Elimination Applied to Early Diagnosis of Rheumatic Diseases
3 Citations1997Nada Lavrač, Dragan Gamberger +1 more
This chapter presents a compression-based method that is capable of detecting data which is suspected to contain errors and is therefore unsuitable for the extraction of genuine regularities.