Revisiting the evaluation of defect prediction models
Published 18 May 2009
Thilo Mende, Rainer Koschke
Citations182
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper describes a trivial model that takes only the module size measured in lines of code into account, and compares it to five classification methods, and finds that the trivial model performs surprisingly well when evaluated using AUC.
Abstract
Defect Prediction Models aim at identifying error-prone parts of a software system as early as possible. Many such models have been proposed, their evaluation, however, is still an open question, as recent publications show.
Keywords
Computer Science
Machine LearningRandom Forests
126,192 Citations2001Leo Breiman
Internal estimates monitor error, strength, and correlation and these are used to show the response to increasing the number of features used in the forest, and are also applicable to regression.
Pattern Recognition LettersAn introduction to ROC analysis
21,254 Citations2005Tom Fawcett
The purpose of this article is to serve as an introduction to ROC graphs and as a guide for using them in research.
Classification and Regression by randomForest
18,390 Citations2007Andy Liaw, Matthew C. Wiener
random forests are proposed, which add an additional layer of randomness to bagging and are robust against overfitting, and the randomForest package provides an R interface to the Fortran programs by Breiman and Cutler.
Statistical Comparisons of Classifiers over Multiple Data Sets
11,198 Citations2006Janez Demšar
A set of simple, yet safe and robust non-parametric tests for statistical comparisons of classifiers is recommended: the Wilcoxon signed ranks test for comparison of two classifiers and the Friedman test with the corresponding post-hoc tests for comparisons of more classifiers over multiple data sets.
Introduction to Data Mining
7,026 Citations2008
Journal of the American Statistical AssociationThe Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance
3,831 Citations1937Milton Friedman
Computer applications in the biosciencesROCR: visualizing classifier performance in R
3,341 Citations2005Tobias Sing, Oliver Sander +2 more
UNLABELLED ROCR is a package for evaluating and visualizing the performance of scoring classifiers in the statistical language R that features over 25 performance measures that can be freely combined to create two-dimensional performance curves.
IEEE Transactions on Software EngineeringBenchmarking Classification Models for Software Defect Prediction: A Proposed Framework and Novel Findings
1,219 Citations2008Stefan Lessmann, Bart Baesens +2 more
A framework for comparative software defect prediction experiments is proposed and applied in a large-scale empirical comparison of 22 classifiers over 10 public domain data sets from the NASA Metrics Data repository, showing an appealing degree of predictive accuracy, which supports the view that metric-based classification is useful.
Journal of Machine Learning ResearchStatistical Comparisons of Classifiers over Multiple Data Sets
875 Citations2006DemšarJanez
IEEE Transactions on Software EngineeringPredicting the location and number of faults in large software systems
721 Citations2005Thomas J. Ostrand, Elaine J. Weyuker +1 more
A negative binomial regression model has been developed and used to predict the expected number of faults in each file of the next release of a system, based on the code of the file in the current release, and fault and modification history of thefile from previous releases.
IEEE Transactions on Software EngineeringPredicting fault-prone software modules in telephone switches
341 Citations1996Niclas Ohlsson, H. Alberg
Applying the method described in this paper makes it possible to use measures at the design phase to predict the most fault-prone modules, which is a valuable complement to linear regression when prediction data is ordinal.
IEEE Transactions on Software EngineeringProblems with Precision: A Response to "Comments on 'Data Mining Static Code Attributes to Learn Defect Predictors'"
273 Citations2007Tim Menzies, Alex Dekhtyar +2 more
Empirical Software EngineeringTechniques for evaluating fault prediction models
267 Citations2008Yue Jiang, Bojan Čukić +1 more
Using the data from a public repository, this study demonstrates the strengths and weaknesses of performance evaluation techniques and points to a conclusion that the selection of the “best” model cannot be made without considering project cost characteristics, which are specific in each development environment.
Journal of Systems and SoftwareComparing case-based reasoning classifiers for predicting high risk software components
160 Citations2001Khaled El Emam, Saïda Benlarbi +2 more
Evaluating the performance of a CBR classifier with different parameters, namely: different distance measures, different standardization techniques, use or non-use of weights, and the number of nearest neighbors to use for the prediction indicates that there is no difference in prediction performance when using any combination of parameters.
IEEE Transactions on Software EngineeringComments on "Data Mining Static Code Attributes to Learn Defect Predictors"
139 Citations2007Hongyu Zhang, Xiuzhen Zhang
In this correspondence, it is pointed out a discrepancy in a recent paper, " data mining static code attributes to learn defect predictors," that was published in this journal.
Comparing design and code metrics for software quality prediction
126 Citations2008Yue Jiang, Bojan Cuki +2 more
It is confirmed that models built from code metrics typically outperform design metrics based models, and models that utilize a combination of design and code level metrics outperform models which use either one or the other metric set.
Data Mining Techniques for Building Fault-proneness Models in Telecom Java Software
122 Citations2007Erik Arisholm, Lionel Briand +1 more
The main research focus of this paper is to use and compare many data mining and machine learning techniques to build fault-proneness models based mostly on source code measures and change/fault history data and demonstrate that the usual classification evaluation criteria based on confusion matrices may not be fully appropriate to compare and evaluate models.
Can data transformation help in the detection of fault-prone modules?
104 Citations2008Yue Jiang, Bojan Čukić +1 more
This study investigates the effect of four different preprocessing methods to fault-proneness prediction using nine datasets from NASA Metrics Data Programs and ten classification algorithms to conclude that no general benefit can be expected from data transformations.
Software Quality JournalOrdering Fault-Prone Software Modules
60 Citations2003Taghi M. Khoshgoftaar, Edward B. Allen
Two empirical case studies of full-scale industrial software systems provide empirical evidence of the usefulness of module-order models for targeting reliability enhancement.
Cost Curve Evaluation of Fault Prediction Models
56 Citations2008Yue Jiang, Bojan Čukić +1 more
It is observed that software quality does not necessarily benefit from the prediction of fault prone components and the inclusion of misclassification cost in model evaluation may indicate that even the "best" models achieve performance no better than trivial classification.
Automating algorithms for the identification of fault-prone files
47 Citations2007Thomas J. Ostrand, Elaine J. Weyuker +1 more
This research introduces new, more sophisticated models that yield more accurate predictions than the earlier LOC model, but which nonetheless can be fully automated.
Adequate and Precise Evaluation of Quality Models in Software Engineering Studies
44 Citations2007Yan Ma, Bojan Čukić
The procedure outlined here, if followed, has a potential to enhance the statistical validity of future experiments and to choose the "best" candidate among many available models.
Evaluating Defect Prediction Models for a Large Evolving Software System
23 Citations2009Thilo Mende, Rainer Koschke +1 more
A history of roughly 3 years is analyzed to extract process and static code metrics that are used to build several defect prediction models with Random Forests, and a new evaluation measure is developed based on the comparison to an optimal model.