Adequate and Precise Evaluation of Quality Models in Software Engineering Studies
Published 1 May 2007
Yan Ma, Bojan Čukić
Citations44
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The procedure outlined here, if followed, has a potential to enhance the statistical validity of future experiments and to choose the "best" candidate among many available models.
Abstract
Many statistical techniques have been proposed and introduced to predict fault-proneness of program modules in software engineering. Choosing the "best" candidate among many available models involves performance assessment and detailed comparison. But these comparisons are not simple due to varying performance measures and the related verification and validation cost implications. Therefore, a methodology for precise definition and evaluation of the predictive models is still needed. We believe the procedure we outline here, if followed, has a potential to enhance the statistical validity of future experiments.
Keywords
Computer Science
Machine LearningRandom Forests
126,192 Citations2001Leo Breiman
Internal estimates monitor error, strength, and correlation and these are used to show the response to increasing the number of features used in the forest, and are also applicable to regression.
IEEE Transactions on Software EngineeringA validation of object-oriented design metrics as quality indicators
1,701 Citations1996Victor R. Basili, Lionel Briand +1 more
Several of Chidamber and Kemerer's 00 metrics appear to be adequate to predict class fault-proneness during the early phases of the life-cycle, and are, on the data set, better predictors than "traditional" code metrics, which can only be collected at a later phase of the software development processes.
Machine LearningMachine Learning for the Detection of Oil Spills in Satellite Radar Images
1,253 Citations1998Miroslav Kubát, Robert C. Holte +1 more
This case study relates issues as problem formulation, selection of evaluation measures, and data preparation to properties of the oil spill application, such as its imbalanced class distribution, that are shown to be common to many applications.
A Sequential Algorithm for Training Text Classifiers
1,158 Citations1994David Lewis, William A. Gale
An algorithm for sequential sampling during machine learning of statistical classifiers was developed and tested on a newswire text categorization task and reduced by as much as 500-fold the amount of training data that would have to be manually classified to achieve a given level of effectiveness.
IEEE Transactions on Software EngineeringLearning from examples: generation and evaluation of decision trees for software resource analysis
280 Citations1988Richard W. Selby, A. Porter
A general solution method for the automatic generation of decision (or classification) trees is investigated to provide insights through in-depth empirical characterization and evaluation of decision trees for one problem domain, specifically, that of software resource data analysis.
Robust Prediction of Fault-Proneness by Random Forests
277 Citations2005Lan Guo, Yan Ma +2 more
This paper presents a novel methodology for predicting fault prone modules, based on random forests, an extension of decision tree learning that generates hundreds or even thousands of trees using subsets of the training data.
IEEE SoftwareBuilding Defect Prediction Models in Practice
170 Citations2005A. Güneş Koru, Hongfang Liu
Effective defect prediction models can help developers focus quality assurance activities on defect-prone modules and thus improve software quality by using resources more efficiently.
International Journal of Artificial Intelligence ToolsEMPIRICAL ASSESSMENT OF MACHINE LEARNING BASED SOFTWARE DEFECT PREDICTION TECHNIQUES
137 Citations2008V.U.B. Challagulla, Farokh Bastani +2 more
An empirical analysis of four different real-time software defect data sets using different predictor models shows that a combination of 1R and Instance-based learning along with Consistency-based subset evaluation technique provides a relatively better consistency in achieving accurate predictions as compared with other models.
Tree-based software quality estimation models for fault prediction
135 Citations2003Taghi M. Khoshgoftaar, Naeem Seliya
A case study from a comprehensive evaluation (with several large case studies) of currently available regression tree algorithms for software fault prediction of CART-LS, S-PLUS, and Cart-LAD, finding that these are simple and effective as software quality prediction models.
Journal of Systems and SoftwareA neural network approach for early detection of program modules having high risk in the maintenance phase
135 Citations1995Taghi M. Khoshgoftaar, D.L. Lanning
The utility of neural networks in isolating high-risk modules where class determination is based on multiple quality metrics is demonstrated, and the absence of the data assumptions typical of statistical techniques is demonstrated.
Predicting fault-prone modules with case-based reasoning
60 Citations2002Taghi M. Khoshgoftaar, K. Ganesan +5 more
A case study provides evidence that case based reasoning can be the basis for useful software quality classification models that are competitive with discriminant models and revisits data from a previously published nonparametric discriminant analysis study.
When can we test less?
36 Citations2004Tim Menzies, Justin Di Stefano +5 more
In the case study presented here, traditional methods of generating defect detectors were found to be inferior to those found via a PACE analysis.
Combining and adapting software quality predictive models by genetic algorithms
22 Citations2002Danielle Azar, Doina Precup +3 more
This paper proposes an approach to combine and/or adapt existing models (experts) in such way that the combined/adapted model works well on the particular system.
ACM SIGSOFT Software Engineering NotesNearest neighbor sampling for better defect prediction
22 Citations2005Gary D. Boetticher
This paper conducts 20 experiments on five different NASA defect datasets using two different learners (J48 and Naïve Bayes) and proposes a new nearest neighbor sampling technique, which consists of those test instances closest to class training instances.