Validation of network measures as indicators of defective modules in software systems
Published 18 May 2009
Ayşe Tosun, Burak Turhan, Ayşe Bener
Citations91
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
In this study, additional experiments on public data are conducted to reproduce and validate Zimmermann and Nagappan's findings that network measures are important indicators of defective modules for large and complex systems, whereas they do not have significant effects on small scale projects.
Abstract
In ICSE'08, Zimmermann and Nagappan show that network measures derived from dependency graphs are able to identify critical binaries of a complex system that are missed by complexity metrics. The system used in their analysis is a Windows product. In this study, we conduct additional experiments on public data to reproduce and validate their results. We use complexity and network metrics from five additional systems. We examine three small scale embedded software and two versions of Eclipse to compare defect prediction performance of these metrics. We select two different granularity levels to perform our experiments: function-level and source file-level. In our experiments, we observe that network measures are important indicators of defective modules for large and complex systems, whereas they do not have significant effects on small scale projects.
Keywords
Computer Science
An Introduction to the Bootstrap
39,744 Citations1994Bradley Efron, Robert Tibshirani
Computer Networks and ISDN SystemsThe anatomy of a large-scale hypertextual Web search engine
15,828 Citations1998Sergey Brin, Lawrence M. Page
This paper provides an in-depth description of Google, a prototype of a large-scale search engine which makes heavy use of the structure present in hypertext and looks at the problem of how to effectively deal with uncontrolled hypertext collections where anyone can publish anything they want.
Series in machine perception and artificial intelligenceIntroduction to Machine Learning
1,654 Citations2019
Introduction to Machine Learning is a comprehensive textbook on the subject, covering a broad array of topics not usually included in introductory machine learning texts, and discusses many methods from different fields, including statistics, pattern recognition, neural networks, artificial intelligence, signal processing, control, and data mining.
IBM Systems JournalDesign and code inspections to reduce errors in program development
1,483 Citations1976Michael Fagan
IEEE Transactions on Software EngineeringPreliminary guidelines for empirical research in software engineering
1,477 Citations2002Barbara Kitchenham, Shari Lawrence Pfleeger +5 more
A preliminary set of research guidelines aimed at stimulating discussion among software researchers, intended to assist researchers, reviewers, and meta-analysts in designing, conducting, and evaluating empirical studies.
IEEE Transactions on Knowledge and Data EngineeringBenchmarking attribute selection techniques for discrete class data mining
1,231 Citations2003Mark A. Hall, Geoffrey Holmes
A benchmark comparison of several attribute selection methods for supervised classification by cross-validating the attribute rankings with respect to a classification learner to find the best attributes.
IEEE Transactions on Software EngineeringA critique of software defect prediction models
1,032 Citations1999Norman Fenton, Martin Neil
H holistic models for software defect prediction, using Bayesian belief networks, are recommended as alternative approaches to the single-issue models used at present and research into a theory of "software decomposition" is argued for.
Use of relative code churn measures to predict system defect density
748 Citations2005Nachiappan Nagappan, Thomas Ball
A technique for early prediction of system defect density using a set of relative code churn measures that relate the amount of churn to other variables such as component size and the temporal extent of churn, which shows that while absolute measures of code chum are poor predictors of defect density, these measures are highly predictive of defectdensity.
Empirical Software EngineeringOn the relative value of cross-company and within-company data for defect prediction
722 Citations2009Burak Turhan, Tim Menzies +2 more
It is demonstrated in this paper that the minimum number of data samples required to build effective defect predictors can be quite small and can be collected quickly within a few months.
IEEE Transactions on Software EngineeringPredicting the location and number of faults in large software systems
721 Citations2005Thomas J. Ostrand, Elaine J. Weyuker +1 more
A negative binomial regression model has been developed and used to predict the expected number of faults in each file of the next release of a system, based on the code of the file in the current release, and fault and modification history of thefile from previous releases.
Predicting defects using network analysis on dependency graphs
575 Citations2008Thomas Zimmermann, Nachiappan Nagappan
This paper proposes to use network analysis on dependency graphs of the entire system to identify central program units that are more likely to face defects and finds that the recall for models building from network measures is by 10% points higher than for models built from complexity metrics.
Design and Code Inspections to Reduce Errors in Program Development
564 Citations2001Michael Fagan
ACM Computing SurveysValidation, Verification, and Testing of Computer Software
431 Citations1982W. Richards Adrion, Martha Branstad +1 more
This paper surveys current verification, validation, and testing approaches and discusses their strengths, weaknesses, and life-cycle usage and describes automated tools used toplement vahdation, verification, and testmg.
The influence of organizational structure on software quality
367 Citations2008Nachiappan Nagappan, Thomas Brendan Murphy +1 more
This paper presents a metric scheme to quantify organizational complexity, in relation to the product development process to identify if the metrics impact failure-proneness, and provides empirical evidence that the organizational metrics are related to, and are effective predictors of failure- proneness.
Static analysis tools as early indicators of pre-release defect density
276 Citations2005Nachiappan Nagappan, Thomas Ball
An empirical approach is presented for the early prediction of pre-release defect density based on the defects found using static analysis tools and it is shown that there exists a strong positive correlation between the static analysis defect density and the pre- release defect density determined by testing.
Implications of ceiling effects in defect predictors
184 Citations2008Tim Menzies, Burak Turhan +4 more
This work uses micro-sampling to find much evidence for the limited information hypothesis, and suggests that further progress in learning defect predictors may not come from better algorithms, but from improving the information content of the training data, perhaps with case-based reasoning methods.
Reviewers
171 Citations2007
Empirical Assessment of Machine Learning based Software Defect Prediction Techniques
153 Citations2006V.U.B. Challagulla, Farokh Bastani +2 more
A combination of IR and instance-based learning along with the consistency-based subset evaluation technique provides a relatively better consistency in accuracy prediction compared to other models, and "size" and "complexity" metrics are not sufficient for accurately predicting real-time software defects.
Data & Knowledge EngineeringAnalysis of Naive Bayes’ assumptions on software fault data: An empirical study
133 Citations2008Burak Turhan, Ayşe Bener
This analysis shows that independence assumption is not harmful for software defect data with PCA pre-processing, and indicates that assigning weights to static code attributes may increase the prediction performance significantly, while removing the need for feature subset selection.
Expert Systems with ApplicationsData mining source code for locating software bugs: A case study in telecommunication industry
56 Citations2008Burak Turhan, Gözde Koçak +1 more
This study analyzed 25 projects of a large telecommunication system and trained models on publicly available Nasa MDP data to predict defect proneness of modules, using static call graph based ranking (CGBR) as well as nearest neighbor sampling for constructing method level defect predictors.
A Framework for Software Engineering Experimental Replications
54 Citations2008Manoel Mendonça, José Carlos Maldonado +7 more
A Framework for Improving the Replication of Experiments (FIRE) is proposed, which encourages coordination of replications in order to facilitate knowledge transfer for lower cost, higher quality replications and more generalizable results.
Ensemble of software defect predictors
41 Citations2008Ayşe Tosun, Burak Turhan +1 more
A defect prediction model based on ensemble of classifiers, which has not been fully explored so far in this type of research, is presented and it turns out that this model considerably improves the defect detection capability compared to Naive Bayes algorithm.
Proceedings of the ... EUROMICRO Conference/EUROMICROSoftware Defect Prediction Using Call Graph Based Ranking (CGBR) Framework
33 Citations2008Burak Turhan, Gözde Koçak +1 more
This research proposes static call graph based ranking (CGBR) framework, which can be applied to any defect prediction model based on SCA, and shows that defect predictors using CGBR framework can detect the same number of defective modules, while yielding significantly lower false alarm rates.