A Ranking Stability Indicator for Selecting the Best Effort Estimator in Software Cost Estimation
Published 1 January 2011
Jacky Keung, Ekrem Kocagüneli, Tim Menzies
Citations15
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Based on the proposed evaluation method, the most suitable local estimator for software cost estimation is able to be determined, an important process in the application of any effort estimation analysis.
Abstract
Software effort estimation research shows that there is no universal agreement on the effort estimation approach. This is largely due to the ranking problem, which is highly contingent on the evaluation criteria and the subset of the data used in the investigation. There are a large number of different method combination exists for software effort estimation, selecting the most suitable combination becomes the subject of research in this paper. Unless we can reasonably determine stable rankings of different estimators, we cannot determine the most suitable estimator for effort estimation. This pa- per reports an empirical study using 90 estimation methods applied to 20 datasets as an attempt to address this question. Performance was assessed using MAR, MMRE, MMER, MBRE, MIBRE, MdMRE, PRED(25) and compared using a Wilcoxon ranked test (95%). An comprehensive empirical experiment was carried out. Result shows prior studies of rank- ing instability of effort estimation approaches may have been overly pessimistic. Given the large number of datasets, it is now possible to draw stable conclusions about the relative performance of different effort estimation methods and to select the most suitable ones for the study under investigation. In this study, regression trees or analogy-based methods are the best performers in the experiment, and we recommend against neural nets or simple lin- ear regression. Based on the proposed evaluation method, we are able to determine the most suitable local estimator for software cost estimation, an important process in the application of any effort estimation analysis.
Keywords
Computer Science
BiometricsClassification and Regression Trees.
23,841 Citations1984Alexander Gordon, Leo Breiman +3 more
Medical Entomology and ZoologyReal World Research: A Resource for Social Scientists and Practitioner-Researchers
6,247 Citations1993Colin Robson
International Joint Conference on Artificial IntelligenceMulti-Interval Discretization of Continuous-Valued Attributes for Classification Learning
2,832 Citations1993Usama M. Fayyad, Keki B. Irani
This paper addresses the use of the entropy minimization heuristic for discretizing the range of a continuous-valued attribute into multiple intervals.
Series in machine perception and artificial intelligenceIntroduction to Machine Learning
1,654 Citations2019
Introduction to Machine Learning is a comprehensive textbook on the subject, covering a broad array of topics not usually included in introductory machine learning texts, and discusses many methods from different fields, including statistics, pattern recognition, neural networks, artificial intelligence, signal processing, control, and data mining.
IEEE Transactions on Knowledge and Data EngineeringBenchmarking attribute selection techniques for discrete class data mining
1,231 Citations2003Mark A. Hall, Geoffrey Holmes
A benchmark comparison of several attribute selection methods for supervised classification by cross-validating the attribute rankings with respect to a classification learner to find the best attributes.
IEEE Transactions on Software EngineeringEstimating software project effort using analogies
965 Citations1997Martin Shepperd, C. Schofield
It is argued that estimation by analogy is a viable technique that, at the very least, can be used by project managers to complement current estimation techniques.
Journal of Systems and SoftwareA review of studies on expert estimation of software development effort
611 Citations2003Magne Jørgensen
An extensive review of studies related to expert estimation of software development effort suggests that expert estimation is the most frequently applied estimation strategy for software projects, that there is no substantial evidence in favour of use of estimation models, and that there are situations where the authors can expect expert estimates to be more accurate than formal estimation models.
IEEE Transactions on Software EngineeringA simulation study of the model evaluation criterion mmre
477 Citations2003Tron Foss, E. Stensrud +2 more
A simulation study demonstrating that MMRE does not always select the best model is performed, casting some doubt on the conclusions of any study of competing software prediction models that use MMRE as a basis of model comparison.
IEEE Transactions on Software EngineeringCross versus Within-Company Cost Estimation Studies: A Systematic Review
420 Citations2007Barbara Kitchenham, Emília Mendes +1 more
It is clear that some organizations would be ill-served by cross-company models whereas others would benefit, and experimenters need to standardize their experimental procedures to enable formal meta-analysis of the results.
IEEE Transactions on ComputersFinding Prototypes For Nearest Neighbor Classifiers
327 Citations1974Chin-Liang Chang
An algorithm is given to find prototypes for a nearest neighbor classifier, which is to start with every sample in a training set as a prototype, and then successively merge any two nearest prototypes of the same class so long as the recognition rate is not downgraded.
IEEE Transactions on Software EngineeringComparing software prediction techniques using simulation
282 Citations2001Martin Shepperd, Gada Kadoda
The authors compare four prediction techniques: regression, rule induction, nearest neighbor (a form of case-based reasoning), and neural nets and suggest that there are significant differences depending upon the characteristics of the data set.
IEEE Transactions on Software EngineeringReliability and validity in comparative studies of software prediction models
260 Citations2005Ingunn Myrtveit, E. Stensrud +1 more
It is suggested that it is the research procedure itself that is unreliable, and this lack of reliability may strongly contribute to the lack of convergence in empirical studies on software prediction models.
International Conference on Software EngineeringEffort estimation using analogy
253 Citations1996Martin Shepperd, Chris Schofield +1 more
Empirical Software EngineeringAn Empirical Study of Analogy-based Software Effort Estimation
239 Citations1999Fiona Walkerden, Ross Jeffery
Comparing several methods of analogy-based software effort estimation with each other and also with a simple linear regression model shows that people are better than tools at selecting analogues for the data set used in this study.
Empirical Software EngineeringA Comparative Study of Cost Estimation Models for Web Hypermedia Applications
200 Citations2003Emília Mendes, Ian Watson +3 more
A comparison of the prediction accuracy of three CBR techniques used to estimate the effort to develop Web hypermedia applications and to choose the one with the best estimates is presented.
Journal of Systems and SoftwareA study of project selection and feature weighting for analogy based software cost estimation
193 Citations2008Yan‐Fu Li, Min Xie +1 more
This paper proposes the project selection technique for ABE (PSABE) which reduces the whole project base into a small subset that consist only of representative projects and shows promising results that could significantly improve analogy based models for software cost estimation.
A comparative study of discretization methods for naive-Bayes classifiers
146 Citations2002Ying Yang, Geoffrey I. Webb
This study leads to a new discretization method, weighted non-disjoint discretiza- tion (WNDD) that combines WPKID and NDD's advantages, and shows that among all the rival discretized methods, WNDD best helps naive-Bayes classifiers reduce average classification error.
IEEE Transactions on Software EngineeringAnalogy-X: Providing Statistical Inference to Analogy-Based Software Cost Estimation
139 Citations2008Jacky Keung, Barbara Kitchenham +1 more
Analogy-X provides a sound statistical basis for analogy, removes the need for heuristic search and greatly improves its algorithmic performance.
IEEE SoftwarePractical guidelines for expert-judgment-based software effort estimation
136 Citations2005Magne Jørgensen
This article presents seven guidelines for producing realistic software development effort estimates that derive from industrial experience and empirical studies and use the most recent findings regarding judgment-based effort estimation.
IEEE Transactions on Software EngineeringOptimal project feature weights in analogy-based cost estimation: improvement and limitations
123 Citations2006Martin Auer, Adam Trendowicz +3 more
This work proposes to allocate separate weights to each project feature and to find the optimal weights by extensive search, and finds that this method increases estimation accuracy and reliability, reduces the model's volatility and, thus, is likely to increase its acceptance in practice.
Discretization from data streams
92 Citations2006João Gama, Carlos Pinto
The proposed architecture processes streaming examples in a single scan, in constant time and space even for infinite sequences of examples, much more appropriate in incremental learning, and in problems where data flows continuously, as in most of the recent data mining applications.
IEE Proceedings - SoftwareMaking inferences with small numbers of training sets
62 Citations2002Colin Kirsopp, Martin Shepperd
A potential methodological problem with empirical studies that assess project effort prediction system is discussed, and it is concluded that using less than five training sets leads to untrustworthy results, and ideally more than 20 sets should be deployed.
Automated Software EngineeringStable rankings for different effort models
50 Citations2010Tim Menzies, Omid Jalali +3 more
While there exists no single universal “best” effort estimation method, there appears to exist a small number of most useful methods, which should be preceded by a “selection study” that finds the best local estimator.
Pattern Recognition LettersSelection of the optimal prototype subset for 1-NN classification
47 Citations1998Uri Lipowezky
An algorithm for the selection of an optimal finite prototype or training sample subset that maximizes the quality or minimizes the recognition error of a finite test sample set and improves the classification quality is presented.
A Hybrid Approach to Expert and Model Based Effort Estimation
44 Citations2007Daniel R. Baker
This thesis describes how 11 of Jorgensen’s 12 expert judgment best practices may be automated in a model, 7 of which are demonstrated in 2CEE, and the distinction between manual and automated methods for effort estimation is questioned.
research memorandumSensitivity analysis and related analysis : A survey of statistical techniques
43 Citations1995J.P.C. Kleijnen
This paper reviews the state of the art in five related types of analysis, namely (i) sensitivity or what-if analysis, (ii) uncertainty or risk analysis,(iii) screening, (iv) validation, and (v) optimization.
Software Quality JournalA new perspective on data homogeneity in software cost estimation: a study in the embedded systems domain
41 Citations2009Ayşe Bakır, Burak Turhan +1 more
A new aspect of data homogeneity, “cross- versus within-application domain”, is focused on, and what kind of training data should be used for software cost estimation in the embedded systems domain is investigated, and the effect of training dataset size on the prediction performance is found.
A comparative study of attribute weighting heuristics for effort estimation by analogy
31 Citations2006Jingzhou Li, Guenther Ruhe
Five heuristics for attribute weighting in analogy-based effort estimation are evaluated over five data sets related to software projects and indicate that three of the rough set analysis based heuristic perform better than the equal weights heuristic.
Empirical evaluation of analogy-x for software cost estimation
31 Citations2008Jacky Keung
The results show that the prediction accuracy of Analogy-X is similar to the one of ANGEL, and the use of Mantel statistics to select project features and detect abnormal data points, which provides a sound statistical basis for analogy-based systems.
Decision Support Analysis for Software Effort Estimation by Analogy
27 Citations2007Jingzhou Li, Guenther Ruhe
This paper proposes a decision-centric process model for EBA by generalizing the existing EBA methods and using the results of the decision support analysis for better understanding of EBA related techniques.
Proceedings - Australian Software Engineering Conference/ProceedingsExperiments with Analogy-X for Software Cost Estimation
22 Citations2008Jacky Keung, Barbara Kitchenham
Analogy-X is a method to statistically evaluate the relationship between useful project features and target features such as effort to be estimated, which ensures the dataset used is relevant to the prediction problem, and project features are selected based on their statistical contribution to the target variables.
Proceedings. 30th Euromicro Conference, 2004.Distribution patterns of effort estimations
21 Citations2004D. Milicic, Claes Wohlin
Case-based reasoning vs parametric models for software quality optimization
17 Citations2010Adam Brady, Tim Menzies
The W case-based reasoning tool is recommended over the SEESAW parametric modeling tool for purposes of quality optimization (except in the case where there is no local data).