Cross versus Within-Company Cost Estimation Studies: A Systematic Review
IEEE Transactions on Software EngineeringPublished 25 April 2007
Barbara Kitchenham, Emília Mendes, Guilherme Horta Travassos
Citations420
SJR quartileQ1
SJR score1.45
SNIP2.43
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
It is clear that some organizations would be ill-served by cross-company models whereas others would benefit, and experimenters need to standardize their experimental procedures to enable formal meta-analysis of the results.
Abstract
The objective of this paper is to determine under what circumstances individual organizations would be able to rely on cross-company-based estimation models. We performed a systematic review of studies that compared predictions from cross-company models with predictions from within-company models based on analysis of project data. Ten papers compared cross-company and within-company estimation models; however, only seven presented independent results. Of those seven, three found that cross-company models were not significantly different from within-company models, and four found that cross-company models were significantly worse than within-company models. Experimental procedures used by the studies differed making it impossible to undertake formal meta-analysis of the results. The main trend distinguishing study results was that studies with small within-company data sets (i.e., $20 projects) that used leave-one-out cross validation all found that the within-company model was significantly different (better) from the cross-company model. The results of this review are inconclusive. It is clear that some organizations would be ill-served by cross-company models whereas others would benefit. Further studies are needed, but they must be independent (i.e., based on different data bases or at least different single company data sets) and should address specific hypotheses concerning the conditions that would favor cross-company or within-company models. In addition, experimenters need to standardize their experimental procedures to enable formal meta-analysis, and recommendations are made in Section 3.
Keywords
Computer Science
Practical Meta-Analysis
8,923 Citations2000Mark W. Lipsey, David B. Wilson
This paper presents a meta-analysis procedure called “Meta-Analysis Interpretation for Meta-Analysis Selecting, Computing and Coding the Effect Size Statistic and its applications to Data Management Analysis Issues and Strategies.
Choice Reviews OnlineSystematic reviews in the social sciences: a practical guide
4,107 Citations2006
The systematic review is a scientific tool that can help with appraising, summarizing, and communicating the results and implications of otherwise unmanageable quantities of data.
Communications of the ACMAn empirical validation of software cost estimation models
903 Citations1987Chris F. Kemerer
Four of the most popular algorithmic models used to estimate software costs (SLIM, COCOMO, Function Points, and ESTIMACS) are evaluated, finding that all of the models tested failed to sufficiently reflect the underlying factors affecting productivity.
IEEE Transactions on Software EngineeringA General Empirical Solution to the Macro Software Sizing and Estimating Problem
801 Citations1978Lawrence H. Putnam
Application software development has been an area of organizational effort that has not been amenable to the normal managerial and cost controls.
Software engineering economics
718 Citations2002Hakan Erdogmus, Barry Boehm +3 more
IEEE Transactions on Software EngineeringA simulation study of the model evaluation criterion mmre
477 Citations2003Tron Foss, E. Stensrud +2 more
A simulation study demonstrating that MMRE does not always select the best model is performed, casting some doubt on the conclusions of any study of competing software prediction models that use MMRE as a basis of model comparison.
IEEE Transactions on Software EngineeringCross versus Within-Company Cost Estimation Studies: A Systematic Review
420 Citations2007Barbara Kitchenham, Emília Mendes +1 more
It is clear that some organizations would be ill-served by cross-company models whereas others would benefit, and experimenters need to standardize their experimental procedures to enable formal meta-analysis of the results.
PubMedSystematic reviews and meta-analyses: an illustrated, step-by-step guide.
350 Citations2004Madhukar Pai, Michael McCulloch +6 more
This article is an illustrated guide for conducting systematic reviews that will enable clinicians to conduct systematic reviews, generate high-quality evidence, and contribute to the evidence-based medicine movement.
An assessment and comparison of common software cost estimation modeling techniques
275 Citations1999Lionel Briand, Khaled El Emam +3 more
This paper assess and compare a selection of common cost modeling techniques fulfilling a number of important criteria using a large multi-organizational database in the business application domain and shows that the performances of the modeling techniques considered were not significantly different, with the exception of the analogy-based models which appear to be less accurate.
A replicated assessment and comparison of common software cost modeling techniques
220 Citations2000Lionel Briand, Tristen Langley +1 more
Traditional techniques, namely, ordinary least-squares regression and analysis of variance outperformed analogy-based estimation and regression trees and no significant difference was found in accuracy between estimates derived from company-specific data and estimatesderived from multi-organizational data.
Using public domain metrics to estimate software development effort
186 Citations2001R. Jeffery, Melanie Ruhe +1 more
The accuracy of cost estimates when applying most commonly used modeling techniques to a large-scale industrial data set which is professionally maintained by the International Software Standards Benchmarking Group (ISBSG) is investigated, and the estimation accuracy is significantly higher than when using the rest of the ISBSG data set.
Information and Software TechnologyA comparative study of two software development cost modeling techniques using multi-organizational and company-specific data
162 Citations2000Richard Jeffery, Martin Ruhé +1 more
Using multi-company data the OLS regression model provided significantly more accurate results than Analogy-based predictions, and it was found in general that models based on the company-specific data resulted in significant more accurate estimates.
Information and Software TechnologyEmpirical studies of assumptions that underlie software cost-estimation models
126 Citations1992BA Kitchenham
The results indicate that the assumption that there is a nonlinear relationship between size and effort is not supported, but the assumption of a non linear relationship between effort and duration is.
Electronic workshops in computingA Systematic Review of Cross- vs. Within-Company Cost Estimation Studies
126 Citations2006Barbara Kitchenham, Emília Mendes +1 more
The consistency of empirical comparisons of regression and analogy-based software project cost prediction
92 Citations2005Carolyn Mair, Martin Shepperd
The objective is to determine the consistency within and between results in empirical studies of software engineering cost estimation, and to argue that this inconsistent pattern from 20 different studies comparing regression and analogy is somewhat disturbing.
Lecture notes in computer scienceUsing Genetic Programming to Improve Software Effort Estimation Based on General Data Sets
79 Citations2003Martin Lefley, Martin Shepperd
The evidence here agrees with other researchers that companies would do well to base estimates on in house data rather than incorporating public data sets, and offers insights into genetic programming's performance, relative to alternative methods, as a problem solver in this domain.
Management SciencePerformance Evaluation of General and Company Specific Models in Software Development Effort Estimation
77 Citations1999Katrina Maxwell, Luk N. Van Wassenhove +1 more
This paper develops and evaluates simple empirical effort estimation models that include only those productivity factors found to be significant for these projects and determines if models based on a multicompany database can be successfully used to make effort estimations within a specific company.
Further comparison of cross-company and within-company effort estimation models for web applications
73 Citations2004E. Mendes, Barbara Kitchenham
The MERMAID Approach to software cost estimation
69 Citations1990P. A. M. Kok, Barbara Kitchenham +1 more
This paper presents an analysis of the problems in the field of software cost models and describes the MERMAID approach to cost estimation, which makes intensive use of local historical project data and is applicable in all sectors where project data can be collected.
A Replicated Comparison of Cross-Company and Within-Company Effort Estimation Models Using the ISBSG Database
63 Citations2005E. Mendes, Chris Lokan +2 more
This paper replicates a previous study by investigating how successful a cross-company cost model is to estimate effort for projects that belong to a single company and were not used to build the cross- company model; i) compared to a within- company cost model.
IEE Proceedings - SoftwareMaking inferences with small numbers of training sets
62 Citations2002Colin Kirsopp, Martin Shepperd
A potential methodological problem with empirical studies that assess project effort prediction system is discussed, and it is concluded that using less than five training sets leads to untrustworthy results, and ideally more than 20 sets should be deployed.
How valuable is company-specific data compared to multi-company data for software cost estimation?
55 Citations2003Isabella Wieczorek, Melanie Ruhe
A large cost database from Finland that collects standard cost factors and includes information on six individual companies was analyzed and it is shown that company-specific models seem not to yield better results than the company external models.
A comparison of cross-company and within-company effort estimation models for Web applications
49 Citations2004Barbara Kitchenham
It is concluded that cross-company effort estimation model can be useful for companies that do not have past projects from which to develop their own models, however, a cross- company data set should only be used for estimation until it is possible for a Web company to build its own within-company model.