The effectiveness of pair programming: A meta-analysis
Information and Software TechnologyPublished 22 February 2009
Jo Erskine Hannay, Tore Dybå, Erik Arisholm, Dag I. K. Sjøberg
Citations258
SJR quartileQ1
SJR score1.04
SNIP2.15
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A meta-analysis of studies on the effects of pair versus solo programming suggests that pair programming is faster than solo programming when programming task complexity is low and yields code solutions of higher quality whentask complexity is high.
Abstract
Several experiments on the effects of pair versus solo programming have been reported in the literature. We present a meta-analysis of these studies. The analysis shows a small significant positive overall effect of pair programming on quality, a medium significant positive overall effect on duration, and a medium significant negative overall effect on effort. However, between-study variance is significant, and there are signs of publication bias among published studies on pair programming. A more detailed examination of the evidence suggests that pair programming is faster than solo programming when programming task complexity is low and yields code solutions of higher quality when task complexity is high. The higher quality for complex tasks comes at a price of considerably greater effort, while the reduced completion time for the simpler tasks comes at a price of noticeably lower quality. We conclude that greater attention should be given to moderating factors on the effects of pair programming.
Keywords
Computer ScienceDecision Sciences
BMJMeasuring inconsistency in meta-analyses
62,756 Citations2003Julian P. T. Higgins, Simon G. Thompson +2 more
A new quantity is developed, I 2, which the authors believe gives a better measure of the consistency between trials in a meta-analysis, which is susceptible to the number of trials included in the meta- analysis.
BiometricsTrim and Fill: A Simple Funnel‐Plot–Based Method of Testing and Adjusting for Publication Bias in Meta‐Analysis
13,841 Citations2000Sue Duval, Richard L. Tweedie
These are simple rank-based data augmentation techniques, which formalize the use of funnel plots, which provide effective and relatively powerful tests for evaluating the existence of publication bias.
Journal of Educational StatisticsStatistical Methods for Meta-Analysis
10,479 Citations1988Stanley Wasserman, Larry V. Hedges +1 more
Practical Meta-Analysis
8,923 Citations2000Mark W. Lipsey, David B. Wilson
This paper presents a meta-analysis procedure called “Meta-Analysis Interpretation for Meta-Analysis Selecting, Computing and Coding the Effect Size Statistic and its applications to Data Management Analysis Issues and Strategies.
Extreme programming explained embrace change
5,875 Citations2008Kent Beck
You may love XP, or you may hate it, but Extreme Programming Explained will force you to take a fresh look at how you develop software.
Journal of Consumer ResearchA Critical Review of Construct Indicators and Measurement Model Misspecification in Marketing and Consumer Research
5,613 Citations2003Cheryl Burke Jarvis, Scott MacKenzie +1 more
A set of conceptual criteria is developed that can be used to determine whether a construct should be modeled as having formative or reflective indicators, and an estimate of the extent of measurement model misspecification in the field is estimated.
Cognitive ScienceCategorization and Representation of Physics Problems by Experts and Novices*
5,230 Citations1981T. H. Michelene, Paul J. Feltovich +1 more
Results from sorting tasks and protocols reveal that experts and novices begin their problem representations with specifiably different problem categories, and completion of the representations depends on the knowledge associated with the categories.
Psychological BulletinConventional wisdom on measurement: A structural equation perspective.
3,430 Citations1991Kenneth A. Bollen, Richard D. Lennox
British Journal of ManagementFormative Versus Reflective Indicators in Organizational Measure Development: A Comparison and Empirical Illustration
3,333 Citations2006Adamantios Diamantopoulos, Judy A. Siguaw
Psychological MethodsFixed- and random-effects models in meta-analysis.
2,838 Citations1998Larry V. Hedges, Jack L. Vevea
Journal of the American Statistical AssociationA Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis
2,596 Citations2000Sue Duval, Richard L. Tweedie
American Sociological ReviewA Framework for the Comparative Analysis of Organizations
2,507 Citations1967Charles Perrow
Publication Bias in Meta‐Analysis
2,197 Citations2005Hannah R. Rothstein, Alex J. Sutton +1 more
This work states that within Conventional Publication Bias: Other Determinants of Data Suppression (Scott D. Halpern and Jesse A. Berlin), differentiating Biases from Genuine Heterogeneity: Distinguishing Artifactual from Substantive Effects (John P.A. Ioannidis).
Psychology Press eBooksThe Nature of Expertise
2,180 Citations2014Michelene T.H., Robert Glaser +1 more
This chapter discusses Expertise on the Bench: Modeling Magistrates' Judicial Decision-Making and Expertise in a Complex Skill: Diagnosing X-Ray Pictures, and examines the relationship between Comprehension and Reasoning in Medical Expertise.
Journal of Personality and Social PsychologySocial loafing: A meta-analytic review and theoretical integration.
2,094 Citations1993Steven J. Karau, Kipling D. Williams
American PsychologistExpert performance: Its structure and acquisition.
1,993 Citations1994K. Anders Ericsson, Neil Charness
Annual Review of PsychologyMeta-Analysis: Recent Developments in Quantitative Methods for Literature Reviews
1,805 Citations2001Robert Rosenthal, M. Robin DiMatteo
Advantages of the meta-analytic procedures include seeing the "landscape" of a research domain, keeping statistical significance in perspective, minimizing wasted data, becoming intimate with the data summarized, asking focused research questions, and finding moderator variables.
Expertise in Problem Solving.
1,429 Citations1981T. H. Michelene, Robert Glaser +1 more
An examination of the shift from consideration of general, domain-independent skills and procedures, in both cognitive psychology and artificial intelligence, to the study of the knowledge base shows the importance of differences in the knowledge bases of experts and novices to their problem solving success.
Annual Review of PsychologyGroup Performance and Decision Making
1,346 Citations2004Norbert L. Kerr, R. Scott Tindale
Recent trends in group performance research have found that process gains as well as losses are possible, and both are frequently explained by situational and procedural contexts that differentially affect motivation and resource coordination.
PsychometrikaH.R. Rothstein, A.J. Sutton, & M. Borenstein (Eds.) (2005). Publication bias in meta-analysis: Prevention, assessment and adjustments. NewYork: Wiley, xvii+356 p. US$100.00. ISBN: 0-470-87014-1.
1,264 Citations2007Wolfgang Viechtbauer
Addison-Wesley Professional eBooksExtreme Programming Explained: Embrace Change (2nd Edition)
944 Citations2004Kent Beck, Cynthia Andres
Annual Review of PsychologyProgress in Small Group Research
933 Citations1990Jerrold M. Levine, Richard L. Moreland
International Journal of Selection and AssessmentFixed Effects vs. Random Effects Meta‐Analysis Models: Implications for Cumulative Research Knowledge
835 Citations2000John E. Hunter, Frank L. Schmidt
IEEE SoftwareStrengthening the case for pair programming
770 Citations2000Laurie Williams, Robert R. Kessler +2 more
Regression Methods to Detect Publication and Other Bias in Meta‐Analysis
740 Citations2005Jonathan A C Sterne, Matthias Egger
Statistics in MedicinePerformance of the trim and fill method in the presence of publication bias and between‐study heterogeneity
709 Citations2007Jaime Peters, Alex J. Sutton +3 more
Using the trim and fill method as a form of sensitivity analysis as intended by the authors of the method can help to reduce the bias in pooled estimates, even though the performance of this method is not ideal.
IEEE Transactions on Software EngineeringA survey of controlled experiments in software engineering
680 Citations2005D.I.K. Sjoeberg, Jo Erskine Hannay +5 more
This paper reports upon the present state of how controlled experiments in software engineering are conducted and the extent to which relevant information is reported, and quantitatively characterizes the topics and their subjects, tasks, environments, and environments.
Defense Technical Information Center (DTIC)Expertise in Problem Solving.
622 Citations1981Chi,Michelene T H, Glaser,Robert +1 more
CERN Document Server (European Organization for Nuclear Research)Pair programming illuminated
560 Citations2007Laurie Williams, Robert R. Kessler
This book describes the technique of pair programming from many perspectives and offers ideas to help enhance the pairing and solutions for most problem pairings; not all pairs will work and the book provides ways to recognize the potential problems before they happen.
Cognitive PsychologyStimulus structures and mental representations in expert comprehension of computer programs
516 Citations1987Nancy Pennington
Investigating the role of programming knowledge in program comprehension and the nature of mental representations of programs suggests that procedural rather than functional units form the basis of expert programmers' mental representations, supporting work in other areas of text comprehension and showing the importance of text structure knowledge in understanding.
IEEE SoftwareEvidence-based software engineering for practitioners
455 Citations2005T. Dyba, Barbara Kitchenham +1 more
It is suggested that practitioners consider evidence-based software engineering as a mechanism to support and improve their technology adoption decisions.
The American StatisticianThe Design of Replicated Studies
450 Citations1993Roger Lindsay, A. S. C. Ehrenberg
Communications of the ACMThe case for collaborative programming
377 Citations1998John T. Nosek
A follow-up field experiment was conducted using experienced programmers who worked on a challenging problem important to their organization, in their own environments, and with their own equipment, and to the surprise of the managers and participants, all the teams outperformed the individual programmers, enjoyed the problem-solving process more, and had greater confidence in their solutions.
Information and Software TechnologyA systematic review of effect size in software engineering experiments
369 Citations2007Vigdis By Kampenes, Tore Dybå +2 more
The standardized effect sizes computed from the reviewed experiments were equal to observations in psychology studies and slightly larger than standard conventions in behavioral science.
Applying Systematic Reviews to Diverse Study Types: An Experience Report
359 Citations2007Tore Dybå, Torgeir Dingsøyr +1 more
The intuition that the more missing values, the poorer the prediction accuracy of AQUA is confirmed and the upper limit of missing values for the applicability of AquA is determined as 40%.
A Visual Text Mining approach for Systematic Reviews
330 Citations2007Viviane Malheiros, Erika Nina Höhn +3 more
It is observed that VTM can contribute to systematic review and a new strategy is proposed, called VTM-Based systematic review, which is proposed to aid systematic reviews of software engineering research topics.
Information and Software TechnologyA systematic review of statistical power in software engineering experiments
288 Citations2005Tore Dybå, Vigdis By Kampenes +1 more
The results show that the statistical power of software engineering experiments falls substantially below accepted norms as well as the levels found in the related discipline of information systems research.
IEEE Transactions on Software EngineeringEvaluating Pair Programming with Respect to System Complexity and Programmer Expertise
267 Citations2007Erik Arisholm, Hans Gallis +2 more
The results of this experiment do not support the hypotheses that pair programming in general reduces the time required to solve the tasks correctly or increases the proportion of correct solutions.
IEEE Transactions on Software EngineeringA Systematic Review of Theory Use in Software Engineering Experiments
228 Citations2007Jo Erskine Hannay, Dag I. K. Sjøberg +1 more
It is found that theory use and awareness of theoretical issues are present, but that theory-driven research is, as yet, not a major issue in empirical software engineering.
Journal of Systems and SoftwareApplying meta-analytical procedures to software engineering experiments
149 Citations2000James Miller
The paper outlines various ideas from other disciplines for controlling this variation and describes a number of recommendations for controlling and reporting empirical work to advance the discipline towards a position, where meta-analysis can be profitably employed.
IEEE SoftwareAre Two Heads Better than One? On the Effectiveness of Pair Programming
139 Citations2007Tore Dybå, Erik Arisholm +3 more
This article concentrates on understanding general aspects related to effectiveness, specifically project duration, effort, and quality.
Apress eBooksExtreme Programming Refactored: The Case Against XP
134 Citations2003Matt Stephens, Doug Rosenberg
Stephens and Rosenberg examine XP in the context of existing methodologies and processes such as RUP, ICONIX, Spiral, RAD, DSDM, etc and show how XP goals can be achieved using these existing processes.
Proceedings/Proceedings - International Conference on Software EngineeringThe Social Dynamics of Pair Programming
127 Citations2007Jan Chong, Tom Hurlbutt
The distribution of expertise among the members of a pair had a strong influence on the tenor of pair programming interaction and keyboard control had a consistent secondary effect on decisionmaking within the pair.
Lecture notes in computer scienceExploring the Efficacy of Distributed Pair Programming
113 Citations2002Prashant Baheti, Edward F. Gehringer +1 more
The results of the experiment indicate that it is feasible to develop software using distributed pair programming, and that the resulting software is comparable to software developed in collocated or virtual teams (without pair programming) in productivity and quality.
Information and Software TechnologyA systematic review of quasi-experiments in software engineering
111 Citations2008Vigdis By Kampenes, Tore Dybå +2 more
Quasi-experiments are useful in many settings in SE, but their design and analysis must be improved to ensure that inferences made from this kind of experiment are valid, according to this paper.
Journal of Applied PsychologyThe influence of degree of expertise and objective task complexity on perceived task complexity and performance.
100 Citations2007Thorvald Hærem, Devaki Rau
The authors examined how 3 sets of similar tasks that differ in their type of complexity can lead to differences in task perceptions and performance among experts, intermediates, and novices.
Journal of Systems and SoftwareTwo controlled experiments concerning the comparison of pair programming to peer review
82 Citations2005Matthias Müller
The study has one major finding concerning the cost of the two development methods, programmer pairs and single developers become interchangeable in terms of development cost.
Evidence Concerning the Consequences of Publication and Related Biases
78 Citations2005Alex J. Sutton
Journal of Systems and SoftwareEvaluating performances of pair designing in industry
69 Citations2006Gerardo Canfora, A. Cimitile +3 more
The results of the experiment suggest that pair programming slows down the task, yet improves quality, and are compared with those of a previous exploratory experiment involving students, and demonstrate how the outcomes exhibit very similar trends.
Effects of pair programming at the development team level: an experiment
68 Citations2005Jari Vanhanen, Casper Lassenius
The pair programming teams wrote code with fewer defects, but were less careful in system testing, and therefore delivered systems with more defects and weak support for higher enjoyment of work in the pair Programming teams is found.
Mining Software Evolution to Predict Refactoring
63 Citations2007Jacek Ratzinger, Thomas Sigmund +2 more
In an empirical study of open source projects, it is found that attributes of software evolution data can be used to predict the need for refactoring in the following two months of development.
IEEE Transactions on Software EngineeringThe Role of Deliberate Artificial Design Elements in Software Engineering Experiments
54 Citations2008Jo Erskine Hannay, Magne Jørgensen
It is concluded that an increased awareness and deliberation as to when and for what purposes both artificial and realistic design elements are applied is valuable for better knowledge gain and quality in empirical software engineering experiments.
Empirical Validation of Test-Driven Pair Programming in Game Development
37 Citations2006Shaochun Xu, Václav Rajlich
The results of the case study showed that paired students completed their tasks faster and with higher quality than individuals, and the programs written by pairs pass more test cases than those developed by individuals.
An empirical comparison between pair development and software inspection in Thailand
33 Citations2006Monvorath Phongpaibul, Barry Boehm
The objectives of this study are to compare the commonalities and differences between pair development and software inspection as verification techniques in Thailand and the impacts of cultural differences to the adoption of pair programming or software inspection in Thailand.
Lecture notes in computer scienceOn the Effects of Pair Programming on Thoroughness and Fault-Finding Effectiveness of Unit Tests
32 Citations2007Lech Madeyski
It appeared that branch coverage, as well as mutation score indicator, was not significantly affected by using pair programming, instead of solo programming, which does not support the positive impact of pair programming on testing to make it more effective and thorough.
Lecture notes in computer scienceTracking Test First Pair Programming — An Experiment
28 Citations2002Matevz Rostaher, Marjan Heričko
The authors ran an experiment where a group of professional programmers working in pairs and a control group programming alone implemented a small system from predefined requirements showed that both groups spent almost the same amount of time to complete the tasks.
"Talking the talk": Is intermediate-level conversation the key to the pair programming success story?
25 Citations2007Sallyann Freudenberg, Pablo Romero +1 more
The findings suggest that the key to the success of pair programming does not lie in the differences in behaviour or focus between the driver and navigator, but in the proliferation of talk at an intermediate level of detail in pair programmers' conversations.
Lecture notes in computer scienceEmpirical Study on the Productivity of the Pair Programming
17 Citations2005Gerardo Canfora, A. Cimitile +1 more
Empirical evidence is obtained that pair programming decreases the effort of the single programmer when performing pair programming and when working as solo programmer.
IEEE SoftwareFaster, better and cheaper
17 Citations2001J. Voas
The author discusses his beliefs on whether the phrase "faster, better, and cheaper" has any real utility for the software industry, or whether this is simply wishful thinking given the authors' current technologies.