Beyond Power Calculations
Perspectives on Psychological SciencePublished 1 November 2014
Andrew Gelman, John B. Carlin
Citations1,358
SJR quartileQ1
SJR score5.39
SNIP4.96
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The largest challenge in a design calculation: coming up with reasonable estimates of plausible effect sizes based on external information is discussed, and design calculations in which the probability of an estimate being in the wrong direction and the magnitude of an effect might be overestimated are recommended.
Abstract
Statistical power analysis provides the conventional approach to assess error rates when designing a research study. However, power analysis is flawed in that a narrow emphasis on statistical significance is placed as the primary focus of study design. In noisy, small-sample settings, statistically significant results can often be misleading. To help researchers address this problem in the context of their own studies, we recommend design calculations in which (a) the probability of an estimate being in the wrong direction (Type S [sign] error) and (b) the factor by which the magnitude of an effect might be overestimated (Type M [magnitude] error or exaggeration ratio) are estimated. We illustrate with examples from recent published research and discuss the largest challenge in a design calculation: coming up with reasonable estimates of plausible effect sizes based on external information.
Keywords
Psychology
Computers Environment and Urban SystemsStatistical power analysis for the behavioral sciences
65,645 Citations1990
Nature reviews. NeurosciencePower failure: why small sample size undermines the reliability of neuroscience
7,878 Citations2013Katherine S. Button, John P. A. Ioannidis +5 more
It is shown that the average statistical power of studies in the neurosciences is very low, and the consequences include overestimates of effect size and low reproducibility of results.
Psychological ScienceFalse-Positive Psychology
6,712 Citations2011Joseph P. Simmons, Leif D. Nelson +1 more
It is shown that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings, flexibility in data collection, analysis, and reporting dramatically increases actual false- positive rates, and a simple, low-cost, and straightforwardly effective disclosure-based solution is suggested.
Proceedings of the National Academy of SciencesEvidence on the impact of sustained exposure to air pollution on life expectancy from China’s Huai River policy
1,765 Citations2013Yuyu Chen, Avraham Ebenstein +2 more
The analysis suggests that long-term exposure to an additional 100 μg/m3 of TSPs is associated with a reduction in life expectancy at birth and that life expectancies are about 5.5 y lower in the north owing to an increased incidence of cardiorespiratory mortality.
Perspectives on Psychological SciencePuzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition
1,506 Citations2009Edward Vul, Christine R. Harris +2 more
It is argued that, in some cases, other analysis problems likely created entirely spurious correlations and the data from these studies could be reanalyzed with unbiased methods to provide accurate estimates of the correlations in question and urge authors to perform such reanalyses.
BMJSifting the evidence---what's wrong with significance tests? Another comment on the role of statistical methods
1,496 Citations2001Jonathan A C Sterne
Perspectives on Psychological ScienceScientific Utopia
1,336 Citations2012Brian A. Nosek, Jeffrey R. Spies +1 more
Strategies for improving scientific practices and knowledge accumulation are developed that account for ordinary human motivations and biases and can reduce the persistence of false findings.
PainAcceptance-based interventions for the treatment of chronic pain: A systematic review and meta-analysis
772 Citations2011M.M. Veehof, Maarten-Jan Oskam +2 more
It is suggested that at present mindfulness‐based stress reduction program and acceptance and commitment therapy are not superior to cognitive behavioral therapy but can be good alternatives and recommended to focus on therapies that integrate mindfulness and behavioral therapy.
Annals of Internal MedicineThe Use of Predicted Confidence Intervals When Planning Experiments and the Misuse of Power When Interpreting Results
633 Citations1994Steven N. Goodman, Jesse A. Berlin
The problems with the framework of statistical power are elucidated in the course of explaining why post hoc estimates of power are of little help in interpreting results and why the focus of attention should be exclusively on confidence intervals.
The LancetSample size calculations in randomised trials: mandatory and mystical
530 Citations2005Kenneth F. Schulz, David A. Grimes
Computational StatisticsType S error rates for classical and Bayesian single and multiple comparison procedures
421 Citations2000Andrew Gelman, Francis Tuerlinckx
The Journal of PoliticsReexamining the "Minimal Effects" Model in Recent Presidential Campaigns
408 Citations1993Steven E. Finkel
American Journal of Political ScienceVoter Decision Making in Election 2000: Campaign Effects, Partisan Activation, and the Clinton Legacy
223 Citations2003D. Sunshine Hillygus, Simon Jackman
EpidemiologyCommentary
218 Citations2012Andrew Gelman
Some connections between P values and Bayesian posterior probabilities are discussed and how P values approximate posterior probabilities under prior distributions that contain little information relative to the data is described.
Journal of Educational StatisticsEstimation of Effect Size under Nonrandom Sampling: The Effects of Censoring Studies Yielding Statistically Insignificant Mean Differences
208 Citations1984Larry V. Hedges
British Journal of Mathematical and Statistical PsychologyEstimating effect size: Bias resulting from the significance criterion in editorial decisions
177 Citations1978David M. Lane, William P. Dunlap
Journal of Animal ScienceStatistical power calculations1
164 Citations2007Russell V. Lenth
This article focuses on how to do meaningful power calculations and sample-size determination for common study designs and certain types of retrospective power calculations should be avoided.
Developmental PsychologyAn examination of stereotype threat effects on girls’ mathematics performance.
143 Citations2013Colleen M. Ganley, Leigh A. Mingle +4 more
No evidence that the mathematics performance of school-age girls was impacted by stereotype threat, and the possibility that the literature regarding stereotype threat in children is subject to publication bias is addressed.
Journal of ManagementThe Connection Between Varying Treatment Effects and the Crisis of Unreplicable Research
118 Citations2014Andrew Gelman
Tobacco ControlMortality and life expectancy in relation to long-term cigarette, cigar and pipe smoking: The Zutphen Study
78 Citations2007Martinette T. Streppel, Hendriek C. Boshuizen +3 more
Cigar or pipe smoking reduces life expectancy to a lesser extent than cigarette smoking, whereas both the number of cigarettes smoked as well as duration of smoking were strongly associated with all-cause mortality.
Psychological ScienceThe Fluctuating Female Vote
76 Citations2013Kristina M. Durante, Ashley Rae +1 more
Overall, the ovulatory cycle not only influences women’s politics but also appears to do so differently for single women than for women in relationships.
BMJPower is indeed irrelevant in interpreting completed studies
61 Citations2002Stephen Senn
It is argued that power is important in choosing between alternative methods of analysing data and in deciding on an appropriate size of experiment and that it is quite …
Journal of Theoretical BiologyBeautiful parents have more daughters: A further implication of the generalized Trivers–Willard hypothesis (gTWH)
44 Citations2006Satoshi Kanazawa
It is predicted that physically attractive parents have a lower-than-expected offspring sex ratio (more daughters) and, if beautiful parents have more daughters and physical attractiveness is heritable, women should gradually become more attractive than men.
European Journal of EpidemiologyDistinguishing true from false positives in genomic studies: p values
41 Citations2013Linda Broer, Christina M. Lill +6 more
This work studied the effect of p value thresholds on the positive predictive value and used the PPV and Venice criteria to define usable thresholds of statistical significance and shown that the p value threshold of 1 × 10−5 gives a very strong evidence of association in almost all studies.
Journal of Theoretical BiologyLetter to the editors regarding some papers of Dr. Satoshi Kanazawa
32 Citations2006Andrew Gelman
PubMedWhat is the chance that this study is clinically significant? A proposal for Q values.
9 Citations1999Gregory W. Froehlich
Q values encourage researchers and clinicians to be explicit about what they think a worthwhile effect is and could help shift the focus of study interpretation away from arbitrary statistical conventions.