A peculiar prevalence of <i>p</i> values just below .05
Quarterly Journal of Experimental PsychologyPublished 2 August 2012
E. J. Masicampo, Daniel Lalande
Citations269
SJR quartileQ2
SJR score0.83
SNIP0.98
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
In null hypothesis significance testing (NHST), p values are judged relative to an arbitrary threshold for significance (.05). The present work examined whether that standard influences the distribution of p values reported in the psychology literature. We examined a large subset of papers from three highly regarded journals. Distributions of p were found to be similar across the different journals. Moreover, p values were much more common immediately below .05 than would be expected based on the number of p values occurring in other ranges. This prevalence of p values just below the arbitrary criterion for significance was observed in all three journals. We discuss potential sources of this pattern, including publication bias and researcher degrees of freedom.
Keywords
Decision SciencesMathematicsEconomics, Econometrics and Finance
Psychological BulletinThe file drawer problem and tolerance for null results.
8,595 Citations1979Robert Rosenthal
Quantitative procedures for computing the tolerance for filed and future null results are reported and illustrated, and the implications are discussed.
Psychological ScienceFalse-Positive Psychology
6,712 Citations2011Joseph P. Simmons, Leif D. Nelson +1 more
It is shown that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings, flexibility in data collection, analysis, and reporting dramatically increases actual false- positive rates, and a simple, low-cost, and straightforwardly effective disclosure-based solution is suggested.
Psychonomic Bulletin & ReviewA practical solution to the pervasive problems ofp values
2,547 Citations2007Eric‐Jan Wagenmakers
The BIC provides an approximation to a Bayesian hypothesis test, does not require the specification of priors, and can be easily calculated from SPSS output.
Journal of Chronic DiseasesBias in analytic research
2,052 Citations1979David L. Sackett
This chapter focuses on the bias in analytic research, which is a general trend toward fewer study subjects but more study authors was also noted.
Psychological MethodsNull hypothesis significance testing: A review of an old and continuing controversy.
1,048 Citations2000Raymond S. Nickerson
The concluding opinion is that NHST is easily misunderstood and misused but that when applied with good judgment it can be an effective aid to the interpretation of experimental data.
American Psychological Association eBooksBeyond significance testing: Reforming data analysis methods in behavioral research.
1,001 Citations2004Rex B. Kline
Journal of Personality and Social PsychologyWhy psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011).
996 Citations2011Eric‐Jan Wagenmakers, Ruud Wetzels +2 more
Journal of the American Statistical AssociationPublication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa
936 Citations1959T. D. Sterling
Harvard Educational ReviewThe Case Against Statistical Significance Testing
933 Citations1978Ronald P. Carver
The American StatisticianCalibration of<i>ρ</i>Values for Testing Precise Null Hypotheses
857 Citations2001Thomas Sellke, M. J. Bayarri +1 more
Psychological BulletinThe fallacy of the null-hypothesis significance test.
738 Citations1960William W. Rozeboom
To the experimental scientist, statistical inference is a research instrument, a processing device by which unwieldy masses of raw data may be refined into a product more suitable for assimilation into the corpus of science, and in this lies both strength and weakness.
Perspectives on Psychological ScienceReplication and <i>p</i> Intervals: <i>p</i> Values Predict the Future Only Vaguely, but Confidence Intervals Do Much Better
455 Citations2008Geoff Cumming
P is so unreliable and gives such dramatically vague information that it is a poor basis for inference that researchers should minimize the role of p by using confidence intervals and model-fitting techniques and by adopting meta-analytic thinking.
Psychological ScienceAn Alternative to Null-Hypothesis Significance Tests
451 Citations2005Peter R. Killeen
The statistic P rep estimates the probability of replicating an effect, and provides all of the information now used in evaluating research, while avoiding many of the pitfalls of traditional statistical inference.
PLoS ONEWillingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results
442 Citations2011Jelte M. Wicherts, Marjan Bakker +1 more
It is suggested that statistical results are particularly hard to verify when reanalysis is more likely to lead to contrasting conclusions, which highlights the importance of establishing mandatory data archiving policies.
Choice Reviews OnlineBecoming a behavioral science researcher: a guide to producing research that matters
240 Citations2009Rex B. Kline
The Good, the Bad, and (the Really) Ugly of Behavioral Science Research, and Career Paths for Behavioral Scientists, and Principles for Creating Effective PowerPoint Presentations, which summarize the good, the bad, and the really, really ugly of behavioral science research.
Theory & PsychologyWhy <i>P</i> Values Are Not a Useful Measure of Evidence in Statistical Significance Testing
217 Citations2008Raymond Hubbard, Roger Lindsay
Psychological ReviewHypothesis testing and theory evaluation at the boundaries: Surprising insights from Bayes's theorem.
161 Citations2003David Trafimow
By considering a wide range of possible values for the unknown variables, it is possible to calculate a range of theoretical values for p(H0\F) and to draw conclusions about both hypothesis testing and theory evaluation.
Psychological ScienceStatistical Reform in Psychology
125 Citations2007Geoff Cumming, Fiona Fidler +7 more
Investigating whether statistical practices in psychology have been changing since 1998 found any changes since 1998 may, of course, have had many causes other than those two sources of advice.
JNCI Journal of the National Cancer InstituteWhen Are "Positive" Clinical Trials in Oncology Truly Positive?
110 Citations2010Alberto Ocaña, I. F. Tannock
Educational and Psychological MeasurementPost Hoc Pair-Wise Comparisons for the Chi-Square Test of Homogeneity of Proportions
64 Citations1993Myron K. Cox, Coretta H. Key
Journal of Autism and Developmental DisordersFrom Bayes Through Marginal Utility to Effect Sizes: A Guide to Understanding the Clinical and Statistical Significance of the Results of Autism Research Findings
49 Citations2010Domenic V. Cicchetti, Kathy Koenig +4 more
The objectives of this report are to trace the theoretical roots of the concept clinical significance that derives from Bayesian thinking, Marginal Utility/Diminishing Returns in Economics, and the “just noticeable difference”, in Psychophysics, and to demonstrate the utility of measures of ES and related concepts for enhancing the meaning of Autism research findings.
Psychonomic Bulletin & ReviewInterpretation of significance levels by psychological researchers: The .05 cliff effect may be overstated
42 Citations2001Jacques Poitevineau, Bruno Lecoutre
Investigation of individual curves reveals that the attitude of researchers towardp values is far from being as homogeneous as might be expected, and most psychological researchers in this study rated graduated confidence judgments, as either exponential or linear.
Review of Educational ResearchHypothesis Testing in Relation to Statistical Methodology
28 Citations1963Cherry Ann Clark
Psychonomic Bulletin & Reviewp rep misestimates the probability of replication
27 Citations2009Geoffrey Iverson, Michael Lee +1 more
A basic test of prep reveals that it misestimates the true probability of replication, especially for small effects, and concludes that prep is not a useful statistic for psychological science.
Psychological MethodsHow often is prep close to the true replication probability?
6 Citations2010David Trafimow, Justin A. MacDonald +2 more
The goal was to determine the relative frequency with which obtained prep statistics are close to true replication probabilities, and it is shown that it is unlikely to be close to the true value unless both the population effect magnitude and the sample size are uncommonly large.