Questions About Questionable Research Practices in the Field of Management
Journal of ManagementPublished 7 December 2015Open access
George C. Banks, Ernest H. O’Boyle, Jeffrey M. Pollack, Charles D. White, John H. Batchelor, Christopher E. Whelpley
Citations171
SJR quartileQ1
SJR score8.64
SNIP5.75
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
The discussion regarding questionable research practices (QRPs) in management as well as the broader natural and social sciences has increased substantially in recent years. Despite the attention, questions remain regarding research norms and the implications for both theoretical and practical advancements. The aim of the current article is to address these issues in a question-and-answer format while drawing upon both past research and the results of a series of new studies conducted using a mixed-methods design. Our goal is to encourage a systematic, collegial, and constructive dialogue regarding QRPs in management research.
Keywords
Social SciencesDecision SciencesBusiness, Management and Accounting
ScienceEstimating the reproducibility of psychological science
8,673 Citations2015Alexander A. Aarts
A large-scale assessment suggests that experimental reproducibility in psychology leaves a lot to be desired, and correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.
Psychological ScienceFalse-Positive Psychology
6,712 Citations2011Joseph P. Simmons, Leif D. Nelson +1 more
It is shown that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings, flexibility in data collection, analysis, and reporting dramatically increases actual false- positive rates, and a simple, low-cost, and straightforwardly effective disclosure-based solution is suggested.
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
5,538 Citations2015Frank L. Schmidt, John E. Hunter
SciencePromoting an open research culture
2,761 Citations2015Brian A. Nosek, George Alter +37 more
Author guidelines for journals could help to promote transparency, openness, and reproducibility in science.
Psychological ScienceMeasuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
2,333 Citations2012Leslie K. John, George Loewenstein +1 more
It is found that the percentage of respondents who have engaged in questionable practices was surprisingly high, which suggests that some questionable practices may constitute the prevailing research norm.
Personality and Social Psychology ReviewHARKing: Hypothesizing After the Results are Known
2,058 Citations1998Norbert L. Kerr
It is conceded that the question of whether HARKing's costs exceed its benefits is a complex one that ought to be addressed through research, open discussion, and debate.
PLoS ONEHow Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data
1,944 Citations2009Daniele Fanelli
Meta-regression showed that self reports surveys, surveys using the words “falsification” or “fabrication”, and mailed surveys yielded lower percentages of misconduct, and when these factors were controlled for, misconduct was reported more frequently by medical/pharmacological researchers than others.
SciencePublication bias in the social sciences: Unlocking the file drawer
1,592 Citations2014Annie Franco, Neil Malhotra +1 more
Fully half of peer-reviewed and implemented social science experiments are not published, providing direct evidence of publication bias and identifying the stage of research production at which publication bias occurs: Authors do not write up and submit null findings.
Perspectives on Psychological ScienceScientific Utopia
1,336 Citations2012Brian A. Nosek, Jeffrey R. Spies +1 more
Strategies for improving scientific practices and knowledge accumulation are developed that account for ordinary human motivations and biases and can reduce the persistence of false findings.
Organizational Research MethodsBest-Practice Recommendations for Defining, Identifying, and Handling Outliers
1,278 Citations2013Herman Aguinis, Ryan K. Gottfredson +1 more
Although the emphasis is on regression, structural equation modeling, and multilevel modeling, the general framework forms the basis for a research agenda regarding outliers in the context of other data-analytic approaches.
Perspectives on Psychological ScienceAn Agenda for Purely Confirmatory Research
1,136 Citations2012Eric‐Jan Wagenmakers, Ruud Wetzels +3 more
This article proposes that researchers preregister their studies and indicate in advance the analyses they intend to conduct, and proposes that only these analyses deserve the label “confirmatory,” and only for these analyses are the common statistical tests valid.
Journal of Personality and Social PsychologyWhy psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011).
996 Citations2011Eric‐Jan Wagenmakers, Ruud Wetzels +2 more
Academy of Management JournalThe Field of Management's Devotion to Theory: Too Much of a Good Thing?
865 Citations2007Donald C. Hambrick
Perspectives on Psychological ScienceThe Rules of the Game Called Psychological Science
853 Citations2012Marjan Bakker, Annette van Dijk +1 more
This paper considers 13 meta-analyses covering 281 primary studies in various fields of psychology and finds indications of biases and/or an excess of significant results in seven, highlighting the need for sufficiently powerful replications and changes in journal policies.
American PsychologistThe poor availability of psychological research data for reanalysis.
559 Citations2006Jelte M. Wicherts, Denny Borsboom +2 more
A failed attempt to obtain data reported in 141 empirical articles recently published by the APA, to assess the robustness of the research findings to outliers.
Behavior Research MethodsThe prevalence of statistical reporting errors in psychology (1985–2013)
541 Citations2015Michèle B. Nuijten, Chris Hartgerink +3 more
Infection and ImmunityRetracted Science and the Retraction Index
405 Citations2011Ferric C. Fang, Arturo Casadevall
Using a novel measure that is called the “retraction index,” it is found that the frequency of retraction varies among journals and shows a strong correlation with the journal impact factor.
Organizational Research MethodsPublication Bias in the Organizational Sciences
398 Citations2012Sven Kepes, George C. Banks +2 more
Advances in meta-analytic techniques from the medical and related sciences are introduced for a comprehensive assessment and evaluation of publication bias and their use on a data set on employment interview validities is illustrated.
Science and Engineering EthicsThe Perverse Effects of Competition on Scientists’ Work and Relationships
395 Citations2007Melissa S. Anderson, Emily A. Ronning +2 more
Focus-group discussions with 51 mid- and early-career scientists reveal a dark side of competition in science that contributes to strategic game-playing in science, a decline in free and open sharing of information and methods, and sabotage of others’ ability to use one’s work.
Psychological InquiryScientific Utopia: I. Opening Scientific Communication
391 Citations2012Brian A. Nosek, Yoav Bar‐Anan
This work outlines a path that moves away from the existing model of scientific communication to improve the efficiency in meeting the purpose of public science—knowledge accumulation and calls for six changes.
Behavior Research MethodsThe (mis)reporting of statistical results in psychology journals
376 Citations2011Marjan Bakker, Jelte M. Wicherts
The authors' results indicate that around 18% of statistical results in the psychological literature are incorrectly reported, and that errors were often in line with researchers’ expectations.
Journal of ManagementBayesian Estimation and Inference
373 Citations2013Michael J. Zyphur, Frederick L. Oswald
The “Bayesian revolution” that is sweeping across multiple disciplines but has yet to gain a foothold in organizational research is introduced and the many benefits and few hindrances of Bayesian methods are discussed.
PLoS ONEPublic Availability of Published Research Data in High-Impact Journals
369 Citations2011Alawi Alsheikh‐Ali, Waqas Qureshi +2 more
A substantial proportion of original research papers published in high-impact journals are either not subject to any data availability policies, or do not adhere to the data availability instructions in their respective journals.
Journal of ManagementAn Introduction to Bayesian Hypothesis Testing for Management Research
364 Citations2014Sandra Andraszewicz, Benjamin Scheibehenne +4 more
More Statistical and Methodological Myths and Urban Legends
355 Citations2014
The British Journal for the Philosophy of SciencePrediction Versus Accommodation and the Risk of Overfitting
301 Citations2004Christopher Hitchcock, Elliott Sober
A new approach to the vexed problem of understanding the epistemic difference between prediction and accommodation is presented, floating the hypothesis that accommodation is a defective methodology only when the methods used to accommodate the data fail to guard against the risk of overfitting.
Journal of Empirical Research on Human Research EthicsNormal Misbehavior: Scientists Talk about the Ethics of Research
260 Citations2006Raymond De Vries, Melissa S. Anderson +1 more
Focus on the “normal misbehaviors” that are part of the ordinary life of researchers allows us to see the way the organization of science generates both compliance and deviance from ethical norms.
Journal of ManagementThe Chrysalis Effect
250 Citations2014Ernest H. O’Boyle, George C. Banks +1 more
Archives of Internal MedicineTesting for the Presence of Positive-Outcome Bias in Peer Review
244 Citations2010Gwendolyn B. Emerson, Winston J. Warme +4 more
Positive-outcome bias was present during peer review and a fabricated manuscript with a positive outcome was more likely to be recommended for publication than was an otherwise identical no-difference manuscript.
Journal of Empirical Research on Human Research EthicsNormative Dissonance in Science: Results from a National Survey of U.S. Scientists
219 Citations2007Melissa S. Anderson, Brian C. Martinson +1 more
This analysis of the extent of dissonance between espoused ideals and scientists' perceptions of their own and others' behavior is based on survey responses from 3,247 mid- and early-career scientists who had research funding from the National Institutes of Health.
NatureStatistics: P values are just the tip of the iceberg
217 Citations2015Jeffrey T. Leek, Roger D. Peng
Journal of Personality and Social PsychologyBest research practices in psychology: Illustrating epistemological and pragmatic considerations with the case of relationship science.
213 Citations2015Eli J. Finkel, Paul W. Eastwick +1 more
This article contrasts the false-positives-reduction (FPR) approach with an alternative, the error balance (EB) approach, which argues that any serious consideration of optimal scientific practice must contend simultaneously with both false-positive and false-negative errors.
Journal of Empirical Research on Human Research EthicsScientists' Perceptions of Organizational Justice and Self-Reported Misbehaviors
184 Citations2006Brian C. Martinson, Melissa S. Anderson +2 more
These findings indicate that when scientists believe they are being treated unfairly they are more likely to behave in ways that compromise the integrity of science.
Academy of Management Learning and EducationManagement Science on the Credibility Bubble: Cardinal Sins and Various Misdemeanors.
179 Citations2010Arthur G. Bedeian, Shannon G. Taylor +1 more
Journal of ManagementPublication Bias in Strategic Management Research
147 Citations2014Jeffrey S. Harrison, George C. Banks +3 more
Personnel PsychologyHARKing's Threat to Organizational Research: Evidence From Primary and Meta‐Analytic Sources
133 Citations2015Frank A. Bosco, Herman Aguinis +3 more
Educational ResearcherOutcome-Reporting Bias in Education Research
130 Citations2013Terri Pigott, Jeffrey C. Valentine +3 more
Industrial and Organizational PsychologyHow Trustworthy Is the Scientific Literature in Industrial and Organizational Psychology?
126 Citations2013Sven Kepes, Michael A. McDaniel
Infection and ImmunityCompetitive Science: Is Competition Ruining Science?
126 Citations2015Ferric C. Fang, Arturo Casadevall
The benefits of competition in providing incentives to scientists and the adverse effects of competition on resource sharing, research integrity, and creativity are considered.
PLoS ONEOutlier Removal and the Relation with Reporting Errors and Quality of Psychological Research
103 Citations2014Marjan Bakker, Jelte M. Wicherts
It is failed to find that the removal of outliers from the analysis in psychological articles was related to weaker evidence, sample size, or the prevalence of errors, but a discrepancy between the reported degrees of freedom of t tests and the reported sample size in 41% of articles that did not report removal of any data values suggests common failure to report data exclusions in psychology articles.
International Journal of Selection and AssessmentPublication Bias: A call for improved meta‐analytic practice in the organizational sciences
90 Citations2012George C. Banks, Sven Kepes +1 more
PLoS ONEExcess Success for Psychology Articles in the Journal Science
71 Citations2014Gregory Francis, Jay Tanzman +1 more
PsychometrikaEncourage Playing with Data and Discourage Questionable Reporting Practices
63 Citations2015Daniël Wigboldus, Ron Dotsch
Making research data and research materials publicly available and consulting methodologists or statisticians for help and a second opinion will help in increasing the transparency of the data analysis phase of the empirical research cycle within psychology.
Management and Organization ReviewPresenting Post Hoc Hypotheses as A Priori: Ethical and Theoretical Issues
61 Citations2011Kwok Leung
Industrial and Organizational PsychologyForgetting What We Learned as Graduate Students: HARKing and Selective Outcome Reporting in I–O Journal Articles
41 Citations2013Joseph J. Mazzola, Jacqueline K. Deuling
Industrial and Organizational PsychologyThe Kryptonite of Evidence-Based I–O Psychology
35 Citations2011George C. Banks, Michael A. McDaniel
Recommendations for preventing and evaluating publication bias to promote improved systematic reviews and evidence-based practice are explained.
PsychometrikaImproving the Conduct and Reporting of Statistical Analysis in Psychology
21 Citations2015Klaas Sijtsma, Coosje Lisabet Sterre Veldkamp +1 more
This work discusses the fear of an increased dichotomy between substantive and statistical aspects of research that may arise when the latter aspects are laid entirely in the hands of a statistician, remedies for false positives and replication failure, and the status of data exploration.
Industrial and Organizational PsychologyWhy We Need Industrial–Organizational Psychology to Fix Industrial–Organizational Psychology
11 Citations2013George C. Banks, Ernest H. O’Boyle
Public Opinion QuarterlyProceedings of the Thirteenth Conference on Public Opinion Research
6 Citations1958John W. Riley
Industrial and Organizational PsychologyThe Fantastic Four Years: Recommendations for Industrial–Organizational Programs
2 Citations2014Christopher W. Wiese, Julia M. Fullick