The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
Judgment and Decision MakingPublished 1 September 2015Open access
Neil Stewart, Christoph Ungemach, Adam J. L. Harris, Daniel M. Bartels, Ben R. Newell, Gabriele Paolacci
Citations240
SJR quartileQ1
SJR score1.03
SNIP1.05
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
Abstract Using capture-recapture analysis we estimate the effective size of the active Amazon Mechanical Turk (MTurk) population that a typical laboratory can access to be about 7,300 workers. We also estimate that the time taken for half of the workers to leave the MTurk pool and be replaced is about 7 months. Each laboratory has its own population pool which overlaps, often extensively, with the hundreds of other laboratories using MTurk. Our estimate is based on a sample of 114,460 completed sessions from 33,408 unique participants and 689 sessions across seven laboratories in the US, Europe, and Australia from January 2012 to March 2015.
Keywords
Computer ScienceMedicineSocial Sciences
Perspectives on Psychological ScienceAmazon's Mechanical Turk
10,062 Citations2011Michael D. Buhrmester, Tracy Kwang +1 more
Findings indicate that MTurk can be used to obtain high-quality data inexpensively and rapidly and the data obtained are at least as reliable as those obtained via traditional methods.
Political AnalysisEvaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
4,080 Citations2012Adam J. Berinsky, Gregory A. Huber +1 more
It is shown that respondents recruited in this manner are often more representative of the U.S. population than in-person convenience samples but less representative than subjects in Internet-based panels or national probability samples.
American PsychologistOn the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications.
4,032 Citations1962Martin T. Orne
Judgment and Decision MakingRunning experiments on Amazon Mechanical Turk
3,803 Citations2010Gabriele Paolacci, Jesse Chandler +1 more
New demographic data about the Mechanical Turk subject population is presented, the strengths of Mechanical Turk relative to other online and offline methods of recruiting subjects are reviewed, and the magnitude of effects obtained using Mechanical Turk and traditional subject pools are compared.
BiometricsThe Estimation of Animal Abundance and Related Parameters.
3,313 Citations1984R. M. Cormack, George A. F. Seber
Journal of Behavioral Decision MakingData Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples
2,291 Citations2012Joseph K. Goodman, Cynthia Cryder +1 more
MTurk offers a highly valuable opportunity for data collection, and it is recommended that researchers using MTurk include screening questions that gauge attention and language comprehension, avoid questions with factual answers, and consider how individual differences in financial and social domains may influence results.
ScienceExperimental Study of Inequality and Unpredictability in an Artificial Cultural Market
2,189 Citations2006Matthew Salganik, Peter Sheridan Dodds +1 more
An artificial “music market” was created in which 14,341 participants downloaded previously unknown songs either with or without knowledge of previous participants' choices, and increasing the strength of social influence increased both inequality and unpredictability of success.
Current Directions in Psychological ScienceInside the Turk
2,173 Citations2014Gabriele Paolacci, Jesse Chandler
Behavior Research MethodsAttentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants
1,764 Citations2015David Hauser, Norbert Schwarz
In three online studies, participants from MTurk and collegiate populations participated in a task that included a measure of attentiveness to instructions (an instructional manipulation check: IMC), and MTurkers were more attentive to the instructions than were college students, even on novel IMCs.
The Online Laboratory: Conducting Experiments in a Real Labor Market∗
1,533 Citations2016John J. Horton
Social PsychologyInvestigating Variation in Replicability
1,253 Citations2014Richard Klein, Kate A. Ratliff +49 more
Behavior Research MethodsThe viability of crowdsourcing for survey research
1,032 Citations2011Tara S. Behrend, David Sharek +2 more
It is concluded that the use of these labor portals is an efficient and appropriate alternative to a university participant pool, despite small differences in personality and socially desirable responding across the samples.
Behavior Research MethodsNonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers
958 Citations2013Jesse Chandler, Pam Mueller +1 more
It is shown that crowdsourced workers are likely to participate across multiple related experiments and that researchers are overzealous in the exclusion of research participants, which can be avoided using advanced interface features that also allow prescreening and longitudinal data collection.
Computational LinguisticsAmazon Mechanical Turk: Gold Mine or Coal Mine?
761 Citations2011Karën Fort, Gilles Adda +1 more
To define precisely what MTurk is and what it is not, it is hoped that this will point out opportunities for the community to deliberately value ethics above cost savings.
Nature CommunicationsSocial heuristics shape intuitive cooperation
747 Citations2014David G. Rand, Alexander Peysakhovich +5 more
A theory of why (and for whom) intuition favors cooperation is presented: cooperation is typically advantageous in everyday life, leading to the formation of generalized cooperative intuitions, which tend to be more cooperative than deliberative responses in one-shot anonymous interactions.
Behavioral and Brain SciencesExperimental practices in economics: A methodological challenge for psychologists?
728 Citations2001Ralph Hertwig, Andreas Ortmann
There is a call for more research on the consequences of methodological preferences, such as the use on monetary payments, and a “do-it-both-ways” rule regarding the enactment of scripts, repetition of trials, and performance-based monetary payments.
PsycEXTRA DatasetExperimental practices in economics: A challenge for psychologists?
680 Citations2000Ralph Hertwig, Andreas Ortmann
Modeling Demographic Processes In Marked Populations
421 Citations2008David Thomson, Evan G. Cooch +1 more
The paper discusses Bayesian Hierarchical Models for Inference About Population Growth, Density-Dependence and Decomposing, and the Robust Design for DNA-Based Capture-Recapture Data.
BiometricsLog-Linear Models for Capture-Recapture
318 Citations1989R. M. Cormack
Log-linear models are developed for capture-recapture experiments, and their advantages and disadvantages discussed, and ways in which they can be extended to open populations, subpopulations, trap dependence, and long chains of recapture periods are presented.
Journal of Statistical SoftwareThe<b>Rcapture</b>Package: Loglinear Models for Capture-Recapture in<i>R</i>
226 Citations2007Sophie Baillargeon, Louis‐Paul Rivest
Rcapture, an R package for capture-recapture experiments, provides several new options for modeling capture probabilities heterogeneity between animals in both closed population models and the primary periods of a robust design models.
Psychological ScienceUsing Nonnaive Participants Can Reduce Effect Sizes
189 Citations2015Jesse Chandler, Gabriele Paolacci +3 more
This study investigated how prior exposure to a task affects subsequent experimental results, and found effect sizes were markedly lower in the second wave than in the first.
Professional respondents in nonprobability online panels
133 Citations2014D. Sunshine Hillygus, N Jackson +1 more
The psychology of the social
130 Citations1998Uwe Flick
BMJEstimated numbers of homeless and homeless mentally ill people in north east Westminster by using capture-recapture analysis
107 Citations1994Nigel Fisher, Stuart Turner +2 more
BMJAssessing the human condition: capture-recapture techniques
100 Citations1994R. E Laporte
The use of capture-recapture techniques could bring about a paradigm shift in how counting is done in all the disciplines that assess human populations, including epidemiology, sociology, political sciences, criminology, and market research.
BiometricsLoglinear Models for the Robust Design in Mark–Recapture Experiments
35 Citations2004Louis‐Paul Rivest, Gaétan Daigle
The proposed loglinear approach to the robust design allows incorporating parameters for a heterogeneity in the capture probabilities of the units within each primary session, and Temporary emigration out of the study area can be accounted for in the loglinear framework.