Inside the Turk
Current Directions in Psychological SciencePublished 1 June 2014
Gabriele Paolacci, Jesse Chandler
Citations2,173
SJR quartileQ1
SJR score3.21
SNIP3.16
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
Mechanical Turk (MTurk), an online labor market created by Amazon, has recently become popular among social scientists as a source of survey and experimental data. The workers who populate this market have been assessed on dimensions that are universally relevant to understanding whether, why, and when they should be recruited as research participants. We discuss the characteristics of MTurk as a participant pool for psychology and other social sciences, highlighting the traits of the MTurk samples, why people become MTurk workers and research participants, and how data quality on MTurk compares to that from other pools and depends on controllable and uncontrollable factors.
Keywords
Computer Science
Perspectives on Psychological ScienceAmazon's Mechanical Turk
10,062 Citations2011Michael D. Buhrmester, Tracy Kwang +1 more
Findings indicate that MTurk can be used to obtain high-quality data inexpensively and rapidly and the data obtained are at least as reliable as those obtained via traditional methods.
Political AnalysisEvaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
4,080 Citations2012Adam J. Berinsky, Gregory A. Huber +1 more
It is shown that respondents recruited in this manner are often more representative of the U.S. population than in-person convenience samples but less representative than subjects in Internet-based panels or national probability samples.
Judgment and Decision MakingRunning experiments on Amazon Mechanical Turk
3,803 Citations2010Gabriele Paolacci, Jesse Chandler +1 more
New demographic data about the Mechanical Turk subject population is presented, the strengths of Mechanical Turk relative to other online and offline methods of recruiting subjects are reviewed, and the magnitude of effects obtained using Mechanical Turk and traditional subject pools are compared.
Journal of Behavioral Decision MakingData Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples
2,291 Citations2012Joseph K. Goodman, Cynthia Cryder +1 more
MTurk offers a highly valuable opportunity for data collection, and it is recommended that researchers using MTurk include screening questions that gauge attention and language comprehension, avoid questions with factual answers, and consider how individual differences in financial and social domains may influence results.
Crowdsourcing user studies with Mechanical Turk
1,970 Citations2008Aniket Kittur, Ed H. +1 more
Although micro-task markets have great potential for rapidly collecting user measurements at low costs, it is found that special care is needed in formulating tasks in order to harness the capabilities of the approach.
PLoS ONEEvaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research
1,672 Citations2013Matthew J. C. Crump, John V. McDonnell +1 more
This paper replicates a diverse body of tasks from experimental psychology including the Stroop, Switching, Flanker, Simon, Posner Cuing, attentional blink, subliminal priming, and category learning tasks using participants recruited using AMT.
Nature Cell BiologyNature Communications
1,656 Citations2010
Peer review, at its best, should aim to provide authors and editors with rigorous and constructive feedback resulting in an improved study, and there is clearly room for improvement, the current system is not broken.
Behavior Research MethodsReputation as a sufficient condition for data quality on Amazon Mechanical Turk
1,625 Citations2013Eyal Peer, Joachim Vosgerau +1 more
It is concluded that sampling high-reputation workers can ensure high-quality data without having to resort to using attention check questions (ACQs), which may lead to selection bias if participants who fail ACQs are excluded post-hoc.
The Online Laboratory: Conducting Experiments in a Real Labor Market∗
1,533 Citations2016John J. Horton
NatureSpontaneous giving and calculated greed
1,378 Citations2012David G. Rand, Joshua D. Greene +1 more
A routine echocardiogram of a patient with rheumatic aortic and mitral valve disease noted spontaneous contrast in the right side of the heart which appeared to reach the heart via the hepatic vein.
Social PsychologyInvestigating Variation in Replicability
1,253 Citations2014Richard Klein, Kate A. Ratliff +49 more
Clinical Psychological ScienceUsing Mechanical Turk to Study Clinical Populations
1,169 Citations2013Danielle N. Shapiro, Jesse Chandler +1 more
Examination of the prevalence of several psychiatric disorders and related problems, as well as the reliability and validity of participant reports on these domains, among users of Amazon's Mechanical Turk suggest that crowdsourcing software offers several advantages for clinical research while providing insight into potential problems, such as misrepresentation, that researchers should address when collecting data online.
Behavior Research MethodsThe viability of crowdsourcing for survey research
1,032 Citations2011Tara S. Behrend, David Sharek +2 more
It is concluded that the use of these labor portals is an efficient and appropriate alternative to a university participant pool, despite small differences in personality and socially desirable responding across the samples.
Who are the crowdworkers?
1,012 Citations2010Joel Ross, Lilly Irani +3 more
How the worker population has changed over time is described, shifting from a primarily moderate-income, U.S. based workforce towards an increasingly international group with a significant population of young, well-educated Indian workers.
American Journal of Political ScienceSeparating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self‐Administered Surveys
972 Citations2013Adam J. Berinsky, Michele F. Margolis +1 more
Behavior Research MethodsNonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers
958 Citations2013Jesse Chandler, Pam Mueller +1 more
It is shown that crowdsourced workers are likely to participate across multiple related experiments and that researchers are overzealous in the exclusion of research participants, which can be avoided using advanced interface features that also allow prescreening and longitudinal data collection.
Crowdsourcing graphical perception
815 Citations2010Jeffrey Heer, M. G. Bostock
The viability of Amazon's Mechanical Turk as a platform for graphical perception experiments is assessed and cost and performance data are reported and recommendations for the design of crowdsourced studies are distill.
Computational LinguisticsAmazon Mechanical Turk: Gold Mine or Coal Mine?
761 Citations2011Karën Fort, Gilles Adda +1 more
To define precisely what MTurk is and what it is not, it is hoped that this will point out opportunities for the community to deliberately value ethics above cost savings.
Journal of Theoretical BiologyThe promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments
704 Citations2011David G. Rand
This paper introduces online labor markets as a tool for behavioral experimentation, and reviews numerous replication studies indicating that AMT data is reliable, and presents two new experiments on the reliability of self-reported demographics.
Financial incentives and the "performance of crowds"
527 Citations2009Winter Mason, Duncan J. Watts
Behavior Research MethodsA validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory
493 Citations2010Jon Sprouse
A quantitative comparison of two identical acceptability judgment experiments, each with 176 participants, suggests that aside from slightly higher participant rejection rates, AMT data are almost indistinguishable from laboratory data.
Journal of Economic Behavior & OrganizationBreaking monotony with meaning: Motivation in crowdsourcing markets
400 Citations2013Dana R. Chandler, Adam Kapelner
It is found that when a task was framed more meaningfully, workers were more likely to participate and the meaningful treatment increased the quantity of output while the shredded treatment decreased the quality of output.
Proceedings of the National Academy of SciencesCollaborative learning in networks
363 Citations2011Winter Mason, Duncan J. Watts
This paper reports on a series of 256 Web-based experiments in which groups of 16 individuals collectively solved a complex problem and shared information through different communication networks, finding that collective exploration improved average success over independent exploration because good solutions could diffuse through the network.
PLoS ONEEconomic Games on the Internet: The Effect of $1 Stakes
350 Citations2012Ofra Amir, David G. Rand +1 more
These results demonstrate that economic game experiments run on MTurk are comparable to those run in laboratory settings, even when using very low stakes.
National Conference on Artificial IntelligenceAn Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets.
267 Citations2011Jakob Rogstadius, Vassilis Kostakos +4 more
Results suggest that intrinsic motivation can indeed improve the quality of workers’ output, confirming the hypothesis and finding a synergistic interaction between intrinsic and extrinsic motivators that runs contrary to previous literature suggesting “crowding out” effects.
Using the Amazon Mechanical Turk for transcription of spoken language
233 Citations2010Matthew Marge, Satanjeev Banerjee +1 more
It was found that transcriptions from MTurk workers were generally quite accurate, and when transcripts for the same utterance produced by multiple workers were combined using the ROVER voting scheme, the accuracy of the combined transcript rivaled that observed for conventional transcription methods.
Judgment and Decision MakingEmotional reactions to losing explain gender differences in entering a risky lottery
174 Citations2010Kimmo Eriksson, Brent Simpson
Task search in a human computation market
172 Citations2010Lydia B. Chilton, John J. Horton +2 more
The main findings are that on a large scale, workers sort by which tasks are most recently posted and which have the largest number of tasks available and that at least some employers try to manipulate the position of their task in the search results to exploit the tendency to search for recently posted tasks.
Computers in Human BehaviorAssessing the reliability of the M5-120 on Amazon’s mechanical Turk
171 Citations2013Christopher J. Holden, Trevor W. Dennie +1 more
Amazon’s online service, Mechanical Turk (MTurk) has become a popular option for data collection among social scientists and a 120-item measure of personality was chosen, indicating that they did not significantly change between administration dates.
Handbook of Human Computation
166 Citations2013Pietro Michelucci
This volume addresses the emerging area of human computation and explores existing and future opportunities to combine the respective strengths of both humans and machines in order to create powerful problem-solving capabilities.
Crowdsourcing performance evaluations of user interfaces
157 Citations2013Steven Komarov, Katharina Reinecke +1 more
MTurk may be a productive setting for conducting performance evaluations of user interfaces providing a complementary approach to existing methodologies, and three previously well-studied user interface designs are evaluated.
The face of quality in crowdsourcing relevance labels
113 Citations2012Gabriella Kazai, Jaap Kamps +1 more
This paper asks how do the relatively well or poorly-performing crowds, working under specific task conditions, actually look like in terms of worker characteristics, such as demographics or personality traits?
Behavior Research MethodsCollecting response times using Amazon Mechanical Turk and Adobe Flash
103 Citations2013Travis Simcox, Julie A. Fiez
It is concluded that a Flash program in conjunction with AMT can be successfully used for running many experimental paradigms that rely on response times, although experimenters must understand the limitations of the method.
Do Mechanical Turks dream of square pie charts?
75 Citations2010Robert Kosara, Caroline Ziemkiewicz
Amazon's Mechanical Turk is a web service that facilitates the assignment of small, web-based tasks to a large pool of anonymous workers to conduct perception and cognition studies, one of which was identical to a previous study performed in the lab.
Risks and Rewards of Crowdsourcing Marketplaces
68 Citations2013Jesse Chandler, Gabriele Paolacci +1 more
The present chapter investigates the role of microtask labor marketplaces in managing human and hybrid human machine computing and provides specific guidance about how to ameliorate these issues through task design, workforce selection, data cleaning and aggregation.
Honesty in an online labor market
65 Citations2011Siddharth Suri, Daniel G. Goldstein +1 more
An established method for detecting dishonest behavior is used in a series of experiments conducted on Amazon's Mechanical Turk, a popular online labor market, to conclude with a demographic and crosscultural analysis of the predictors of dishonest reporting in this market.
Assessing Crowdsourcing Quality through Objective Tasks
47 Citations2012Ahmet Aker, Mahmoud El‐Haj +2 more
One of the interesting findings is that the results do not confirm previous studies which concluded that an increase in payment attracts more noise, and the country of origin only has an impact in some of the categories and only in general text questions but there is no significant difference at the top pay.