Crowdsourcing for search evaluation
ACM SIGIR ForumPublished 3 January 2011
Vitor R. Carvalho, Matthew Lease, Emine Yılmaz
Citations47
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The Crowdsourcing for Search Evaluation Workshop (CSE 2010) was held on July 23, 2010 in Geneva, Switzerland, in conjunction with the 33rd Annual ACM SIGIR Conference.
Abstract
The Crowdsourcing for Search Evaluation Workshop (CSE 2010) was held on July 23, 2010 in Geneva, Switzerland, in conjunction with the 33rd Annual ACM SIGIR Conference. The workshop addressed the latest advances in theory and empirical methods in crowdsourcing for search evaluation, as well as novel applications of crowdsourcing for evaluating search systems. Three invited talks were presented, along with seven refereed papers. Proceedings from the workshop, along with presentation slides, have been made available online.
Keywords
Computer Science
BiometricsThe Measurement of Observer Agreement for Categorical Data
78,934 Citations1977J. Richard Landis, Gary G. Koch
A general statistical methodology for the analysis of multivariate categorical data arising from observer reliability studies is presented and tests for interobserver bias are presented in terms of first-order marginal homogeneity and measures of interob server agreement are developed as generalized kappa-type statistics.
Psychological ReviewSelf-efficacy: Toward a unifying theory of behavioral change.
35,454 Citations1977Albert Bandura
An integrative theoretical framework to explain and to predict psychological changes achieved by different modes of treatment is presented and findings are reported from microanalyses of enactive, vicarious, and emotive mode of treatment that support the hypothesized relationship between perceived self-efficacy and behavioral changes.
The Annals of Mathematical StatisticsOn a Test of Whether one of Two Random Variables is Stochastically Larger than the Other
13,914 Citations1947H. B. Mann, Douglas R. Whitney
What is Twitter, a social network or a news media?
6,692 Citations2010Haewoon Kwak, Changhyun Lee +2 more
This work is the first quantitative study on the entire Twittersphere and information diffusion on it and finds a non-power-law follower distribution, a short effective diameter, and low reciprocity, which all mark a deviation from known characteristics of human social networks.
MIS QuarterlyComputer Self-Efficacy: Development of a Measure and Initial Test
6,248 Citations1995Deborah Compeau, Christopher A. Higgins
Self-efficacy represents an important individual trait, which moderates organizational influences on an individual's decision to use computers, and is important to the successful implementation of systems in organizations.
Earthquake shakes Twitter users
3,667 Citations2010Takeshi Sakaki, Makoto Okazaki +1 more
This paper investigates the real-time interaction of events such as earthquakes in Twitter and proposes an algorithm to monitor tweets and to detect a target event and produces a probabilistic spatiotemporal model for the target event that can find the center and the trajectory of the event location.
Why we twitter
2,639 Citations2007Akshay Java, Xiaodan Song +2 more
It is found that people use microblogging to talk about their daily activities and to seek or share information and the user intentions associated at a community level are analyzed to show how users with similar intentions connect with each other.
Cognition & EmotionEmotion elicitation using films
2,182 Citations1995James J. Gross, Robert W. Levenson
Crowdsourcing user studies with Mechanical Turk
1,970 Citations2008Aniket Kittur, Ed H. +1 more
Although micro-task markets have great potential for rapidly collecting user measurements at low costs, it is found that special care is needed in formulating tasks in order to harness the capabilities of the approach.
Cheap and fast---but is it good?
1,906 Citations2008Rion Snow, Brendan O’Connor +2 more
This work explores the use of Amazon's Mechanical Turk system, a significantly cheaper and faster method for collecting annotations from a broad base of paid non-expert contributors over the Web, and proposes a technique for bias correction that significantly improves annotation quality on two tasks.
ACM Transactions on Mathematical SoftwareRandom sampling with a reservoir
1,759 Citations1985Jeffrey Scott Vitter
Theoretical and empirical results indicate that Algorithm Z outperforms current methods by a significant margin, and an efficient Pascal-like implementation is given that incorporates these modifications and that is suitable for general use.
Quality management on Amazon Mechanical Turk
988 Citations2010Panagiotis G. Ipeirotis, Foster Provost +1 more
This work presents algorithms that improve the existing state-of-the-art techniques, enabling the separation of bias and error, and illustrates how to incorporate cost-sensitive classification errors in the overall framework and how to seamlessly integrate unsupervised and supervised techniques for inferring the quality of the workers.
How and why people Twitter
873 Citations2009Dejin Zhao, Mary Beth Rosson
This exploratory research project is aimed at gaining an in-depth understanding of how and why people use Twitter and exploring micro-blog's impacts on informal communication at work.
Proceedings of the International AAAI Conference on Web and Social MediaCharacterizing Microblogs with Topic Models
724 Citations2010Daniel Ramage, Susan Dumais +1 more
A scalable implementation of a partially supervised learning model (Labeled LDA) that maps the content of the Twitter feed into dimensions that correspond roughly to substance, style, status, and social characteristics of posts is presented.
A few chirps about twitter
711 Citations2008Balachander Krishnamurthy, Phillipa Gill +1 more
A detailed characterization of Twitter, an application that allows users to send short messages, is presented, which identifies distinct classes of Twitter users and their behaviors, geographic growth patterns and current size of the network.
A dynamic bayesian network click model for web search ranking
543 Citations2009Olivier Chapelle, Ya Zhang
A Dynamic Bayesian Network is proposed which aims at providing us with unbiased estimation of the relevance from the click logs and shows that the proposed click model outperforms other existing click models in predicting both click-through rate and relevance.
Aslib ProceedingsThe CRANFIELD TESTS ON INDEX LANGUAGE DEVICES
470 Citations1967Cyril W. Cleverdon
It appeared that the most important consideration was the specificity of the index terms; within the context of the conditions existing in this test, single‐word terms were more effective than concept terms or a controlled vocabulary.
Building a question answering test collection
448 Citations2000Ellen M. Voorhees, Dawn M. Tice
The TREC-8 Question Answering (QA) Track was the first large-scale evaluation of domain-independent question answering systems and was used to investigate whether the evaluation methodology used for document retrieval is appropriate for a different natural language processing task.
Are your participants gaming the system?
427 Citations2010Julie S. Downs, Mandy B. Holbrook +2 more
Fast, cheap, and creative
426 Citations2009Chris Callison-Burch
It is found that when combined non-expert judgments have a high-level of agreement with the existing gold-standard judgments of machine translation quality, and correlate more strongly with expert judgments than Bleu does, Mechanical Turk can be used to calculate human-mediated translation edit rate (HTER), to conduct reading comprehension experiments with machine translation, and to create high quality reference translations.
Cognition & EmotionInducing and assessing differentiated emotion-feeling states in the laboratory
390 Citations1993Pierre Philippot
Overall, results indicate that film segments can elicit a diversity of predictable emotions, in the same way, in a majority of individuals.
Lecture notes in computer scienceThe Philosophy of Information Retrieval Evaluation
385 Citations2002Ellen M. Voorhees
The fundamental assumptions and appropriate uses of the Cranfield paradigm, especially as they apply in the context of the evaluation conferences, are reviewed.
Supervised learning from multiple experts
342 Citations2009Vikas C. Raykar, Shipeng Yu +6 more
A probabilistic approach for supervised learning when there are multiple experts/annotators providing (possibly noisy) labels but no absolute gold standard and the algorithm evaluates the different experts and also gives an estimate of the actual hidden labels.
ACM SIGIR ForumCrowdsourcing for relevance evaluation
333 Citations2008Omar Alonso, Daniel E. Rose +1 more
A new approach to evaluation called TERC is described, based on the crowdsourcing paradigm, in which many online users, drawn from a large community, each performs a small evaluation task.
How reliable are annotations via crowdsourcing
308 Citations2010Stefanie Nowak, Stefan Rüger
The majority vote applied to generate one annotation set out of several opinions, is able to filter noisy judgments of non-experts to some extent and the resulting annotation set is of comparable quality to the annotations of experts.
Journal of Personality and Social PsychologyWhen nonsense sounds happy or helpless: The Implicit Positive and Negative Affect Test (IPANAT).
265 Citations2009Markus Quirin, Miguel Kazén +1 more
The present findings suggest that the IPANAT is a reliable and valid measure with a straightforward application procedure, and shows adequate internal consistency, test-retest reliability, stability, and construct validity.
Minimal test collections for retrieval evaluation
242 Citations2006Ben Carterette, James Allan +1 more
This work links evaluation with test collection construction to gain an understanding of the minimal judging effort that must be done to have high confidence in the outcome of an evaluation.
A simple and efficient sampling method for estimating AP and NDCG
232 Citations2008Emine Yılmaz, Evangelos Kanoulas +1 more
Beyond DCG
211 Citations2010Ahmed H. Yousef, Rosie Jones +1 more
This work shows empirically that user behavior alone can give an accurate picture of the success of the user's web search goals, without considering the relevance of the documents displayed.
Sources of evidence for vertical selection
201 Citations2009Jaime Arguello, Fernando Díaz +2 more
This work addresses the problem of vertical selection, predicting relevant verticals for queries issued to the search engine's main web search page by focusing on 18 different verticals, which differ in terms of semantics, media type, size, and level of query traffic.
Time is of the essence
200 Citations2010Anlei Dong, Ruiqiang Zhang +6 more
A method to use the micro-blogging data stream to detect fresh URLs and to compute novel and effective features for ranking fresh URLs is proposed and demonstrated to improve effective of the portal web search engine for realtime web search.
A statistical method for system evaluation using incomplete judgments
173 Citations2006Javed A. Aslam, Virgil Pavlu +1 more
This work considers the problem of large-scale retrieval evaluation, and proposes a statistical method for evaluating retrieval systems using incomplete judgments based on random sampling, which produces unbiased estimates of the standard measures themselves.
Predicting searcher frustration
155 Citations2010Henry Feild, James Allan +1 more
By modeling searcher frustration, search engines can predict the current state of user frustration and decide when to intervene with alternative search strategies to prevent the user from becoming more frustrated, giving up, or switching to another search engine.
Integration of news content into web results
130 Citations2009Fernando Díaz
This paper addresses the issue of integrating search results from a news vertical into web search results, and defines several click-based metrics which allow a system to be monitored and tuned without annotator effort.
Studies in computational intelligenceThe Music Information Retrieval Evaluation eXchange: Some Observations and Insights
110 Citations2010J. Stephen Downie, Andreas F. Ehmann +2 more
This chapter outlines some of the major highlights of the past four years of MIREX evaluations, including its organizing principles, the selection of evaluation metrics, and the evolution of evaluation tasks.
Studies in computational intelligenceAdvances in Music Information Retrieval
104 Citations2010Zbigniew W. Raś, Alicja Wieczorkowska
This book is divided into four sections: MIR Methods and Platforms, Harmony, Music Similarity, and Content Based Identification and Retrieval, to familiarize readers with vocabulary referring to music information retrieval.
HAL (Le Centre pour la Communication Scientifique Directe)The 2005 Music Information Retrieval Evaluation Exchange (Mirex 2005): Preliminary Overview.
98 Citations2005J. Stephen Downie, Kris West +2 more
This paper is an extended abstract which provides a brief preliminary overview of the 2005 Music Information Retrieval Evaluation eXchange (MIREX 2005) and lists of targets items to be undertaken before MIREX 2006 to ensure the ongoing success of the MIreX framework.
Archive ouverte UNIGE (University of Geneva)Crowdsourcing for Affective Annotation of Video: Development of a Viewer-reported Boredom Corpus
83 Citations2010Mohammad Soleymani, Martha Larson
A new corpus to be used to evaluate algorithms for prediction of viewer-reported boredom and a use of crowdsourcing to address two shortcomings of previous affective video corpora are reported on.
Computer Music JournalThe Scientific Evaluation of Music Information Retrieval Systems: Foundations and Future
82 Citations2004J. Stephen Downie
This article provides an overview of the current scientific problem facing MIR research and reports upon the findings of the Music Information Retrieval (MIR)/ Music Digital Library (MDL) Evaluation Frameworks Project.
Information Processing & ManagementA day in the life of Web searching: an exploratory study
76 Citations2003Seda Özmutlu, Amanda Spink +1 more
A comparative time-based Web study of US-based Excite and Norwegian-based Fast Web search logs explores variations in user searching related to changes in time of the day, suggesting fluctuations in Web user behavior over the day.
Communications of the ACMTREC
66 Citations2007Ellen M. Voorhees
Large-scale test collections drive improvement in search technology to help users find information in free text in order to improve the quality of search results.
Lecture notes in computer scienceThe SRI-ICSI Spring 2007 Meeting and Lecture Recognition System
55 Citations2008Andreas Stolcke, Xavier Anguera +6 more
The latest version of the SRI-ICSI meeting and lecture recognition system, as was used in the NIST RT-07 evaluations, is described, highlighting improvements made over the last year, and a new NIST metric designed to evaluate combined speech diarization and recognition is reported.
Detecting Uninteresting Content in Text Streams
54 Citations2010Omar Alonso, Chad Carson +2 more
The premise is that truly mundane content is not interesting in any context, and thus can be quickly filtered using simple queryindependent features, which could be used for tiering indexes in a micro-blog search engine, with the filtered uninteresting content relegated to the less frequently accessed tiers.
Incremental test collections
46 Citations2005Ben Carterette, James Allan
An algorithm that intelligently selects documents to be judged and decides when to stop in such a way that with very little work there can be a high degree of confidence in the result of the evaluation is presented.
Information Processing & ManagementAnalysis of large data logs: an application of Poisson sampling on excite web queries
46 Citations2002H. Cenk Özmutlu, Amanda Spink +1 more
Poisson sampling is used to develop a sampling strategy, and it is shown how sample sets selected by Poisson sampling statistically effectively represent the characteristics of the entire dataset.
Data Archiving and Networked Services (DANS)A Ground Truth For Half A Million Musical Incipits
45 Citations2005Rainer Typke, Marc den Hoed +3 more
This work filtered the RISM A/II collection so that about 50 candidates per query were left, which was then presented to 35 human experts for a final ranking, and obtained ground truths, which can be used for evaluating music information retrieval systems.
Evaluation measures for preference judgments
42 Citations2008Ben Carterette, Paul N. Bennett
A class of such measures like precision, recall, and DCG are described and compared to absolute and preference measures over a large TREC collection.
Crowdsourcing Preference Judgments for Evaluation of Music Similarity Tasks
41 Citations2010Julián Urbano, Jorge Morato +2 more
It is shown that crowdsourcing is a perfectly viable alternative to evaluate music systems without the need for experts, and produces lists very similar to the original ones, while dealing with some defects of the original methodology.
Journal of the American Society for Information ScienceThe simple scalability of documents
41 Citations1990Mark E. Rorvig
The test results indicate that, with some reservations, this theory of scaling is applicable to documents, and this finding is further applied to the construction of test collections for Information Retrieval research that could more sensitively measure retrieval system alterations through the use of documents scaled not merely by relevance, but rather, by preference.
A Measure for Evaluating Retrieval Techniques based on Partially Ordered Ground Truth Lists
37 Citations2006Rainer Typke, Remco C. Veltkamp +1 more
The "average dynamic recall" (ADR) is introduced that averages the recall among a dynamic set of relevant documents, taking into account the fact that the ground truth reliably orders groups of matches, but not always individual matches.
Data Archiving and Networked Services (DANS)Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation
34 Citations2009Shlomo Geva, Jaap Kamps +4 more
These proceedings contain the invited talks and posters of the SIGIR 2009 Workshop on the Future of IR Evaluation, Boston, Massachusetts 23 July, 2009.
International Symposium/Conference on Music Information RetrievalHuman Similarity Judgments: Implications For The Design Of Formal Evaluations.
32 Citations2007M. Cameron Jones, J. Stephen Downie +1 more
Findings of a series of analyses of human similarity judgments from the Symbolic Melodic Similarity, and Audio Music Similarity tasks from the Music Information Retrieval Evaluation Exchange (MIREX) 2006 are presented.
Crowdsourcing a News Query Classification Dataset
28 Citations2010Richard McCreadie, Craig Macdonald +1 more
Overall, it is found that crowdsourcing is suitable for building a news query classification dataset.
Machine LearningA case based approach to expressivity-aware tempo transformation
25 Citations2006Maarten Grachten, Josep Lluís Arcos +1 more
This work investigates the problem of how a performance played at a particular tempo can be rendered automatically at another tempo, while preserving naturally sounding expressivity, and presents a case-based reasoning system called TempoExpress for addressing this problem.
On the Evaluation of the Quality of Relevance Assessments Collected through Crowdsourcing
20 Citations2009Gabriella Kazai, Nataša Milić-Frayling
The components that could be used to measure the quality of the collected relevance assessments are discussed and recommendations are based on experiments with collecting relevance assessments for digitized books, conducted as part of the INEX Book Track in 2008.
Singapore Management University Institutional Knowledge (InK) (Singapore Management University)Estimating the Quality of Postings in the Real-time Web
18 Citations2010Hady W. Lauw, Alexandros Ntoulas +1 more
This paper identifies several important factors that are indicative of the quality of postings, and presents metrics that capture these factors and presents early experimental results on Twitter.
Improving The Generation Of Ground Truths Based On Partially Ordered Lists.
14 Citations2010Julián Urbano, Mónica Marrero +2 more
It is shown that it is not possible to ensure lists completely consistent, and a measure of consistency based on Average Dynamic Recall is developed and several alternatives to arrange the lists prove to be more consistent than the original method.
A generalized graph-spectral approach to melodic modeling and retrieval
13 Citations2008Alberto A. Pinto, Paolo Tagliolato
Different spectral approaches to structural queries in databases of symbolic music, which exploits mathematical music theory results to improve the descriptive power of representative graphs are compared.
Logging the Search Self-Efficacy of Amazon Mechanical Turkers
10 Citations2010Henry Feild, Rosie Jones +4 more
The initial tests of using Amazon Mechanical Turk to conduct experiments in the relationship between searcher self-ecacy assessments and their strategies for conducting complex searches indicate Turkers have a similar range of search self-ECacy.
Lexikon des gesamten Buchwesens OnlineRepertoire International des Sources Musicales (RISM)
8 Citations2020Klaus Keil
Journal of computing sciences in collegesAnd, or, not
2 Citations2012N MichaudLisa
This paper addresses the centrality of Boolean expressions as a repeating theme in an introductory computer science course by addressing the use of logical expressions in introductory programming.