A Protocol for Cross-Validating Large Crowdsourced Data
Published 3 November 2014
Yoann Baveye, Christel Chamaret, Emmanuel Dellandréa, Liming Chen
Citations6
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
An annotation protocol is designed which enables the scoring of induced affective feelings for cross-validating the annotations of the LIRIS-ACCEDE dataset and identifying any potential bias.
Abstract
International audience
Keywords
PsychologyComputer Science
Journal of Behavior Therapy and Experimental PsychiatryMeasuring emotion: The self-assessment manikin and the semantic differential
9,415 Citations1994Margaret M. Bradley, Peter J. Lang
Reports of affective experience obtained using SAM are compared to the Semantic Differential scale devised by Mehrabian and Russell (An approach to environmental psychology, 1974), which requires 18 different ratings.
Review of General PsychologyBad is Stronger than Good
7,354 Citations2001Roy F. Baumeister, Ellen Bratslavsky +2 more
Psychological ReviewCore affect and the psychological construction of emotion.
5,815 Citations2003James A. Russell
At the heart of emotion, mood, and any other emotionally charged event are states experienced as simply feeling good or bad, energized or enervated, which influence reflexes, perception, cognition, and behavior.
Review of General PsychologyWhat Good Are Positive Emotions?
5,597 Citations1998Barbara L. Fredrickson
A new model is advanced to describe the form and function of a subset of positive emotions, including joy, interest, contentment, and love, that serve to broaden an individual's momentary thought–action repertoire, which in turn has the effect of building that individual's physical, intellectual, and social resources.
IEEE Transactions on Affective ComputingDEAP: A Database for Emotion Analysis ;Using Physiological Signals
4,847 Citations2011Sander Koelstra, Christian Mühl +7 more
A multimodal data set for the analysis of human affective states was presented and a novel method for stimuli selection is proposed using retrieval by affective tags from the last.fm website, video highlight detection, and an online assessment tool.
PsychophysiologyLooking at pictures: Affective, facial, visceral, and behavioral reactions
3,197 Citations1993Peter J. Lang, Mark K. Greenwald +2 more
Responsibility specificity, particularly facial expressiveness, supported the view that specific affects have unique patterns of reactivity, and consistency of the dimensional relationships between evaluative judgments and physiological response emphasizes that emotion is fundamentally organized by these motivational parameters.
Computational IntelligenceCROWDSOURCING A WORD–EMOTION ASSOCIATION LEXICON
2,584 Citations2012Saif M. Mohammad, Peter D. Turney
It is shown how the combined strength and wisdom of the crowds can be used to generate a large, high‐quality, word–emotion and word–polarity association lexicon quickly and inexpensively.
European Review of Social PsychologyPositive-Negative Asymmetry in Evaluations: The Distinction Between Affective and Informational Negativity Effects
1,110 Citations1990Guido Peeters, Janusz Czapiński
Cognition & EmotionAssessing the effectiveness of a large database of emotion-eliciting films: A new tool for emotion researchers
837 Citations2009Alexandre Schaefer, Frédéric Nils +2 more
IEEE Transactions on MultimediaAffective video content representation and modeling
593 Citations2005Alan Hanjalić, Li-Qun Xu
A computational framework for affective video content representation and modeling is proposed based on the dimensional approach to affect that is known from the field of psychophysiology that is characterized by the dimensions of arousal (intensity of affect) and valence (type of affect).
Cognition & EmotionInducing and assessing differentiated emotion-feeling states in the laboratory
390 Citations1993Pierre Philippot
Overall, results indicate that film segments can elicit a diversity of predictable emotions, in the same way, in a majority of individuals.
Applied Psychophysiology and BiofeedbackThe Emotional Movie Database (EMDB): A Self-Report and Psychophysiological Study
221 Citations2012Sandra Carvalho, Jorge Leite +2 more
The results showed that sustained exposure to the affective film clips resulted in a pattern of a SCL increase and HR deceleration in high arousal conditions (i.e., horror and erotic conditions) and can reliably be used in research requiring the presentation of non-auditory film clips with different ratings of valence, arousal and dominance.
CROWDMOS: An approach for crowdsourcing mean opinion score studies
212 Citations2011Flávio Ribeiro, Dinei Florêncio +2 more
To automate crowdMOS testing, a set of freely distributable, open-source tools for Amazon Mechanical Turk, a platform designed to facilitate crowdsourcing, are offered, providing researchers with a user-friendly means of performing subjective quality evaluations without the overhead associated with laboratory studies.
A crowdsourceable QoE evaluation framework for multimedia content
189 Citations2009Kuan-Ta Chen, Chen‐Chi Wu +2 more
A crowdsourceable framework to quantify the QoE of multimedia content is proposed, which enables crowdsourcing because it supports systematic verification of participants' inputs; the rating procedure is simpler than that of MOS, so there is less burden on participants; and it derives interval-scale scores that enable subsequent quantitative analysis andQoE provisioning.
IEEE Transactions on Affective ComputingCrowdsourcing Facial Responses to Online Videos
124 Citations2012Daniel McDuff, Rana el Kaliouby +1 more
The framework, data collected, and analysis demonstrate an ecologically valid method for unobtrusive evaluation of facial responses to media content that is robust to challenging real-world conditions and requires no explicit recruitment or compensation of participants.
A supervised approach to movie emotion tracking
90 Citations2011Nikos Malandrakis, Alexandros Potamianos +2 more
A database of movie affect, annotated in continuous time, on a continuous valence-arousal scale is developed and supervised learning methods are proposed to model the continuous affective response using hidden Markov Models (independent) in each dimension.
Archive ouverte UNIGE (University of Geneva)Crowdsourcing for Affective Annotation of Video: Development of a Viewer-reported Boredom Corpus
83 Citations2010Mohammad Soleymani, Martha Larson
A new corpus to be used to evaluate algorithms for prediction of viewer-reported boredom and a use of crowdsourcing to address two shortcomings of previous affective video corpora are reported on.
Crowdsourcing-based multimedia subjective evaluations
58 Citations2013Judith Redi, Tobias Hoßfeld +4 more
High correlation between crowdsourcing and lab scores for recognizability but not for aesthetic appeal is found, indicating that crowdsourcing can be used for QoE subjective assessments as long as the workers' tasks are designed with extreme care to avoid misinterpretations.
IEEE Transactions on MultimediaCorpus Development for Affective Video Indexing
58 Citations2014Mohammad Soleymani, Martha Larson +2 more
Three dimensions emerge as critical for affective video corpora, and form the basis for the proposed guidelines: the context of viewer response, personal variation among viewers, and the effectiveness and efficiency of corpus creation.
A Large Video Database for Computational Models of Induced Emotion
49 Citations2013Yoann Baveye, Jean-Noel Bettinelli +3 more
The LIRIS-ACCEDE is an Annotated Creative Commons Emotional DatabasE composed of 9800 video clips extracted from 160 movies shared under Creative Commons licenses that allows to make this database publicly available without copyright issues.
Lecture notes in computer scienceGuess What? A Game for Affective Annotation of Video Using Crowd Sourcing
17 Citations2011Laurel D. Riek, Maria Francesca O'Connor +1 more
A Facebook game called Guess What?
From crowdsourced rankings to affective ratings
16 Citations2014Yoann Baveye, Emmanuel Dellandréa +2 more
A regression analysis is proposed to map crowdsourced rankings into affective scores in the induced valence-arousal emotional space by taking advantages of the Gaussian Processes for regression that can take into account the variance of the ratings and thus the subjectivity of emotions.