Text Analysis in R
Communication Methods and MeasuresPublished 2 October 2017Open access
Kasper Welbers, Wouter van Atteveldt, Kenneth Benoit
Citations266
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This teacher’s corner provides an overview of general steps and operations in a computational text analysis project, and demonstrates how each step can be performed using the R statistical software.
Abstract
Computational text analysis has become an exciting research field with many applications in communication research. It can be a difficult method to apply, however, because it requires knowledge of various techniques, and the software required to perform most of these techniques is not readily available in common statistical software packages. In this teacher’s corner, we address these barriers by providing an overview of general steps and operations in a computational text analysis project, and demonstrate how each step can be performed using the R statistical software. As a popular open-source platform, R has an extensive user community that develops and maintains a wide range of text analysis packages. We show that these packages make it easy to perform advanced text analytics.
Keywords
Social SciencesComputer Science
Journal of Machine Learning ResearchLatent dirichlet allocation
27,049 Citations2003David M. Blei, Andrew Y. Ng +1 more
American QuarterlyUnderstanding Media: The Extensions of Man
12,396 Citations1964Stuart Levine, Marshall McLuhan
arXiv (Cornell University)Efficient Estimation of Word Representations in Vector Space
11,710 Citations2013Tomáš Mikolov, Kai Chen +2 more
Cambridge University Press eBooksIntroduction to Information Retrieval
10,873 Citations2008Christopher D. Manning, Prabhakar Raghavan +1 more
This textbook teaches classical and web information retrieval, including web search and the related areas of text classification and text clustering from basic concepts, making it perfect for introductory courses in information retrieval for advanced undergraduates and graduate students in computer science.
The Stanford CoreNLP Natural Language Processing Toolkit
7,238 Citations2014Christopher D. Manning, Mihai Surdeanu +4 more
The design and use of the Stanford CoreNLP toolkit is described, an extensible pipeline that provides core natural language analysis, and it is suggested that this follows from a simple, approachable design, straightforward interfaces, the inclusion of robust and good quality analysis components, and not requiring use of a large amount of associated baggage.
Proceedings of the National Academy of SciencesFinding scientific topics
6,017 Citations2004Thomas L. Griffiths, Mark Steyvers
A generative model for documents is described, introduced by Blei, Ng, and Jordan, and a Markov chain Monte Carlo algorithm is presented for inference in this model, which is used to analyze abstracts from PNAS by using Bayesian model selection to establish the number of topics.
Journal of Language and Social PsychologyThe Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods
5,776 Citations2009Yla Tausczik, James W. Pennebaker
Empirical results using LIWC demonstrate its ability to detect meaning in a wide variety of experimental settings, including to show attentional focus, emotionality, social relationships, thinking styles, and individual differences.
A Comparative Study on Feature Selection in Text Categorization
4,766 Citations1997Yiming Yang, Jan Pedersen
DF thresholding, the simplest method with the lowest cost in computation, can be reliably used instead of IG or CHI when the computation of these measures are too expensive, and strong correlations between the DF, IG and CHI values of a term are found.
Wiley Interdisciplinary Reviews Computational StatisticsJournal of Statistical Software
3,607 Citations2009Jan de Leeuw
The history, motivation, and implementation of the Journal of Statistical Software are discussed, including the history of the journal itself and its purpose and scope.
Computational LinguisticsLexicon-Based Methods for Sentiment Analysis
3,255 Citations2011Maite Taboada, Julian Brooke +3 more
The Semantic Orientation CALculator (SO-CAL) uses dictionaries of words annotated with their semantic orientation (polarity and strength), and incorporates intensification and negation, and is applied to the polarity classification task.
Political AnalysisText as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
3,169 Citations2013Justin Grimmer, Brandon Stewart
American Journal of Political ScienceStructural Topic Models for Open‐Ended Survey Responses
1,929 Citations2014Margaret E. Roberts, Brandon Stewart +6 more
The structural topic model makes analyzing open-ended responses easier, more revealing, and capable of being used to estimate treatment effects, and is illustrated with analysis of text from surveys and experiments.
Big DataData Science and its Relationship to Big Data and Data-Driven Decision Making
1,533 Citations2013Foster Provost, Tom Fawcett
It is argued that there are good reasons why it has been hard to pin down exactly what is data science, and that for data science to serve business effectively, it is important to understand its relationships to other important related concepts, and to begin to identify the fundamental principles underlying data science.
Journal of Statistical SoftwareText Mining Infrastructure in<i>R</i>
1,145 Citations2008Ingo Feinerer, Kurt Hornik +1 more
The tm package is presented which provides a framework for text mining applications within R and techniques for count-based analysis methods, text clustering, text classification and string kernels are presented.
Journal of Statistical Software<b>topicmodels</b>: An<i>R</i>Package for Fitting Topic Models
1,088 Citations2011Bettina Grün, Kurt Hornik
The R package topicmodels provides basic infrastructure for fitting topic models based on data structures from the text mining package tm to estimate the similarity between documents as well as between a set of specified keywords using an additional layer of latent variables.
The Journal of Open Source Softwaretidytext: Text Mining and Analysis Using Tidy Data Principles in R
930 Citations2016Julia Silge, David M. Robinson
This package provides functions and supporting data sets to allow conversion of text to and from tidy formats, and to switch seamlessly between tidy tools and existing text mining packages.
Behavior Research MethodsMTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment
844 Citations2010Philip M. McCarthy, Scott Jarvis
Researchers are advised to consider using MTLD, vocd-D (or HD-D), and Maas in their studies, rather than any single index, noting that lexical diversity can be assessed in many ways and each approach may be informative as to the construct under investigation.
American Journal of Political ScienceA Scaling Model for Estimating Time‐Series Party Positions from Texts
681 Citations2008Jonathan Slapin, Sven‐Oliver Proksch
A scaling algorithm called WORDFISH to estimate policy positions based on word frequencies in texts is proposed and it is found that words with strong political connotations are the best discriminators between parties.
Expert Systems with ApplicationsMore than words: Social networks’ text mining for consumer brand sentiments
619 Citations2013Mohamed M. Mostafa
This study uses a random sample of 3516 tweets to evaluate consumers' sentiment towards well-known brands such as Nokia, T-Mobile, IBM, KLM and DHL and indicates a generally positive consumer sentiment towards several famous brands.
Machine LearningText Categorization with Support Vector Machines. How to Represent Texts in Input Space?
403 Citations2002Edda Leopold, Jörg Kindermann
It is shown that in the case of text classification, term-frequency transformations have a larger impact on the performance of SVM than the kernel itself.
Recent Advances in Natural Language ProcessingCustomizing Sentiment Classifiers to New Domains: a Case Study
370 Citations2005Anthony Aue, Michael Gamon
Four different approaches to customizing a sentiment classification system to a new target domain in the absence of large amounts of labeled data are surveyed and their advantages, disadvantages and performance are discussed.
European Journal of Operational ResearchThe impact of preprocessing on data mining: An evaluation of classifier sensitivity in direct marketing
214 Citations2005Sven F. Crone, Stefan Lessmann +1 more
Investigating the influence of different preprocessing techniques of attribute scaling, sampling, coding of categorical as well as coding of continuous attributes on the classifier performance of decision trees, neural networks and support vector machines provides empirical evidence that data preprocessing has a significant impact on predictive accuracy.
Digital JournalismTaking Stock of the Toolkit
209 Citations2015Jelle W. Boumans, Damian Trilling
A systematic inventory of recent applications of computational methods in journalism studies is presented, distinguishing between dictionary-based approaches, supervised machine learning, and unsupervised machine learning.
Public Relations ReviewStrategic framing in the BP crisis: A semantic network analysis of associative frames
196 Citations2011Friederike Schultz, Jan Kleinnijenhuis +3 more
arXiv (Cornell University)The jsonlite Package: A Practical and Consistent Mapping Between JSON Data and R Objects
178 Citations2014Jeroen Ooms
This paper explicitly describes a mapping between R classes and JSON data, highlights potential problems, and proposes conventions that generalize the mapping to cover all common structures.
Electoral StudiesParty contestation and Europe on the news agenda: The 2009 European Parliamentary Elections
139 Citations2010Andreas Schuck, Georgios Xezonakis +3 more
German PoliticsHow to Avoid Pitfalls in Statistical Analysis of Political Texts: The Case of Germany
115 Citations2009Sven‐Oliver Proksch, Jonathan Slapin
Journal of Elections Public Opinion and PartiesAnti-Immigrant Party Support and Media Visibility: A Cross-Party, Over-Time Perspective
103 Citations2012Rens Vliegenthart, Hajo G. Boomgaarden +1 more
Journalism StudiesA Gatekeeper among Gatekeepers
75 Citations2016Kasper Welbers, Wouter van Atteveldt +2 more
Journal of Statistical SoftwareR and the Journal of Statistical Software
57 Citations2016John L. Fox, Allison Leanage
This work traces the continuing development of R and CRAN, and the representation of R & CRAN and other statistical software in the pages of JSS.
Political AnalysisClause Analysis: Using Syntactic Information to Automatically Extract Source, Subject, and Predicate from Texts with an Application to the 2008–2009 Gaza War
41 Citations2017Wouter van Atteveldt, Tamir Sheafer +2 more
It is shown that a small set of syntactic patterns can extract clauses and identify quotes with good accuracy, significantly outperforming a baseline system based on word order.
Journal of International CommunicationThe spread of the Kremlin’s narratives by a western news agency during the Ukraine crisis
38 Citations2017Kohei Watanabe