Jumping NLP Curves: A Review of Natural Language Processing Research [Review Article]
IEEE Computational Intelligence MagazinePublished 10 April 2014
Erik Cambria, Bebo White
Citations1,022
SJR quartileQ1
SJR score1.85
SNIP3.05
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This survey article reinterprets the evolution of NLP research as the intersection of three overlapping curves-namely Syntactics, Semantics, and Pragmatics Curves which will eventually lead NLPResearch to evolve into natural language understanding.
Abstract
Natural language processing (NLP) is a theory-motivated range of computational techniques for the automatic analysis and representation of human language. NLP research has evolved from the era of punch cards and batch processing (in which the analysis of a sentence could take up to 7 minutes) to the era of Google and the likes of it (in which millions of webpages can be processed in less than a second). This review paper draws on recent developments in NLP research to look at the past, present, and future of NLP technology in a new light. Borrowing the paradigm of `jumping curves' from the field of business management and marketing prediction, this survey article reinterprets the evolution of NLP research as the intersection of three overlapping curves-namely Syntactics, Semantics, and Pragmatics Curveswhich will eventually lead NLP research to evolve into natural language understanding.
Keywords
Computer Science
Choice Reviews OnlineGenetic algorithms in search, optimization, and machine learning
49,278 Citations1989
This book brings together the computer techniques, mathematical tools, and research results that will enable both students and practitioners to apply genetic algorithms to problems in many fields.
ScholarlyCommons (University of Pennsylvania)Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
12,978 Citations2001John Lafferty, Andrew McCallum +1 more
This work presents iterative parameter estimation algorithms for conditional random fields and compares the performance of the resulting models to HMMs and MEMMs on synthetic and natural-language data.
The PageRank Citation Ranking : Bringing Order to the Web
12,645 Citations1999Lawrence M. Page, Sergey Brin +2 more
This paper describes PageRank, a mathod for rating Web pages objectively and mechanically, effectively measuring the human interest and attention devoted to them, and shows how to efficiently compute PageRank for large numbers of pages.
Foundations of statistical natural language processing
9,996 Citations1999Christopher D. Manning, Hinrich Schütze
ACM Computing SurveysMachine learning in automated text categorization
7,899 Citations2002Fabrizio Sebastiani
This survey discusses the main approaches to text categorization that fall within the machine learning paradigm and discusses in detail issues pertaining to three different problems, namely, document representation, classifier construction, and classifier evaluation.
Building a Large Annotated Corpus of English: The Penn Treebank
7,528 Citations1993Mitchell P. Marcus
Cambridge University Press eBooksThe Cognitive Structure of Emotions
6,211 Citations1988Andrew Ortony, Gerald L. Clore +1 more
arXiv (Cornell University)Natural Language Processing (almost) from Scratch
5,172 Citations2011Ronan Collobert, Jason Weston +4 more
Freebase
4,892 Citations2008Kurt Bollacker, Colin Evans +3 more
MQL provides an easy-to-use object-oriented interface to the tuple data in Freebase and is designed to facilitate the creation of collaborative, Web-based data-oriented applications.
Natural Language Processing (almost) from Scratch
3,987 Citations2011Ronan Collobert, Jason Weston +4 more
Artificial IntelligenceA logic for default reasoning
3,920 Citations1980Raymond Reiter
This paper proposes a logic for default reasoning, develops a complete proof theory and shows how to interface it with a top down resolution theorem prover, and provides criteria under which the revision of derived beliefs must be effected.
Probabilistic latent semantic indexing
3,916 Citations1999Thomas Hofmann
Probabilistic Latent Semantic Indexing is a novel approach to automated document indexing which is based on a statistical latent class model for factor analysis of count data.
Empirical Methods in Natural Language ProcessingTextRank: Bringing Order into Text
3,347 Citations2004Rada Mihalcea, Paul Tarau
TextRank, a graph-based ranking model for text processing, is introduced and it is shown how this model can be successfully used in natural language applications.
Automatic acquisition of hyponyms from large text corpora
3,283 Citations1992Marti A. Hearst
A set of lexico-syntactic patterns that are easily recognizable, that occur frequently and across text genre boundaries, and that indisputably indicate the lexical relation of interest are identified.
Computational LinguisticsA maximum entropy approach to natural language processing
3,120 Citations1996Adam Berger, Vincent J. Della Pietra +1 more
A maximum-likelihood approach for automatically constructing maximum entropy models is presented and how to implement this approach efficiently is described, using as examples several problems in natural language processing.
Machine LearningText Classification from Labeled and Unlabeled Documents using EM
2,749 Citations2000Kamal Nigam, Andrew Kachites McCallum +2 more
This paper shows that the accuracy of learned text classifiers can be improved by augmenting a small number of labeled training documents with a large pool of unlabeled documents, and presents two extensions to the algorithm that improve classification accuracy under these conditions.
IEEE Transactions on Information TheoryThree models for the description of language
2,749 Citations1956Noam Chomsky
It is found that no finite-state Markov process that produces symbols with transition from state to state can serve as an English grammar and the particular subclass of such processes that produce n -order statistical approximations to English do not come closer to matching the output of an English grammar.
Machine LearningUnsupervised Learning by Probabilistic Latent Semantic Analysis
2,443 Citations2001Thomas Hofmann
This paper proposes to make use of a temperature controlled version of the Expectation Maximization algorithm for model fitting, which has shown excellent performance in practice, and results in a more principled approach with a solid foundation in statistical inference.
Semantic Information Processing
2,175 Citations1968Marvin Minsky
This book solves different problems like resolving ambiguities in word meanings, finding analogies between things, making logical and nonlogical inference, resolving inconsistency in information engaging in coherent discourse with a person and more.
SWITCHBOARD: telephone speech corpus for research and development
2,140 Citations1992J. Godfrey, E. Holliman +1 more
Proceedings of the AAAI Conference on Artificial IntelligenceToward an Architecture for Never-Ending Language Learning
1,991 Citations2010J. Andrew Carlson, Justin Betteridge +4 more
Trends in Cognitive SciencesConstructions: a new theoretical approach to language
1,707 Citations2003Adele Ε. Goldberg
Researchers in this field argue that unusual constructions shed light on more general issues, and can illuminate what is required for a complete account of language.
Journal of Artificial Intelligence ResearchLexRank: Graph-based Lexical Centrality as Salience in Text Summarization
1,650 Citations2004G. Erkan, D. R. Radev
A new approach, LexRank, for computing sentence importance based on the concept of eigenvector centrality in a graph representation of sentences is considered and the LexRank with threshold method outperforms the other degree-based techniques including continuous LexRank.
IEEE Transactions on Neural NetworksSupport vector machines for spam categorization
1,457 Citations1999Harris Drucker, Donghui Wu +1 more
The use of support vector machines in classifying e-mail as spam or nonspam is studied by comparing it to three other classification algorithms: Ripper, Rocchio, and boosting decision trees, which found SVM's performed best when using binary features.
Learning to Classify Text Using Support Vector Machines
1,366 Citations2002Thorsten Joachims
This book gives a concise introduction to SVMs for pattern recognition, and it includes a detailed description of how to formulate text-classification tasks for machine learning.
Building Large Knowledge-Based Systems: Representation and Inference in the Cyc Project
1,281 Citations1990Douglas B. Lenat, Ramanathan V. Guha
This review has been difficult for me to write, because my thoughts about Cyc have changed a great deal since I first read the book in the spring of 1990, and my thoughts about the project today have changed a great deal.
IEEE Transactions on Neural NetworksInput space versus feature space in kernel-based methods
1,166 Citations1999Bernhard Schölkopf, Mika Sirén +5 more
The geometry of feature space is reviewed, and the connection between feature space and input space is discussed by dealing with the question of how one can, given some vector in feature space, find a preimage in input space.
Word association norms, mutual information, and lexicography
1,066 Citations1989Kenneth Church, Patrick Hanks
The proposed measure, the association ratio, estimates word association norms directly from computer readable corpora, making it possible to estimate norms for tens of thousands of words.
IEEE Transactions on Evolutionary ComputationGrammatical evolution
950 Citations2001Michael O’Neill, Conor Ryan
Conference on Computational Natural Language LearningBetter Word Representations with Recursive Neural Networks for Morphology
820 Citations2013Thang Luong, Richard Socher +1 more
This paper combines recursive neural networks, where each morpheme is a basic unit, with neural language models to consider contextual information in learning morphologicallyaware word representations and proposes a novel model capable of building representations for morphologically complex words from their morphemes.
Probase
776 Citations2012Wentao Wu, Hongsong Li +2 more
This paper presents a universal, probabilistic taxonomy that is more comprehensive than any existing ones, and contains 2.7 million concepts harnessed automatically from a corpus of 1.68 billion web pages.
Research Portal (King's College London)SKOS Simple Knowledge Organization System Reference
682 Citations2009Alistair Miles, Sean Bechhofer
This document defines the Simple Knowledge Organization System (SKOS), a common data model for sharing and linking knowledge organization systems via the Web, which provides a standard, low-cost migration path for porting existing knowledge organizations systems to the Semantic Web.
LinguisticsThe effect of usage on degrees of constituency: the reduction of don't in English
648 Citations1999Joan Bybee, Joanne Scheibman
University of North Texas Digital Library (University of North Texas)TextRank: Bringing Order into Texts
633 Citations2004Mihalcea, Rada, 1974-, Tarau, Paul
Fast collapsed gibbs sampling for latent dirichlet allocation
567 Citations2008Ian R. Porteous, David Newman +4 more
A novel collapsed Gibbs sampling method for the widely used latent Dirichlet allocation (LDA) model, which can be as much as 8 times faster than the standard collapsed Gibbs sampler for LDA and results in significant speedups on real world text corpora.
A model of textual affect sensing using real-world knowledge
531 Citations2003Hugo Liu, Henry Lieberman +1 more
This paper demonstrates a new approach, using large-scale real-world knowledge about the inherent affective nature of everyday situations to classify sentences into "basic" emotion categories, and suggests that the approach is robust enough to enable plausible affective text user interfaces.
MADOC (University of Mannheim)Deriving a large scale taxonomy from Wikipedia
447 Citations2007Simone Paolo Ponzetto, Michael Strube
A large scale taxonomy containing a large amount of subsumption is derived using methods based on connectivity in the network and lexicosyntactic matching to label the semantic relations between categories in Wikipedia.
Computer NetworksAnnotea: an open RDF infrastructure for shared Web annotations
413 Citations2002James P. Kahan, Marja-Riitta Koivunen +2 more
ACM Transactions on Information SystemsContext-sensitive learning methods for text categorization
359 Citations1999William W. Cohen, Yoram Singer
RIPPER and sleeping-experts perform extremely well across a wide variety of categorization problems, generally outperforming previously applied learning methods and are viewed as a confirmation of the usefulness of classifiers that represent contextual information.
IEEE Transactions on Fuzzy SystemsAffect analysis of text using fuzzy semantic typing
291 Citations2001Pero Subašić, Alison K. Huettner
IEEE Computational Intelligence MagazineLearning deep physiological models of affect
285 Citations2013Héctor P. Martínez, Yoshua Bengio +1 more
All three key components of an affective model are touched upon and the use of deep learning (DL) methodologies for affective modeling from multiple physiological signals are introduced.
IEEE Intelligent SystemsEnhanced SenticNet with Affective Labels for Concept-Based Opinion Mining
253 Citations2013Soujanya Poria, Alexander Gelbukh +4 more
The presented methodology enriches SenticNet concepts with affective information by assigning an emotion label by way of concept-based opinion mining.
Behavior Research MethodsCharacterization of the Affective Norms for English Words by discrete emotional categories
249 Citations2007Ryan A. Stevenson, Joseph A. Mikels +1 more
Lecture notes in computer scienceTowards Semantically-Interlinked Online Communities
245 Citations2005John G. Breslin, Andreas Harth +2 more
This work presents the SIOC ontology which combines terms from vocabularies that already exist with new terms needed to describe the relationships between concepts in the realm of online community sites.
Studies in logic and the foundations of mathematicsAn Introduction to First-Order Logic
226 Citations1977
This chapter discusses the formulas that are certain finite strings of symbols, and provides information of what can and what cannot be expressed in first-order logic.
University Libraries (University of Maryland)SHOE: A Knowledge Representation Language for Internet Applications
200 Citations1999Jeff Heflin, James Hendler +1 more
The Simple HTML Ontology Extensions (SHOE), a KR language which allows web pages to be annotated with semantics, is described and a formalism for the language is presented and the features which make it well suited for the Web are discussed.
IEEE Transactions on Fuzzy SystemsComputing With Words Is an Implementable Paradigm: Fuzzy Queries, Linguistic Data Summaries, and Natural-Language Generation
186 Citations2010Janusz Kacprzyk, Sławomir Zadrożny
This paper presents crucial elements of CWW, in particular Zadeh's protoforms, and indicates their power and stress a need to develop new tools to handle more modalities, and presents the approach to linguistic data(base) summaries, which is a very intuitive and human-consistent natural-language-based knowledge-discovery tool.
IEEE Intelligent SystemsFeature Ensemble Plus Sample Selection: Domain Adaptation for Sentiment Classification
169 Citations2013Rui Xia, Chengqing Zong +2 more
Experimental results show that the proposed SS-FE approach could gain significant improvements, compared to FE or PCA-SS, because of its comprehensive consideration of both labeling adaptation and instance adaptation.
Oxford University Press eBooksLanguage Evolution: The Hardest Problem in Science?
166 Citations2003Morten H. Christiansen, Simon Kirby
IEEE Transactions on Knowledge and Data EngineeringNatural language grammatical inference with recurrent neural networks
142 Citations2000Sandra Lawrence, C. Lee Giles +1 more
It was found that certain architectures are better able to learn an appropriate grammar than others, and the extraction of rules in the form of deterministic finite state automata is investigated.
IEEE Computational Intelligence MagazineFrame-Based Detection of Opinion Holders and Topics: A Model and a Tool
125 Citations2014Aldo Gangemi, Valentina Presutti +1 more
Sentilo implements an approach based on the neo-Davidsonian assumption that events and situations are the primary entities for contextualizing opinions, which makes it able to distinguish holders, main topics, and sub-topics of an opinion.
NeurocomputingAn ELM-based model for affective analogical reasoning
110 Citations2014Erik Cambria, Paolo Gastaldo +2 more
This paper explores how the high generalization performance, low computational complexity, and fast learning speed of extreme learning machines can be exploited to perform analogical reasoning in a vector space model of affective common-sense knowledge.
Cognitive ScienceConnectionist Natural Language Processing: The State of the Art
105 Citations1999Morten H. Christiansen, Nick Chater
IEEE Computational Intelligence MagazineAn Intelligent Self-Organization Scheme for the Internet of Things
93 Citations2013Yongsheng Ding, Yanling Jin +2 more
Simulation results verify the performance of the proposed mechanism that entitles the IoT to the ability of maintaining its status in a globally stable status, while effectively discovering the random service requests in a resource-critical configuration.
Human-centric Computing and Information SciencesIntention awareness: improving upon situation awareness in human-centric environments
93 Citations2013Newton Howard, Erik Cambria
This paper argues that computational analysis and tracking of semantic and affective information associated with human actors' intentions are an effective way to minimize miscommunication and uncertainty, particularly in time-sensitive and information-saturated situations.
IEEE Computational Intelligence MagazineA Probabilistic Generative Model for Mining Cybercriminal Networks from Online Social Media
88 Citations2014Raymond Y.K. Lau, Yunqing Xia +1 more
This is the first successful research of applying a probabilistic generative model to mine cybercriminal networks from online social media using a novel weakly supervised cybercriminal network mining method to facilitate cybercrime forensics.
DSpace@MIT (Massachusetts Institute of Technology)The Strong Story Hypothesis and the Directed Perception Hypothesis
71 Citations2011Patrick Henry Winston
The Strong Story Hypothesis, which holds that story telling and understanding have a central role in human intelligence, is introduced and the representations and methods embodied in the Genesis system are described, a story-understanding system that analyzes stories ranging.
Big Social Data Analysis
70 Citations2013Erik Cambria, Dheeraj Rajagopal +2 more
IEEE Transactions on Evolutionary ComputationComputational Intelligence for Evolving Trading Rules
69 Citations2009Adam Ghandar, Zbigniew Michalewicz +3 more
An adaptive computational intelligence system for learning trading rules is described, and using an artificial evolutionary process the system learns to form rules that can perform well in dynamic market conditions.
Interaction Studies Social Behaviour and Communication in Biological and Artificial SystemsStory and discourse
58 Citations2007R. Michael Young
This approach adopts a bipartite model taken from narrative theory, in which narrative is composed of story and discourse, which is defined in terms of plans that drive the dynamics of a virtual environment.
Developing a fuzzy search engine based on fuzzy ontology and semantic search
43 Citations2011Lien‐Fu Lai, Chao‐Chin Wu +2 more
A fuzzy search engine is developed, called Fuzzy-Go, which can automatically retrieve web pages that contain synonyms or terms similar to keywords and a fuzzy ontology is constructed by using fuzzy logic to capture the similarities of terms in the ontology.
Literary and Linguistic ComputingModelling Space and Time in Narratives about Restaurants
41 Citations2006Erik T. Mueller
This study investigated the automatic modelling of space and time in narratives involving dining in a restaurant by building a program that uses information extraction techniques to convert narrative texts into templates containing key information about the dining episodes discussed in the narratives.
IEEE Computational Intelligence MagazineFuzzy Logic Models for the Meaning of Emotion Words
40 Citations2013Abe Kazemzadeh, Sungbok Lee +1 more
Two models that use interval type-2 fuzzy sets (IT2 FSs) for representing the meaning of words that refer to emotions are presented and the theoretical insights that the two models allow and the advantages and disadvantages of each are discussed.
HMM-based Passage Models for Document Classification and Ranking
40 Citations2001Ludovic Denoyer, Hugo Zaragoza +1 more
A family of models that take into account the fact that relevant documents may contain irrelevant passages are considered, which generalizes the multinomial Naive Bayes and is derived from a more general model for different access tasks.
Journal of Product Innovation ManagementJumping the Curve: Innovation and Strategic Choice in an Age of Transition
38 Citations1996
Text categorization for multi-page documents
34 Citations2001Paolo Frasconi, Giovanni Soda +1 more
A critical survey on the use of Fuzzy Sets in Speech and Natural Language Processing
30 Citations2012João Paulo Carvalho, Fernando Batista +1 more
This paper shows how the use and applications of Fuzzy Sets in Speech and Natural Language Processing have seen a steady decline to a point where FS are virtually unknown or unappealing, and proposes some guidelines on what could be done to reverse it and make FS assume a relevant role in SNLP.
Common Sense Conversations: Understanding Casual Conversation using a Common Sense Database
27 Citations2003Nathan Eagle, Push Singh
This work introduces a method for situation understanding in natural, face-to-face conversation that combines a network of commonsense knowledge with keyword spotting and contextual information automatically obtained from a wearable device such as a PDA or cell phone.
IEEE Transactions on Evolutionary ComputationA semantically guided and domain-independent evolutionary model for knowledge discovery from texts
23 Citations2003John Atkinson, Chris Mellish +1 more
A novel evolutionary model for knowledge discovery from texts (KDTs), which deals with issues concerning shallow text representation and processing for mining purposes in an integrated way, and some promising results indicate the plausibility of the model for effective KDT.
Connectionist Natural Language Processing: A Status Report
23 Citations2007Michael G. Dyer
The summation and thresholding of activation allows for smooth integration of multiple sources of knowledge and the hope is that connectionist approaches to NLP will replace the more traditional, symbolic approaches toNLP.
IEEE Transactions on Evolutionary ComputationSymbiosis of Evolutionary Techniques and Statistical Natural Language Processing
21 Citations2004Lourdes Araujo
A general scheme of application of evolutionary techniques to NLP is defined, which gives the mainstream for the design of the elements of the algorithm and has been illustrated with two fundamental applications in NLP: tagging and parsing.
IEEE Computational Intelligence MagazineReasoning with Large Scale Ontologies in Fuzzy pD* Using MapReduce
20 Citations2012Chang Liu, Guilin Qi +2 more
This is the first work to investigate how MapReduce can be applied to solve the scalability issue of fuzzy reasoning in OWL, and the running time of the system is comparable with that of WebPIE, the state-of-the-art inference engine for scalable reasoning in pD* semantics.
IEEE Transactions on Neural NetworksSymbolic connectionism in natural language disambiguation
19 Citations1998Samuel W. K. Chan, James Franklin
This hybrid system is more sophisticated in tackling language disambiguation problems by using linguistic clues from disparate sources as well as modeling context effects into the sentence analysis, and is potentially more powerful than any systems relying on one processing paradigm.
Relative Performance Guarantees for Approximate Inference in Latent Dirichlet Allocation
19 Citations2008Indraneel Mukherjee, David M. Blei
It is proved that the difference in the tightness of the bound on the likelihood of a document decreases as O(k - 1) + √log m/m, where k is the number of topics in the model and m is thenumber of words in a document, and the advantage of CVB over VB is lost for long documents but increases with the numberof topics.
COGVIEW & INTELNET: Nuanced energy-based knowledge representation and integrated cognitive-conceptual framework for realistic culture, values, and concept-affected systems simulation
16 Citations2013Daniel Olsher
This paper presents a novel `Energy-Based' Knowledge Representation formalism (INTELNET) ideal for modeling, fusing, and reasoning about nuanced semantics, cultures, affects, and worldviews and advances a detailed example within the suicide terrorism domain.
Fuzzy Sets in Natural Language Processing
16 Citations1992Vilém Novák
There are many linguistic systems, often based on set theory and logic, attempting to grasp (at least some phenomena) of the natural language, but none of them is fully acceptted and satisfactory in all respects.
IEEE Computational Intelligence MagazineEvolutionary and Swarm Computing for the Semantic Web
13 Citations2012Christophe Guéret, Stefan Schlobach +3 more
This paper introduces and systemically discusses the typical application problems on the Semantic Web and argues that the existing approaches to address their underlying reasoning tasks consistently fail because of the increasing size, dynamicity and complexity of the data.
IEEE Transactions on Neural NetworksAmbiguity Resolution Analysis in Incremental Parsing of Natural Language
11 Citations2005Fabrizio Costa, Paolo Frasconi +3 more
A systematic analysis of the behavior of the network allows for important insights about the kind of information that is exploited to resolve different forms of ambiguity in incremental parsing.
The MIT Press eBooksCorrelational Analysis and Mechanical Translation
11 Citations2003Silvio Ceccato
This chapter contains sections titled: The Three Approaches to Mechanical Translation, Linguistics for Machines, The Fourth Approach to MT, An Illustration of the Third Approach, References.
Communications in computer and information scienceNarratarium: An Immersive Storytelling Environment
8 Citations2013Kasia Hayden, Dan Novy +4 more
The Narratarium Colorizer device receives either keyboard input or speech recognition input and uses natural language processing to extract key terms, which are outputs uniformly throughout the room using a custom designed curved mirror projection system.
Narrative is a Key Cognitive Competency.
5 Citations2011Mark A. Finlayson, Patrick Henry Winston
It is speculated that culture is, in fact, a necessary cognitive re- source (akin to commonsense, or the lexicon), and that any cognitive architecture intended to achieve human-level performance will need to have a 'culture' of its own.
…