Recent Trends in Deep Learning Based Natural Language Processing [Review Article]
IEEE Computational Intelligence MagazinePublished 20 July 2018
Tom Young, Devamanyu Hazarika, Soujanya Poria, Erik Cambria
Citations2,860
SJR quartileQ1
SJR score1.85
SNIP3.05
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper reviews significant deep learning related models and methods that have been employed for numerous NLP tasks and provides a walk-through of their evolution.
Abstract
Deep learning methods employ multiple processing layers to learn hierarchical representations of data, and have produced state-of-the-art results in many domains. Recently, a variety of model designs and methods have blossomed in the context of natural language processing (NLP). In this paper, we review significant deep learning related models and methods that have been employed for numerous NLP tasks and provide a walk-through of their evolution. We also summarize, compare and contrast the various models and put forward a detailed understanding of the past, present and future of deep learning in NLP.
Keywords
Computer Science
Neural ComputationLong Short-Term Memory
98,079 Citations1997Sepp Hochreiter, Jürgen Schmidhuber
A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)MizAR 60 for Mizar 50
76,311 Citations2023Jakubův, Jan, Chvalovský, Karel +7 more
Communications of the ACMImageNet classification with deep convolutional neural networks
75,709 Citations2017Alex Krizhevsky, Ilya Sutskever +1 more
A large, deep convolutional neural network was trained to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes and employed a recently developed regularization method called "dropout" that proved to be very effective.
Glove: Global Vectors for Word Representation
33,769 Citations2014Jeffrey Pennington, Richard Socher +1 more
A new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods and produces a vector space with meaningful substructure.
Journal of Machine Learning ResearchLatent dirichlet allocation
27,049 Citations2003David M. Blei, Andrew Y. Ng +1 more
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
24,447 Citations2014Kyunghyun Cho, Bart van Merriënboer +5 more
Qualitatively, the proposed RNN Encoder‐Decoder model learns a semantically and syntactically meaningful representation of linguistic phrases.
Journal of Japan Society for Fuzzy Theory and Intelligent InformaticsGAN（Generative Adversarial Nets）
21,715 Citations2017Ian Goodfellow
Generative Adversarial Nets
19,938 Citations2023Raphael Labaca-Castro
arXiv (Cornell University)Distributed Representations of Words and Phrases and their Compositionality
18,086 Citations2013Tomáš Mikolov, Ilya Sutskever +3 more
This paper presents a simple method for finding phrases in text, and shows that learning good vector representations for millions of phrases is possible and describes a simple alternative to the hierarchical softmax called negative sampling.
arXiv (Cornell University)Efficient Estimation of Word Representations in Vector Space
18,050 Citations2013Tomáš Mikolov, Kai Chen +2 more
Two novel model architectures for computing continuous vector representations of words from very large data sets are proposed and it is shown that these vectors provide state-of-the-art performance on the authors' test set for measuring syntactic and semantic word similarities.
Learning Internal Representations by Error Propagation
16,235 Citations1985David E. Rumelhart, Geoffrey E. Hinton +1 more
UvA-DARE (University of Amsterdam)Auto-Encoding Variational Bayes
15,628 Citations2013Diederik P. Kingma, Max Welling
A stochastic variational inference and learning algorithm that scales to large datasets and, under some mild differentiability conditions, even works in the intractable case is introduced.
arXiv (Cornell University)Neural Machine Translation by Jointly Learning to Align and Translate
14,565 Citations2014Dzmitry Bahdanau
Convolutional Neural Networks for Sentence Classification
13,798 Citations2014Yoon Kim
The CNN models discussed herein improve upon the state of the art on 4 out of 7 tasks, which include sentiment analysis and question classification, and are proposed to allow for the use of both task-specific and static vectors.
arXiv (Cornell University)Sequence to Sequence Learning with Neural Networks
13,362 Citations2014Ilya Sutskever, Oriol Vinyals +1 more
arXiv (Cornell University)Efficient Estimation of Word Representations in Vector Space
11,710 Citations2013Tomáš Mikolov, Kai Chen +2 more
Caffe
11,194 Citations2014Yangqing Jia, Evan Shelhamer +6 more
Caffe provides multimedia scientists and practitioners with a clean and modifiable framework for state-of-the-art deep learning algorithms and a collection of reference models for training and deploying general-purpose convolutional neural networks and other deep models efficiently on commodity architectures.
Cognitive ScienceFinding Structure in Time
10,841 Citations1990Jeffrey L. Elman
A proposal along these lines first described by Jordan (1986) which involves the use of recurrent links in order to provide networks with a dynamic memory and suggests a method for representing lexical categories and the type/token distinction is developed.
arXiv (Cornell University)Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
10,728 Citations2014Jun‐Young Chung, Çaǧlar Gülçehre +2 more
These advanced recurrent units that implement a gating mechanism, such as a long short-term memory (LSTM) unit and a recently proposed gated recurrent unit (GRU), are found to be comparable to LSTM.
Transactions of the Association for Computational LinguisticsEnriching Word Vectors with Subword Information
9,784 Citations2017Piotr Bojanowski, Édouard Grave +2 more
A new approach based on the skipgram model, where each word is represented as a bag of character n-grams, with words being represented as the sum of these representations, which achieves state-of-the-art performance on word similarity and analogy tasks.
Speech recognition with deep recurrent neural networks
8,837 Citations2013Alex Graves, Abdelrahman Mohamed +1 more
This paper investigates deep recurrent neural networks, which combine the multiple levels of representation that have proved so effective in deep networks with the flexible use of long range context that empowers RNNs.
arXiv (Cornell University)Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
7,525 Citations2015Kelvin Xu, Jimmy Ba +6 more
An attention based model that automatically learns to describe the content of images is introduced that can be trained in a deterministic manner using standard backpropagation techniques and stochastically by maximizing a variational lower bound.
Machine LearningSimple statistical gradient-following algorithms for connectionist reinforcement learning
7,386 Citations1992Ronald J. Williams
This article presents a general class of associative reinforcement learning algorithms for connectionist networks containing stochastic units that are shown to make weight adjustments in a direction that lies along the gradient of expected reinforcement in both immediate-reinforcement tasks and certain limited forms of delayed-reInforcement tasks, and they do this without explicitly computing gradient estimates.
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
6,774 Citations2013Richard Socher, Alex Perelygin +5 more
A Sentiment Treebank that includes fine grained sentiment labels for 215,154 phrases in the parse trees of 11,855 sentences and presents new challenges for sentiment compositionality, and introduces the Recursive Neural Tensor Network.
Show and tell: A neural image caption generator
6,363 Citations2015Oriol Vinyals, Alexander Toshev +2 more
This paper presents a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image.
arXiv (Cornell University)Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
5,634 Citations2016Yonghui Wu, Mike Schuster +29 more
GNMT, Google's Neural Machine Translation system, is presented, which attempts to address many of the weaknesses of conventional phrase-based translation systems and provides a good balance between the flexibility of "character"-delimited models and the efficiency of "word"-delicited models.
Recurrent neural network based language model
5,349 Citations2010Tomáš Mikolov, Martin Karafiát +3 more
Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model.
A unified architecture for natural language processing
5,204 Citations2008Ronan Collobert, Jason Weston
This work describes a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense using a language model.
arXiv (Cornell University)Natural Language Processing (almost) from Scratch
5,172 Citations2011Ronan Collobert, Jason Weston +4 more
arXiv (Cornell University)Distributed Representations of Sentences and Documents
5,128 Citations2014Quoc V. Le, Tomáš Mikolov
Paragraph Vector is an unsupervised algorithm that learns fixed-length feature representations from variable-length pieces of texts, such as sentences, paragraphs, and documents, and its construction gives the algorithm the potential to overcome the weaknesses of bag-of-words models.
Deep visual-semantic alignments for generating image descriptions
4,978 Citations2015Andrej Karpathy, Li Fei-Fei
A model that generates natural language descriptions of images and their regions using a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks over sentences, and a structured objective that aligns the two modalities through a multimodal embedding is presented.
Neural Architectures for Named Entity Recognition
4,428 Citations2016Guillaume Lample, Miguel Ballesteros +3 more
Comunicacio presentada a la 2016 Conference of the North American Chapter of the Association for Computational Linguistics, celebrada a San Diego (CA, EUA) els dies 12 a 17 of juny 2016.
CNN Features Off-the-Shelf: An Astounding Baseline for Recognition
4,341 Citations2014Ali Sharif Razavian, Hossein Azizpour +2 more
A series of experiments conducted for different recognition tasks using the publicly available code and model of the OverFeat network which was trained to perform object classification on ILSVRC13 suggest that features obtained from deep learning with convolutional nets should be the primary candidate in most visual recognition tasks.
IEEE Transactions on Pattern Analysis and Machine IntelligenceMultimodal Machine Learning: A Survey and Taxonomy
4,232 Citations2018Tadas Baltrušaitis, Chaitanya Ahuja +1 more
This paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy to enable researchers to better understand the state of the field and identify directions for future research.
Natural Language Processing (almost) from Scratch
3,987 Citations2011Ronan Collobert, Jason Weston +4 more
A Convolutional Neural Network for Modelling Sentences
3,561 Citations2014Nal Kalchbrenner, Edward Grefenstette +1 more
A convolutional architecture dubbed the Dynamic Convolutional Neural Network (DCNN) is described that is adopted for the semantic modelling of sentences and induces a feature graph over the sentence that is capable of explicitly capturing short and long-range relations.
Leibniz-Zentrum für Informatik (Schloss Dagstuhl)NER in Archival Finding Aids
3,270 Citations2021Zhiheng Huang, Wei Xu +1 more
This work has created a set of datasets to train Machine Learning algorithms to find named entities and geographical locations and tested them in several datasets and registered their precision and accuracy.
Statistical phrase-based translation
3,268 Citations2003Philipp Koehn, Franz Josef Och +1 more
The empirical results suggest that the highest levels of performance can be obtained through relatively simple means: heuristic learning of phrase translations from word-based alignments and lexical weighting of phrase translation.
Leibniz-Zentrum für Informatik (Schloss Dagstuhl)LOL: An Investigation into Cybernetic Humor, or: Can Machines Laugh?
3,084 Citations2016Alex Graves, Gervasi, Vincenzo +1 more
This paper shows how Long Short-term Memory recurrent neural networks can be used to generate complex sequences with long-range structure, simply by predicting one data point at a time.
Journal of Artificial Intelligence ResearchFrom Frequency to Meaning: Vector Space Models of Semantics
2,883 Citations2010Peter D. Turney, Patrick Pantel
The goal in this survey is to show the breadth of applications of VSMs for semantics, to provide a new perspective on VSMs, and to provide pointers into the literature for those who are less familiar with the field.
Journal of Machine Learning ResearchA neural probabilistic language model
2,663 Citations2003BengioYoshua, DucharmeRéjean +2 more
This work reports on experiments using neural networks for the probability function, showing on two text corpora that the proposed approach very significantly improves on a state-of-the-art trigram model.
IEEE Transactions on Acoustics Speech and Signal ProcessingPhoneme recognition using time-delay neural networks
2,625 Citations1989Alexander Waibel, Toshiyuki Hanazawa +3 more
The authors present a time-delay neural network (TDNN) approach to phoneme recognition which is characterized by two important properties: using a three-layer arrangement of simple computing units, a hierarchy can be constructed that allows for the formation of arbitrary nonlinear decision surfaces, which the TDNN learns automatically using error backpropagation.
Learning to forget: continual prediction with LSTM
2,383 Citations1999Felix A. Gers
Attention-based LSTM for Aspect-level Sentiment Classification
2,344 Citations2016Yequan Wang, Minlie Huang +2 more
This paper reveals that the sentiment polarity of a sentence is not only determined by the content but is also highly related to the concerned aspect, and proposes an Attention-based Long Short-Term Memory Network for aspect-level sentiment classification.
Proceedings of the AAAI Conference on Artificial IntelligenceSeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
2,302 Citations2017Lantao Yu, Weinan Zhang +2 more
Modeling the data generator as a stochastic policy in reinforcement learning (RL), SeqGAN bypasses the generator differentiation problem by directly performing gradient policy update.
IEEE/ACM Transactions on Audio Speech and Language ProcessingConvolutional Neural Networks for Speech Recognition
2,262 Citations2014Ossama Abdel‐Hamid, Abdelrahman Mohamed +4 more
It is shown that further error rate reduction can be obtained by using convolutional neural networks (CNNs), and a limited-weight-sharing scheme is proposed that can better model speech features.
Seeing stars
2,127 Citations2005Bo Pang, Lillian Lee
A meta-algorithm is applied, based on a metric labeling formulation of the rating-inference problem, that alters a given n-ary classifier's output in an explicit attempt to ensure that similar items receive similar labels.
A Diversity-Promoting Objective Function for Neural Conversation Models
1,986 Citations2016Jiwei Li, Michel Galley +3 more
This work proposes using Maximum Mutual Information (MMI) as the objective function in neural models, and demonstrates that the proposed MMI models produce more diverse, interesting, and appropriate responses, yielding substantive gains in BLEU scores on two conversational datasets and in human evaluations.
arXiv (Cornell University)Convolutional Sequence to Sequence Learning
1,897 Citations2017Jonas Gehring, Michael Auli +3 more
This work introduces an architecture based entirely on convolutional neural networks, which outperform the accuracy of the deep LSTM setup of Wu et al. (2016) on both WMT'14 English-German and WMT-French translation at an order of magnitude faster speed, both on GPU and CPU.
A Fast and Accurate Dependency Parser using Neural Networks
1,879 Citations2014Danqi Chen, Christopher D. Manning
This work proposes a novel way of learning a neural network classifier for use in a greedy, transition-based dependency parser that can work very fast, while achieving an about 2% improvement in unlabeled and labeled attachment scores on both English and Chinese datasets.
Towards End-To-End Speech Recognition with Recurrent Neural Networks
1,852 Citations2014Alex Graves, Navdeep Jaitly
A speech recognition system that directly transcribes audio data with text, without requiring an intermediate phonetic representation is presented, based on a combination of the deep bidirectional LSTM recurrent neural network architecture and the Connectionist Temporal Classification objective function.
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models
1,725 Citations2016Iulian Vlad Serban, Alessandro Sordoni +3 more
Extensions of recurrent neural network language model
1,596 Citations2011Tomáš Mikolov, Stefan Kombrink +3 more
Several modifications of the original recurrent neural network language model are presented, showing approaches that lead to more than 15 times speedup for both training and testing phases and possibilities how to reduce the amount of parameters in the model.
Domain adaptation for large-scale sentiment classification: A deep learning approach
1,563 Citations2011Xavier Glorot, Antoine Bordes +1 more
A deep learning approach is proposed which learns to extract a meaningful representation for each review in an unsupervised fashion and clearly outperform state-of-the-art methods on a benchmark composed of reviews of 4 types of Amazon products.
Document Modeling with Gated Recurrent Neural Network for Sentiment Classification
1,535 Citations2015Duyu Tang, Bing Qin +1 more
A neural network model is introduced to learn vector-based document representation in a unified, bottom-up fashion and dramatically outperforms standard recurrent neural network in document modeling for sentiment classification.
arXiv (Cornell University)A Neural Conversational Model
1,500 Citations2015Oriol Vinyals, Quoc V. Le
A simple approach to conversational modeling which uses the recently proposed sequence to sequence framework, and is able to extract knowledge from both a domain specific dataset, and from a large, noisy, and general domain dataset of movie subtitles.
Machine LearningDistributed representations, simple recurrent networks, and grammatical structure
1,381 Citations1991Jeffrey L. Elman
International Conference on Machine LearningConvolutional Sequence to Sequence Learning
1,340 Citations2017Jonas Gehring, Michael Auli +3 more
Neural Information Processing SystemsPointer networks
1,332 Citations2015Oriol Vinyals, Meire Fortunato +1 more
A new neural architecture to learn the conditional probability of an output sequence with elements that are discrete tokens corresponding to positions in an input sequence using a recently proposed mechanism of neural attention, called Ptr-Nets, which improves over sequence-to-sequence with input attention, but also allows it to generalize to variable size output dictionaries.
Semantic Compositionality through Recursive Matrix-Vector Spaces
1,297 Citations2012Richard Socher, Brody Huval +2 more
A recursive neural network model that learns compositional vector representations for phrases and sentences of arbitrary syntactic type and length and can learn the meaning of operators in propositional logic and natural language is introduced.
arXiv (Cornell University)A Deep Reinforced Model for Abstractive Summarization
1,272 Citations2017Romain Paulus, Caiming Xiong +1 more
arXiv (Cornell University)Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
1,268 Citations2015Samy Bengio, Oriol Vinyals +2 more
Max-Margin Markov Networks
1,251 Citations2003Ben Taskar, Carlos Guestrin +1 more
Maximum margin Markov (M3) networks incorporate both kernels, which efficiently deal with high-dimensional features, and the ability to capture correlations in structured data, and a new theoretical bound for generalization in structured domains is provided.
Parsing Natural Scenes and Natural Language with Recursive Neural Networks
1,209 Citations2011Richard Socher, Cliff Chiung-Yu Lin +2 more
A max-margin structure prediction architecture based on recursive neural networks that can successfully recover such structure both in complex scene images as well as sentences is introduced.
Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions
1,198 Citations2011Richard Socher, Jeffrey Pennington +3 more
A novel machine learning framework based on recursive autoencoders for sentence-level prediction of sentiment label distributions that outperform other state-of-the-art approaches on commonly used datasets, without using any pre-defined sentiment lexica or polarity shifting rules.
International Conference on Computational LinguisticsDeep Convolutional Neural Networks for Sentiment Analysis of Short Texts
1,191 Citations2014Cícero dos Santos, Maíra Gatti de Bayser
A new deep convolutional neural network is proposed that exploits from characterto sentence-level information to perform sentiment analysis of short texts and achieves state-of-the-art results for single sentence sentiment prediction.
Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification
1,167 Citations2014Duyu Tang, Furu Wei +4 more
Three neural networks are developed to effectively incorporate the supervision from sentiment polarity of text (e.g. sentences or tweets) in their loss functions and the performance of SSWE is improved by concatenating SSWE with existing feature set.
Generating Text with Recurrent Neural Networks
1,166 Citations2011Ilya Sutskever, James Martens +1 more
The power of RNNs trained with the new Hessian-Free optimizer by applying them to character-level language modeling tasks is demonstrated, and a new RNN variant that uses multiplicative connections which allow the current input character to determine the transition matrix from one hidden state vector to the next is introduced.
arXiv (Cornell University)Language Modeling with Gated Convolutional Networks
1,129 Citations2016Yann Dauphin, Angela Fan +2 more
Annual Review of Information Science and TechnologyLatent semantic analysis
1,106 Citations2004Susan Dumais
Latent semantic analysis (LSA) approximates human understanding of relations between word and passage meanings in a wide variety of ways.
arXiv (Cornell University)Training Very Deep Networks
1,100 Citations2015Rupesh K. Srivastava, Klaus Greff +1 more
Deep Reinforcement Learning for Dialogue Generation
1,058 Citations2016Jiwei Li, Will Monroe +4 more
This work simulates dialogues between two virtual agents, using policy gradient methods to reward sequences that display three useful conversational properties: informativity, non-repetitive turns, coherence, and ease of answering.
Proceedings of the AAAI Conference on Artificial IntelligenceCharacter-Aware Neural Language Models
1,046 Citations2016Yoon Kim, Yacine Jernite +2 more
A simple neural language model that relies only on character-level inputs that is able to encode, from characters only, both semantic and orthographic information and suggests that on many languages, character inputs are sufficient for language modeling.
arXiv (Cornell University)Character-Aware Neural Language Models
1,022 Citations2015Yoon Kim, Yacine Jernite +2 more
IEEE Computational Intelligence MagazineJumping NLP Curves: A Review of Natural Language Processing Research [Review Article]
1,022 Citations2014Erik Cambria, Bebo White
This survey article reinterprets the evolution of NLP research as the intersection of three overlapping curves-namely Syntactics, Semantics, and Pragmatics Curves which will eventually lead NLPResearch to evolve into natural language understanding.
arXiv (Cornell University)Convolutional Neural Network Architectures for Matching Natural Language Sentences
970 Citations2015Baotian Hu, Zhengdong Lu +2 more
Convolutional neural network models for matching two sentences are proposed, by adapting the convolutional strategy in vision and speech and nicely represent the hierarchical structures of sentences with their layer-by-layer composition and pooling.
arXiv (Cornell University)Sequence Level Training with Recurrent Neural Networks
949 Citations2015Marc’Aurelio Ranzato, Sumit Chopra +2 more
Event Extraction via Dynamic Multi-Pooling Convolutional Neural Networks
929 Citations2015Yubo Chen, Liheng Xu +3 more
A word-representation model to capture meaningful semantic regularities for words and a framework based on a convolutional neural network to capture sentence-level clues are introduced.
A Persona-Based Neural Conversation Model
897 Citations2016Jiwei Li, Michel Galley +4 more
This work presents persona-based models for handling the issue of speaker consistency in neural response generation that yield qualitative performance improvements in both perplexity and BLEU scores over baseline sequence-to-sequence models.
arXiv (Cornell University)Comparative Study of CNN and RNN for Natural Language Processing
894 Citations2017Wenpeng Yin, Katharina Kann +2 more
This work is the first systematic comparison of CNN and RNN on a wide range of representative NLP tasks, aiming to give basic guidance for DNN selection.
Proceedings of the IEEEPOMDP-Based Statistical Spoken Dialog Systems: A Review
892 Citations2013Steve Young, Milica Gašić +2 more
This review article provides an overview of the current state of the art in the development of POMDP-based spoken dialog systems.
International Conference on Machine LearningLanguage modeling with gated convolutional networks
884 Citations2017Yann Dauphin, Angela Fan +2 more
arXiv (Cornell University)Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition
856 Citations2014Haşim Sak, Andrew Senior +1 more
Novel LSTM based RNN architectures which make more effective use of model parameters to train acoustic models for large vocabulary speech recognition are presented.
Knowledge-Based SystemsAspect extraction for opinion mining with a deep convolutional neural network
852 Citations2016Soujanya Poria, Erik Cambria +1 more
This paper used a 7-layer deep convolutional neural network to tag each word in opinionated sentences as either aspect or non-aspect word, and developed a set of linguistic patterns for the same purpose and combined them with the neural network.
Context-Dependent Sentiment Analysis in User-Generated Videos
839 Citations2017Soujanya Poria, Erik Cambria +4 more
A LSTM-based model is proposed that enables utterances to capture contextual information from their surroundings in the same video, thus aiding the classification process and showing 5-10% performance improvement over the state of the art and high robustness to generalizability.
Learning accurate, compact, and interpretable tree annotation
812 Citations2006Slav Petrov, Leon Barrett +2 more
An automatic approach to tree annotation in which basic nonterminal symbols are alternately split and merged to maximize the likelihood of a training treebank is presented.
A Neural Network Approach to Context-Sensitive Generation of Conversational Responses
804 Citations2015Alessandro Sordoni, Michel Galley +7 more
A neural network architecture is used to address sparsity issues that arise when integrating contextual information into classic statistical models, allowing the system to take into account previous dialog utterances.
Adversarial Learning for Neural Dialogue Generation
765 Citations2017Jiwei Li, Will Monroe +4 more
This work applies adversarial training to open-domain dialogue generation, training a system to produce sequences that are indistinguishable from human-generated dialogue utterances, and investigates models for adversarial evaluation that uses success in fooling an adversary as a dialogue evaluation metric, while avoiding a number of potential pitfalls.
Journal of Memory and LanguageSymbol Grounding and Meaning: A Comparison of High-Dimensional and Embodied Theories of Meaning
758 Citations2000Arthur M. Glenberg, David A. Robertson
arXiv (Cornell University)End-To-End Memory Networks
689 Citations2015Sainbayar Sukhbaatar, Arthur Szlam +2 more
A neural network with a recurrent attention model over a possibly large external memory that is trained end-to-end, and hence requires significantly less supervision during training, making it more generally applicable in realistic settings.
A Latent Semantic Model with Convolutional-Pooling Structure for Information Retrieval
682 Citations2014Yelong Shen, Xiaodong He +3 more
A new latent semantic model that incorporates a convolutional-pooling structure over word sequences to learn low-dimensional, semantic vector representations for search queries and Web documents is proposed.
IEEE Intelligent SystemsMultimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages
674 Citations2016Amir Zadeh, Rowan Zellers +2 more
This article addresses the fundamental question of exploiting the dynamics between visual gestures and verbal messages to be able to better model sentiment by introducing the first multimodal dataset with opinion-level sentiment intensity annotations and proposing a new computational representation, called multi-modal dictionary, based on a language-gesture study.
International Conference on Learning RepresentationsSequence Level Training with Recurrent Neural Networks
671 Citations2016Marc’Aurelio Ranzato, Sumit Chopra +2 more
This work proposes a novel sequence level training algorithm that directly optimizes the metric used at test time, such as BLEU or ROUGE, and outperforms several strong baselines for greedy generation.
arXiv (Cornell University)Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
631 Citations2015Ankit Kumar, Ozan İrsoy +7 more
The dynamic memory network (DMN), a neural network architecture which processes input sequences and questions, forms episodic memories, and generates relevant answers, is introduced.
Proceedings of the AAAI Conference on Artificial IntelligenceTargeted Aspect-Based Sentiment Analysis via Embedding Commonsense Knowledge into an Attentive LSTM
608 Citations2018Yukun Ma, Haiyun Peng +1 more
A novel solution to targeted aspect-based sentiment analysis, which tackles the challenges of both aspect- based sentiment analysis and targeted sentiment analysis by exploiting commonsense knowledge by augmenting the LSTM network with a hierarchical attention mechanism.
arXiv (Cornell University)Dynamic Memory Networks for Visual and Textual Question Answering
595 Citations2016Caiming Xiong, Stephen Merity +1 more
The new DMN+ model improves the state of the art on both the Visual Question Answering dataset and the \babi-10k text question-answering dataset without supporting fact supervision.
…