Efficient Natural Language Response Suggestion for Smart Reply
arXiv (Cornell University)Published 1 May 2017Open access
Matthew Henderson, Rami Al‐Rfou, Brian Strope, Yun-Hsuan Sung, Lukács László, Ruiqi Guo
Citations218
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A computationally efficient machine-learned method for natural language response suggestion using feed-forward neural networks using n-gram embedding features that achieves the same quality at a small fraction of the computational requirements and latency.
Abstract
This paper presents a computationally efficient machine-learned method for natural language response suggestion. Feed-forward neural networks using n-gram embedding features encode messages into vectors which are optimized to give message-response pairs a high dot-product value. An optimized search finds response suggestions. The method is evaluated in a large-scale commercial e-mail application, Inbox by Gmail. Compared to a sequence-to-sequence approach, the new system achieves the same quality at a small fraction of the computational requirements and latency.
Keywords
Computer Science
Neural ComputationLong Short-Term Memory
98,079 Citations1997Sepp Hochreiter, Jürgen Schmidhuber
A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
Glove: Global Vectors for Word Representation
33,769 Citations2014Jeffrey Pennington, Richard Socher +1 more
A new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods and produces a vector space with meaningful substructure.
arXiv (Cornell University)Efficient Estimation of Word Representations in Vector Space
11,710 Citations2013Tomáš Mikolov, Kai Chen +2 more
arXiv (Cornell University)Sequence to Sequence Learning with Neural Networks
3,526 Citations2014Ilya Sutskever, Oriol Vinyals +1 more
This paper presents a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure, and finds that reversing the order of the words in all source sentences improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier.
IEEE Transactions on Pattern Analysis and Machine IntelligenceProduct Quantization for Nearest Neighbor Search
2,885 Citations2010H. Jegou, Matthijs Douze +1 more
This paper introduces a product quantization-based approach for approximate nearest neighbor search to decompose the space into a Cartesian product of low-dimensional subspaces and to quantize each subspace separately.
Learning deep structured semantic models for web search using clickthrough data
2,044 Citations2013Po-Sen Huang, Xiaodong He +4 more
A series of new latent semantic models with a deep structure that project queries and documents into a common low-dimensional space where the relevance of a document given a query is readily computed as the distance between them are developed.
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models
1,725 Citations2016Iulian Vlad Serban, Alessandro Sordoni +3 more
arXiv (Cornell University)A Neural Conversational Model
1,500 Citations2015Oriol Vinyals, Quoc V. Le
A simple approach to conversational modeling which uses the recently proposed sequence to sequence framework, and is able to extract knowledge from both a domain specific dataset, and from a large, noisy, and general domain dataset of movie subtitles.
Transactions of the Association for Computational LinguisticsABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs
924 Citations2016Wenpeng Yin, Hinrich Schütze +2 more
This work presents a general Attention Based Convolutional Neural Network (ABCNN) for modeling a pair of sentences and proposes three attention schemes that integrate mutual influence between sentences into CNNs; thus, the representation of each sentence takes into consideration its counterpart.
Choice Reviews OnlineHow to create a mind: the secret of human thought revealed
419 Citations2013
Ray Kurzweil presents a provocative exploration of the most important project in the human-machine civilization: reverse-engineering the brain to understand precisely how it functions and using that knowledge to create even more intelligent machines.
Multi-Perspective Sentence Similarity Modeling with Convolutional Neural Networks
397 Citations2015Hua He, Kevin Gimpel +1 more
This work proposes a model for comparing sentences that uses a multiplicity of perspectives, first model each sentence using a convolutional neural network that extracts features at multiple levels of granularity and uses multiple types of pooling.
Evaluation of spoken language systems
397 Citations1990P. Price
This paper will address the emerging standards for evaluation of spoken language systems with quantifiable measures essential for comparing results and assessing claims.
Word-Based Dialog State Tracking with Recurrent Neural Networks
366 Citations2014Matthew Henderson, Blaise Thomson +1 more
A new wordbased tracking method which maps directly from the speech recognition results to the dialog state without using an explicit semantic decoder is presented, based on a recurrent neural network structure which is capable of generalising to unseen dialog state hypotheses, and which requires very little feature engineering.
Recurrent neural networks for language understanding
324 Citations2013Kaisheng Yao, Geoffrey Zweig +3 more
This paper modify the architecture to perform Language Understanding, and advance the state-of-the-art for the widely used ATIS dataset.
Cartesian K-Means
309 Citations2013Mohammad Norouzi, David J. Fleet
New models with a compositional parameterization of cluster centers are developed, so representational capacity increases super-linearly in the number of parameters, allowing one to effectively quantize data using billions or trillions of centers.
arXiv (Cornell University)Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)
255 Citations2014Anshumali Shrivastava, Ping Li
arXiv (Cornell University)A Network-based End-to-End Trainable Task-oriented Dialogue System
170 Citations2016Tsung-Hsien Wen, David Vandyke +6 more
Learning Binary Codes for Maximum Inner Product Search
151 Citations2015Fumin Shen, Wei Liu +3 more
This paper investigates learning binary codes to exclusively handle the MIPS problem, and proposes an asymmetric binary code learning framework based on inner product fitting, dubbed Asymmetric Inner-product Binary Coding (AIBC), which is evaluated on several large-scale image datasets.
arXiv (Cornell University)Smart Reply: Automated Response Suggestion for Email
141 Citations2016Anjuli Kannan, Karol Kurach +9 more
Talking to machines (statistically speaking)
138 Citations2002Steve Young
This paper reviews spoken dialogue systems from a statistical modelling perspective and samples of existing work are reviewed, including dialogue control and optimisation, semantic interpretation, goal detection, natural language generation and synthesis.
arXiv (Cornell University)End-to-end LSTM-based dialog control optimized with supervised and reinforcement learning
122 Citations2016J. D. Williams, Geoffrey Zweig
The main component of the model is a recurrent neural network (an LSTM), which maps from raw dialog history directly to a distribution over system actions, which relieves the system developer of much of the manual feature engineering of dialog state.
arXiv (Cornell University)Conversational Contextual Cues: The Case of Personalization and History for Response Ranking
57 Citations2016Rami Al‐Rfou, Marc Pickett +4 more
This work evaluates its models on the task of predicting the next response in a conversation, and finds that modeling both context and participants improves prediction accuracy.
arXiv (Cornell University)Swivel: Improving Embeddings by Noticing What's Missing
56 Citations2016Noam Shazeer, Ryan Doherty +2 more
Submatrix-wise Vector Embedding Learner (Swivel), a method for generating low-dimensional feature embeddings from a feature co-occurrence matrix, performs approximate factorization of the point-wise mutual information matrix via stochastic gradient descent.
arXiv (Cornell University)Quantization based Fast Inner Product Search
50 Citations2015Ruiqi Guo, Sanjiv Kumar +2 more
Experimental results on a variety of datasets including those arising from deep neural networks show that the proposed approach significantly outperforms the existing state-of-the-art MIPS.
arXiv (Cornell University)Clustering is Efficient for Approximate Maximum Inner Product Search
26 Citations2015Alex Auvolat, Sarath Chandar +3 more
This paper proposes to train a spherical k-means, after having reduced the MIPS problem to a Maximum Cosine Similarity Search (MCSS), and shows that this simple approach yields much higher speedups, for the same retrieval precision, than current state-of-the-art hashing-based and tree-based methods.