Linguistic Regularities in Continuous Space Word Representations
Published 27 May 2013
Tomáš Mikolov, Wen-tau Yih, Geoffrey Zweig
Citations2,882
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The vector-space word representations that are implicitly learned by the input-layer weights are found to be surprisingly good at capturing syntactic and semantic regularities in language, and that each relationship is characterized by a relation-specific vector offset.
Abstract
Continuous space language models have recently demonstrated outstanding results across a variety of tasks. In this paper, we examine the vector-space word representations that are implicitly learned by the input-layer weights. We find that these representations are surprisingly good at capturing syntactic and semantic regularities in language, and that each relationship is characterized by a relation-specific vector offset. This allows vector-oriented reasoning based on the offsets between words. For example, the male/female relationship is automatically learned, and with the induced vector representations, “King-Man + Woman ” results in a vector very close to “Queen. ” We demonstrate that the word vectors capture syntactic regularities by means of syntactic analogy questions (provided with this paper), and are able to correctly answer almost 40 % of the questions. We demonstrate that the word vectors capture semantic regularities by using the vector offset method to answer SemEval-2012 Task 2 questions. Remarkably, this method outperforms the best previous systems. 1
Keywords
Computer Science
ScienceReducing the Dimensionality of Data with Neural Networks
20,953 Citations2006Geoffrey E. Hinton, Ruslan Salakhutdinov
Journal of the American Society for Information ScienceIndexing by latent semantic analysis
12,677 Citations1990Scott Deerwester, Susan Dumais +3 more
arXiv (Cornell University)Efficient Estimation of Word Representations in Vector Space
11,710 Citations2013Tomáš Mikolov, Kai Chen +2 more
Building a Large Annotated Corpus of English: The Penn Treebank
7,528 Citations1993Mitchell P. Marcus
Recurrent neural network based language model
5,349 Citations2010Tomáš Mikolov, Martin Karafiát +3 more
Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model.
A unified architecture for natural language processing
5,204 Citations2008Ronan Collobert, Jason Weston
This work describes a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense using a language model.
Word Representations: A Simple and General Method for Semi-Supervised Learning
1,944 Citations2010Joseph Turian, Lev-Arie Ratinov +1 more
This work evaluates Brown clusters, Collobert and Weston (2008) embeddings, and HLBL (Mnih & Hinton, 2009) embeds of words on both NER and chunking, and finds that each of the three word representations improves the accuracy of these baselines.
Applied Physics Letters10.1162/153244303322533223
1,771 Citations2000
This work proposes to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences.
Machine LearningDistributed representations, simple recurrent networks, and grammatical structure
1,381 Citations1991Jeffrey L. Elman
Artificial IntelligenceRecursive distributed representations
866 Citations1990Jordan Pollack
This paper presents a connectionist architecture which automatically develops compact distributed representations for variable-sized recursive data structures, as well as efficient accessing mechanisms for them.
A Scalable Hierarchical Distributed Language Model
852 Citations2008Andriy Mnih, Geoffrey E. Hinton
A fast hierarchical language model along with a simple feature-based algorithm for automatic construction of word trees from the data are introduced and it is shown that the resulting models can outperform non-hierarchical neural models as well as the best n-gram models.
International Conference on Artificial Intelligence and StatisticsHierarchical Probabilistic Neural Network Language Model.
840 Citations2005Fréderic Morin, Yoshua Bengio
A point-of-care POC analyzer that can rapidly measure both medications used for treatment and illicit drugs in patient saliva and shows evidence of accuracy in measuring treatment medications and relapse to drug use is shown.
Learning distributed representations of concepts.
821 Citations1989Geoffrey E. Hinton
Strategies for training large scale neural network language models
512 Citations2011Tomáš Mikolov, Anoop Deoras +3 more
This work describes how to effectively train neural network based language models on large data sets and introduces hash-based implementation of a maximum entropy model, that can be trained as a part of the neural network model.
Computer Speech & LanguageContinuous space language models
503 Citations2006Holger Schwenk
Highly efficient learning algorithms are described that enable the use of training corpora of several hundred million words and it is shown that this approach can be incorporated into a large vocabulary continuous speech recognizer using a lattice rescoring framework at a very low additional processing time.
Studies in fuzziness and soft computingNeural Probabilistic Language Models
491 Citations2006Yoshua Bengio, Holger Schwenk +3 more
This work proposes to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences, and incorporates this new language model into a state-of-the-art speech recognizer of conversational speech.
Joint learning of words and meaning representations for open-text semantic parsing
332 Citations2012Antoine Bordes, Xavier Glorot +2 more
This work proposes a method that learns to assign MRs to a wide range of text thanks to a training scheme that combines learning from knowledge bases with learning from raw text.
National Research Council Canada (Government of Canada)SemEval-2012 Task 2 : measuring degrees of relational similarity
191 Citations2012David Jurgens, Saif M. Mohammad +2 more
A new SemEval task based on identifying the degree of prototypicality for instances within a given class is presented, and the first dataset of graded relational similarity ratings across 79 relation categories is assembled.
Structured Output Layer neural network language model
131 Citations2011Hai-Son Le, Ilya Oparin +3 more
A new neural network language model (NNLM) based on word clustering to structure the output vocabulary: Structured Output Layer NNLM, able to handle vocabularies of arbitrary size, hence dispensing with the design of short-lists that are commonly used in NNLMs.
Topics in Cognitive ScienceDiscovering Binary Codes for Documents by Learning Deep Generative Models
99 Citations2010Geoffrey E. Hinton, Ruslan Salakhutdinov
A deep generative model in which the lowest layer represents the word-count vector of a document and the top layer represents a learned binary code for that document is described, which allows more accurate and much faster retrieval than latent semantic analysis.
Joint Conference on Lexical and Computational SemanticsUTD: Determining Relational Similarity Using Lexical Patterns
18 Citations2012Bryan Rink, Sanda M. Harabagiu
A class of NAD-dependent active-site SARM1 inhibitors that function by intercepting NAD hydrolysis and undergoing covalent conjugation with the reaction product adenosine diphosphate ribose are described, validating this mode of inhibition as a viable therapeutic strategy.