A Latent Variable Model Approach to PMI-based Word Embeddings
Transactions of the Association for Computational LinguisticsPublished 1 December 2016Open access
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, Andrej Risteski
Citations296
SJR quartileQ1
SJR score1.82
SNIP4.13
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A new generative model is proposed, a dynamic version of the log-linear topic model of Mnih and Hinton (2007) to use the prior to compute closed form expressions for word statistics, and it is shown that latent word vectors are fairly uniformly dispersed in space.
Abstract
Semantic word embeddings represent the meaning of a word via a vector, and are created by diverse methods. Many use nonlinear operations on co-occurrence statistics, and have hand-tuned hyperparameters and reweighting methods. This paper proposes a new generative model, a dynamic version of the log-linear topic model of Mnih and Hinton (2007). The methodological novelty is to use the prior to compute closed form expressions for word statistics. This provides a theoretical justification for nonlinear models like PMI, word2vec, and GloVe, as well as some hyperparameter choices. It also helps explain why low-dimensional semantic embeddings contain linear algebraic structure that allows solution of word analogies, as shown by Mikolov et al. (2013a) and many subsequent papers. Experimental support is provided for the generative model assumptions, the most important of which is that latent word vectors are fairly uniformly dispersed in space.
Keywords
Computer Science
Glove: Global Vectors for Word Representation
33,769 Citations2014Jeffrey Pennington, Richard Socher +1 more
A new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods and produces a vector space with meaningful substructure.
NatureLearning representations by back-propagating errors
30,885 Citations1986David E. Rumelhart, Geoffrey E. Hinton +1 more
Back-propagation repeatedly adjusts the weights of the connections in the network so as to minimize a measure of the difference between the actual output vector of the net and the desired output vector, which helps to represent important features of the task domain.
Journal of Political EconomyThe Pricing of Options and Corporate Liabilities
29,429 Citations1973Fischer Black, Myron S. Scholes
arXiv (Cornell University)Distributed Representations of Words and Phrases and their Compositionality
18,086 Citations2013Tomáš Mikolov, Ilya Sutskever +3 more
This paper presents a simple method for finding phrases in text, and shows that learning good vector representations for millions of phrases is possible and describes a simple alternative to the hierarchical softmax called negative sampling.
arXiv (Cornell University)Efficient Estimation of Word Representations in Vector Space
18,050 Citations2013Tomáš Mikolov, Kai Chen +2 more
Two novel model architectures for computing continuous vector representations of words from very large data sets are proposed and it is shown that these vectors provide state-of-the-art performance on the authors' test set for measuring syntactic and semantic word similarities.
Journal of the American Society for Information ScienceIndexing by latent semantic analysis
12,677 Citations1990Scott Deerwester, Susan Dumais +3 more
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
8,612 Citations2011John C. Duchi, Elad Hazan +1 more
Communications of the ACMProbabilistic topic models
5,549 Citations2012David M. Blei
Surveying a suite of algorithms that offer a solution to managing large document archives suggests they are well-suited to handle large amounts of data.
A unified architecture for natural language processing
5,204 Citations2008Ronan Collobert, Jason Weston
This work describes a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense using a language model.
Word association norms, mutual information, and lexicography
3,674 Citations1990Kenneth Church, Patrick Hanks
Learning Word Vectors for Sentiment Analysis
3,299 Citations2011Andrew L. Maas, Raymond E. Daly +4 more
This work presents a model that uses a mix of unsupervised and supervised techniques to learn word vectors capturing semantic term--document information as well as rich sentiment content, and finds it out-performs several previously introduced methods for sentiment classification.
Journal of Artificial Intelligence ResearchFrom Frequency to Meaning: Vector Space Models of Semantics
2,883 Citations2010Peter D. Turney, Patrick Pantel
The goal in this survey is to show the breadth of applications of VSMs for semantics, to provide a new perspective on VSMs, and to provide pointers into the literature for those who are less familiar with the field.
Linguistic Regularities in Continuous Space Word Representations
2,882 Citations2013Tomáš Mikolov, Wen-tau Yih +1 more
The vector-space word representations that are implicitly learned by the input-layer weights are found to be surprisingly good at capturing syntactic and semantic regularities in language, and that each relationship is characterized by a relation-specific vector offset.
Dynamic topic models
2,322 Citations2006David M. Blei, John Lafferty
A family of probabilistic time series models is developed to analyze the time evolution of topics in large document collections, and dynamic topic models provide a qualitative window into the contents of a large document collection.
arXiv (Cornell University)Probabilistic Latent Semantic Analysis
2,092 Citations2013Thomas Hofmann
This work proposes a widely applicable generalization of maximum likelihood model fitting by tempered EM, based on a mixture decomposition derived from a latent class model which results in a more principled approach which has a solid foundation in statistics.
Neural Information Processing SystemsNeural Word Embedding as Implicit Matrix Factorization
1,592 Citations2014Omer Levy, Yoav Goldberg
It is shown that using a sparse Shifted Positive PMI word-context matrix to represent words improves results on two word similarity tasks and one of two analogy tasks, and conjecture that this stems from the weighted nature of SGNS's factorization.
Latent semantic indexing
870 Citations1998Christos H. Papadimitriou, Hisao Tamaki +2 more
It is proved that, under certain conditions, LSI does succeed in capturing the underlying semantics of the corpus and achieves improved retrieval performance, and the technique of random projection is proposed as a way of speeding up LSI.
Linguistic Regularities in Sparse and Explicit Word Representations
641 Citations2014Omer Levy, Yoav Goldberg
It is demonstrated that analogy recovery is not restricted to neural word embeddings, and that a similar amount of relational similarities can be recovered from traditional distributional word representations.
Three new graphical models for statistical language modelling
582 Citations2007Andriy Mnih, Geoffrey E. Hinton
It is shown how real-valued distributed representations for words can be learned at the same time as learning a large set of stochastic binary hidden features that are used to predict the distributed representation of the next word from previous distributed representations.
Studies in fuzziness and soft computingNeural Probabilistic Language Models
491 Citations2006Yoshua Bengio, Holger Schwenk +3 more
This work proposes to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences, and incorporates this new language model into a state-of-the-art speech recognizer of conversational speech.
Journal of Computer and System SciencesA spectral algorithm for learning Hidden Markov Models
283 Citations2012Daniel Hsu, Sham M. Kakade +1 more
It is proved that under a natural separation condition (bounds on the smallest singular value of the HMM parameters), there is an efficient and provably correct algorithm for learning HMMs.
Euclidean Embedding of Co-occurrence Data
177 Citations2007Amir Globerson, Gal Chechik +2 more
Transactions of the Association for Computational LinguisticsWord Embeddings as Metric Recovery in Semantic Spaces
73 Citations2016Tatsunori Hashimoto, David Alvarez-Melis +1 more
A simple, principled, direct metric recovery algorithm is proposed that performs on par with the state-of-the-art word embedding and manifold learning methods and is complemented by constructing two new inductive reasoning datasets and demonstrating that word embeddings can be used to solve them.
When and why are log-linear models self-normalizing?
40 Citations2015Jacob Andreas, Dan Klein
A novel variant of an established procedure for training self-normalized models is described, which avoids computing normalizers for most training examples, and decreases training time by as much as factor of ten while preserving model quality.
Sphere Embedding: An Application to Part-of-Speech Induction
27 Citations2010Yariv Maron, Elie Bienenstock +1 more
An algorithm for the Euclidean embedding of large sets of categorical data based on co-occurrence statistics using the CODE model but constrain the embedding to lie on a high-dimensional unit sphere to allow for efficient optimization.
International Conference on Machine LearningA Linear Dynamical System Model for Text
10 Citations2015David Belanger, Sham M. Kakade
This work provides an inference procedure for continuous states, where words' representations are given by the posterior mean of a linear dynamical system, and demonstrates that using the parameters of the model to initialize a non-linear recurrent neural network language model reduces its training time by a day and yields lower perplexity.