Multi-Relational Latent Semantic Analysis
Published 1 January 2013Open access
Kai-Wei Chang, Wen-tau Yih, Christopher Meek
Citations59
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
It is demonstrated that by integrating multiple relations from both homogeneous and heterogeneous information sources, MRLSA achieves state-of-the-art performance on existing benchmark datasets for two relations, antonymy and is-a.
Abstract
We present Multi-Relational Latent Semantic Analysis (MRLSA) which generalizes Latent Semantic Analysis (LSA).MRLSA provides an elegant approach to combining multiple relations between words by constructing a 3-way tensor.Similar to LSA, a lowrank approximation of the tensor is derived using a tensor decomposition.Each word in the vocabulary is thus represented by a vector in the latent semantic space and each relation is captured by a latent square matrix.The degree of two words having a specific relation can then be measured through simple linear algebraic operations.We demonstrate that by integrating multiple relations from both homogeneous and heterogeneous information sources, MRLSA achieves stateof-the-art performance on existing benchmark datasets for two relations, antonymy and is-a.
Keywords
Computer Science
Journal of Machine Learning ResearchLatent dirichlet allocation
27,049 Citations2003David M. Blei, Andrew Y. Ng +1 more
Journal of the American Society for Information ScienceIndexing by latent semantic analysis
12,677 Citations1990Scott Deerwester, Susan Dumais +3 more
SIAM ReviewTensor Decompositions and Applications
10,455 Citations2009Tamara G. Kolda, Brett W. Bader
This survey provides an overview of higher-order tensor decompositions, their applications, and available software.
Communications of the ACMA vector space model for automatic indexing
7,434 Citations1975Gerard Salton, Anita M.-Y. Wong +1 more
An approach based on space density computations is used to choose an optimum indexing vocabulary for a collection of documents, demonstating the usefulness of the model.
PsychometrikaSome Mathematical Notes on Three-Mode Factor Analysis
4,269 Citations1966Ledyard R Tucker
The model for three-mode factor analysis is discussed in terms of newer applications of mathematical processes including a type of matrix process termed the Kronecker product and the definition of combination variables.
Journal of Artificial Intelligence ResearchFrom Frequency to Meaning: Vector Space Models of Semantics
2,883 Citations2010Peter D. Turney, Patrick Pantel
The goal in this survey is to show the breadth of applications of VSMs for semantics, to provide a new perspective on VSMs, and to provide pointers into the literature for those who are less familiar with the field.
Linguistic Regularities in Continuous Space Word Representations
2,882 Citations2013Tomáš Mikolov, Wen-tau Yih +1 more
The vector-space word representations that are implicitly learned by the input-layer weights are found to be surprisingly good at capturing syntactic and semantic regularities in language, and that each relationship is characterized by a relation-specific vector offset.
arXiv (Cornell University)Probabilistic Latent Semantic Analysis
2,092 Citations2013Thomas Hofmann
This work proposes a widely applicable generalization of maximum likelihood model fitting by tempered EM, based on a mixture decomposition derived from a latent class model which results in a more principled approach which has a solid foundation in statistics.
Document clustering based on non-negative matrix factorization
1,847 Citations2003Wei Xu, Xin Liu +1 more
This paper proposes a novel document clustering method based on the non-negative factorization of the term-document matrix of the given document corpus that surpasses the latent semantic indexing and the spectral clustering methods not only in the easy and reliable derivation of document clustered results, but also in document clusters accuracies.
Parsing Natural Scenes and Natural Language with Recursive Neural Networks
1,209 Citations2011Richard Socher, Cliff Chiung-Yu Lin +2 more
A max-margin structure prediction architecture based on recursive neural networks that can successfully recover such structure both in complex scene images as well as sentences is introduced.
A study on similarity and relatedness using distributional and WordNet-based approaches
880 Citations2009Eneko Agirre, Enrique Alfonseca +4 more
This paper presents and compares WordNet-based and distributional similarity approaches, and pioneer cross-lingual similarity, showing that the methods are easily adapted for a cross-lingsual task with minor losses.
Parsing with Compositional Vector Grammars
832 Citations2013Richard Socher, John Bauer +2 more
A Compositional Vector Grammar (CVG), which combines PCFGs with a syntactically untied recursive neural network that learns syntactico-semantic, compositional vector representations and improves performance on the types of ambiguities that require semantic information such as PP attachments.
CORE Scholar (Wright State University)Relation Extraction with Matrix Factorization and Universal Schemas
587 Citations2013Sebastian Riedel, Limin Yao +2 more
Scalable Tensor Decompositions for Multi-aspect Data Mining
361 Citations2008Tamara G. Kolda, Jimeng Sun
Memory-Efficient Tucker (MET) is proposed, which achieves over 1000X space reduction without sacrificing speed; it also allows us to work with much larger tensors that were too big to handle before.
Multi-Prototype Vector-Space Models of Word Meaning
355 Citations2010Joseph Reisinger, Raymond J. Mooney
Experimental comparisons to human judgements of semantic similarity for both isolated words as well as words in sentential contexts demonstrate the superiority of this approach over both prototype and exemplar based vector-space models.
Empirical Methods in Natural Language ProcessingA Topic Model for Word Sense Disambiguation
224 Citations2007Jordan Boyd‐Graber, David M. Blei +1 more
A probabilistic posterior inference algorithm for simultaneously disambiguating a corpus and learning the domains in which to consider each word is developed.
Learning Discriminative Projections for Text Similarity Measures
223 Citations2011Wen-tau Yih, Kristina Toutanova +2 more
A novel discriminative training method that projects the raw term vectors into a common, low-dimensional vector space, which not only outperforms existing state-of-the-art approaches, but also achieves high accuracy at low dimensions and is thus more efficient.
National Research Council Canada (Government of Canada)SemEval-2012 Task 2 : measuring degrees of relational similarity
191 Citations2012David Jurgens, Saif M. Mohammad +2 more
A new SemEval task based on identifying the degree of prototypicality for instances within a given class is presented, and the first dataset of graded relational similarity ratings across 79 relation categories is assembled.
A uniform approach to analogies, synonyms, antonyms, and associations
179 Citations2008Peter D. Turney
A supervised corpus-based machine learning algorithm is introduced for classifying analogous word pairs and it is shown that it can solve multiple-choice SAT analogy questions, TOEFL synonyms questions, ESL synonym-antonym questions, and similar-associated-both questions from cognitive psychology.
Automatic Cross-Language Retrieval Using Latent Semantic Indexing
177 Citations1997Susan Dumais, Todd A. Letsche +2 more
A method for fully automated cross-language document retrieval in which no query translation is required and this automatic method performs comparably to a retrieval method based on machine translation (MT-LSI).
The Psychology of learning and motivation/The psychology of learning and motivationOn the computational basis of learning and cognition: Arguments from LSA
167 Citations2002Thomas K. Landauer
The principal goal of this chapter has been to suggest that high-dimensional vector space computations based on empirical associations among very large numbers of components could be a close model of a fundamental computational basis of most learning in both verbal and perceptual domains.
OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information)MATLAB Tensor Toolbox
154 Citations2006Tamara G. Kolda, Brett W. Bader
The tensor class extends the functionality of MATLAB's multidimensional arrays by supporting additional operations such as tensor multiplication and has also added support for sparse tensor, tensors in Kruskal or Tucker format, and tensors stored as matrices.
Modeling Sentences in the Latent Space
119 Citations2012Weiwei Guo, Mona Diab
By carefully handling words that are not in the sentences (missing words), a reliable latent variable model on sentences is trained and a new evaluation framework for sentence similarity is proposed: Concept Definition Retrieval.
Learning Human-like Knowledge by Singular Value Decomposition: A Progress Report
114 Citations1997Thomas K. Landauer, Darrell Laham +1 more
Singular value decomposition was used to learn and represent relations among very large numbers of words and natural text passages and the result was 100-350 dimensional "semantic spaces" in which any trained or newly added word or passage could be represented as a vector, and similarities were measured by the cosine of the contained angle between vectors.
Empirical Methods in Natural Language ProcessingTranslingual Document Representations from Discriminative Projections
100 Citations2010John Platt, Kristina Toutanova +1 more
This work uses discriminative training to create a projection of documents from multiple languages into a single translingual vector space and evaluates these algorithms on two tasks: parallel document retrieval for Wikipedia and Europarl documents, and cross-lingual text classification on Reuters.
Computing word-pair antonymy
93 Citations2008Saif M. Mohammad, Bonnie J. Dorr +1 more
A new automatic and empirical measure of antonymy that combines corpus statistics with the structure of a published thesaurus is presented, obtaining a precision of over 80%.
Polarity Inducing Latent Semantic Analysis
69 Citations2012Wen-tau Yih, Geoffrey Zweig +1 more
The key contribution of this work is to show how to assign signs to the entries in the co-occurrence matrix on which LSA operates, so as to induce a subspace with the desired property.
A tensor-based factorization model of semantic compositionality
57 Citations2016Tim Van de Cruys, Thierry Poibeau +2 more
A novel method is presented for the computation of compositionality within a distributional framework that is modeled as a multi-way interaction between latent factors, which are automatically constructed from corpus data.
Combining Heterogeneous Models for Measuring Relational Similarity
57 Citations2013Alisa Zhila, Wen-tau Yih +3 more
The overall system consists of two novel general-purpose relational similarity models and three specific word relation models that outperforms the previous best system substantially and achieves a 54.1% relative increase in Spearman's rank correlation.
North American Chapter of the Association for Computational LinguisticsMeasuring Word Relatedness Using Heterogeneous Vector Space Models
44 Citations2012Wen-tau Yih, Vahed Qazvinian
This model consists of a committee of vector space models built on a text corpus, Web search results and theauruses, and measures the semantic word relatedness using the averaged cosine similarity scores.
Edinburgh Research Explorer (University of Edinburgh)Approximate PCFG Parsing Using Tensor Decomposition
25 Citations2013Shay B. Cohen, Giorgio Satta +1 more
An approximation algorithm for PCFG parsing is provided, which asymptotically improves time complexity with respect to the input grammar size, and upper bounds on the approximation quality are proved.
Joint Conference on Lexical and Computational SemanticsUTD: Determining Relational Similarity Using Lexical Patterns
18 Citations2012Bryan Rink, Sanda M. Harabagiu
A class of NAD-dependent active-site SARM1 inhibitors that function by intercepting NAD hydrolysis and undergoing covalent conjugation with the reaction product adenosine diphosphate ribose are described, validating this mode of inhibition as a viable therapeutic strategy.
Latent Semantic Tensor Indexing for Community-based Question Answering
11 Citations2013Xipeng Qiu, Le Tian +1 more
This paper proposes a unified question retrieval model based on latent semantic indexing with tensor analysis, which can capture word associations among different parts of CQA triples simultaneously and can reduce lexical chasm of question retrieval.
Improving Lexical Semantics for Sentential Semantics: Modeling Selectional Preference and Similar Words in a Latent Variable Model
8 Citations2013Weiwei Guo, Mona Diab
This work incorporates both corpus-based and knowledge-based lexical semantics into a latent variable model, which shows state-of-the-art performance among unsupervised systems on two SS datasets.