Learning Multi-Relational Semantics Using Neural-Embedding Models
arXiv (Cornell University)Published 14 November 2014Open access
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, Li Deng
Citations21
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The results show several interesting findings, enabling the design of a simple embedding model that achieves the new state-of-the-art performance on a popular knowledge base completion task evaluated on Freebase.
Abstract
In this paper we present a unified framework for modeling multi-relational representations, scoring, and learning, and conduct an empirical study of several recent multi-relational embedding models under the framework. We investigate the different choices of relation operators based on linear and bilinear transformations, and also the effects of entity representations by incorporating unsupervised vectors pre-trained on extra textual resources. Our results show several interesting findings, enabling the design of a simple embedding model that achieves the new state-of-the-art performance on a popular knowledge base completion task evaluated on Freebase.
Keywords
Computer Science
arXiv (Cornell University)Distributed Representations of Words and Phrases and their Compositionality
18,086 Citations2013Tomáš Mikolov, Ilya Sutskever +3 more
This paper presents a simple method for finding phrases in text, and shows that learning good vector representations for millions of phrases is possible and describes a simple alternative to the hierarchical softmax called negative sampling.
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
8,612 Citations2011John C. Duchi, Elad Hazan +1 more
Translating embeddings for modeling multi-relational data
5,178 Citations2015Antoine Bordes, Nicolas Usunier +2 more
TransE is proposed, a method which models relationships by interpreting them as translations operating on the low-dimensional embeddings of the entities, which proves to be powerful since extensive experiments show that TransE significantly outperforms state-of-the-art methods in link prediction on two knowledge bases.
Machine LearningMarkov logic networks
2,678 Citations2006Matthew Richardson, Pedro Domingos
Experiments with a real-world database and knowledge base in a university domain illustrate the promise of this approach to combining first-order logic and probabilistic graphical models in a single representation.
Learning deep structured semantic models for web search using clickthrough data
2,044 Citations2013Po-Sen Huang, Xiaodong He +4 more
A series of new latent semantic models with a deep structure that project queries and documents into a common low-dimensional space where the relevance of a document given a query is readily computed as the distance between them are developed.
Deep Neural Networks for Acoustic Modeling in Speech Recognition
1,899 Citations2012Geoffrey E. Hinton, Li Deng +9 more
This paper provides an overview of this progress and repres nts the shared views of four research groups who have had recent successes in using deep neural networks for a coustic modeling in speech recognition.
Reasoning With Neural Tensor Networks for Knowledge Base Completion
1,669 Citations2013Richard Socher, Danqi Chen +2 more
An expressive neural tensor network suitable for reasoning over relationships between two entities given a subset of the knowledge base is introduced and performance can be improved when entities are represented as an average of their constituting word vectors.
A Three-Way Model for Collective Learning on Multi-Relational Data
1,647 Citations2011Maximilian Nickel, Volker Tresp +1 more
This work presents a novel approach to relational learning based on the factorization of a three-way tensor that is able to perform collective learning via the latent components of the model and provide an efficient algorithm to compute the factorizations.
Knowledge vault
1,558 Citations2014Xin Luna Dong, Evgeniy Gabrilovich +7 more
The Knowledge Vault is a Web-scale probabilistic knowledge base that combines extractions from Web content (obtained via analysis of text, tabular data, page structure, and human annotations) with prior knowledge derived from existing knowledge repositories that computes calibrated probabilities of fact correctness.
Relational learning via collective matrix factorization
1,235 Citations2008Ajit Singh, Geoffrey J. Gordon
This model generalizes several existing matrix factorization methods, and therefore yields new large-scale optimization algorithms for these problems, which can handle any pairwise relational schema and a wide variety of error models.
New types of deep neural network learning for speech recognition and related applications: an overview
1,194 Citations2013Li Deng, Geoffrey E. Hinton +1 more
An overview of the invited and contributed papers presented at the special session at ICASSP-2013, entitled “New Types of Deep Neural Network Learning for Speech Recognition and Related Applications,” as organized by the authors is provided.
Learning Structured Embeddings of Knowledge Bases
861 Citations2011Antoine Bordes, Jason Weston +2 more
Learning semantic representations using convolutional neural networks for web search
691 Citations2014Yelong Shen, Xiaodong He +3 more
This paper presents a series of new latent semantic models based on a convolutional neural network to learn low-dimensional semantic vectors for search queries and Web documents that significantly outperforms other se-mantic models in retrieval performance.
Machine LearningA semantic matching energy function for learning with multi-relational data
687 Citations2013Antoine Bordes, Xavier Glorot +2 more
A new neural network architecture designed to embed multi-relational graphs into a flexible continuous vector space in which the original data is kept and enhanced, demonstrating that it can scale up to tens of thousands of nodes and thousands of types of relation.
A Latent Semantic Model with Convolutional-Pooling Structure for Information Retrieval
682 Citations2014Yelong Shen, Xiaodong He +3 more
A new latent semantic model that incorporates a convolutional-pooling structure over word sequences to learn low-dimensional, semantic vector representations for search queries and Web documents is proposed.
Learning systems of concepts with an infinite relational model
552 Citations2006Charles Kemp, Joshua B. Tenenbaum +3 more
A nonparametric Bayesian model is presented that discovers systems of related concepts and applies the approach to four problems: clustering objects and features, learning ontologies, discovering kinship systems, and discovering structure in political data.
FigshareRandom Walk Inference and Learning in A Large Scale Knowledge Base
521 Citations2018Ni Lao, Tom M. Mitchell +1 more
It is shown that a soft inference procedure based on a combination of constrained, weighted, random walks through the knowledge base graph can be used to reliably infer new beliefs for theknowledge base.
Semantic Parsing for Single-Relation Question Answering
423 Citations2014Wen-tau Yih, Xiaodong He +1 more
A semantic parsing framework based on semantic similarity for open domain question answering (QA) that achieves higher precision across different recall points compared to the previous approach, and can improve F1 by 7 points.
Factorizing YAGO
376 Citations2012Maximilian Nickel, Volker Tresp +1 more
This work presents an efficient approach to relational learning on LOD data, based on the factorization of a sparse tensor that scales to data consisting of millions of entities, hundreds of relations and billions of known facts, and shows how ontological knowledge can be incorporated in the factorizations to improve learning results and how computation can be distributed across multiple nodes.
HAL (Le Centre pour la Communication Scientifique Directe)A latent factor model for highly multi-relational data
366 Citations2012Rodolphe Jenatton, Nicolas Le Roux +2 more
This paper proposes a method for modeling large multi-relational datasets, with possibly thousands of relations, based on a bilinear structure, which captures various orders of interaction of the data and also shares sparse latent factors across different relations.
DSpace@MIT (Massachusetts Institute of Technology)Modelling Relational Data using Bayesian Clustered Tensor Factorization
241 Citations2009Ilya Sutskever, Joshua B. Tenenbaum +1 more
The Bayesian Clustered Tensor Factorization (BCTF) model is introduced, which embeds a factorized representation of relations in a nonparametric Bayesian clustering framework that is fully Bayesian but scales well to large data sets.
Typed Tensor Decomposition of Knowledge Bases for Relation Extraction
191 Citations2014Kai-Wei Chang, Wen-tau Yih +2 more
A tensor decomposition approach for knowledge base embedding that is highly scalable, and is especially suitable for relation extraction by leveraging relational domain knowledge about entity type information, which is significantly faster than previous approaches and better able to discover new relations missing from the database.
Modeling Interestingness with Deep Neural Networks
175 Citations2014Jianfeng Gao, Patrick Pantel +3 more
The results on large-scale, real-world datasets show that the semantics of documents are important for modeling interest-ingness and that the DSSM leads to significant quality improvement on both tasks, outperforming not only the classic document models that do not use semantics but also state-of-the-art topic models.
IEEE Transactions on Pattern Analysis and Machine IntelligenceTensor Deep Stacking Networks
158 Citations2013Brian Hutchinson, Li Deng +1 more
A sufficient depth of the T-DSN, a symmetry in the two hidden layers structure in each T- DSN block, the model parameter learning algorithm, and a softmax layer on top of T-DsN are shown to have all contributed to the low error rates observed in the experiments for all three tasks.
IEEE Transactions on Knowledge and Data EngineeringLearning distributed representations of concepts using linear relational embedding
130 Citations2001Alberto Paccanaro, Geoffrey E. Hinton
IEEE Transactions on Audio Speech and Language ProcessingThe Deep Tensor Neural Network With Applications to Large Vocabulary Speech Recognition
113 Citations2012Dong Yu, Li Deng +1 more
Evaluation on Switchboard tasks indicates that DTNNs can outperform the already high-performing DNNs with 4-5% and 3% relative word error reduction, respectively, using 30-hr and 309-hr training sets.
ACM SIGKDD Explorations NewsletterProspects and challenges for multi-relational data mining
74 Citations2003Pedro Domingos
This short paper argues that multi-relational data mining has a key role to play in the growth of KDD, and briefly surveys some of the main drivers, research problems, and opportunities in this emerging field.
Learning with Recursive Perceptual Representations
59 Citations2012Oriol Vinyals, Yangqing Jia +2 more
A deep non-linear classifier whose layers are SVMs and which incorporates random projection as its core stacking element, which scales as linear SVMs, does not rely on any kernel computations or nonconvex optimization, and exhibits better generalization ability than kernel-based SVMs.
arXiv (Cornell University)A Semantic Matching Energy Function for Learning with Multi-relational Data
32 Citations2013Xavier Glorot, Antoine Bordes +2 more