Dict2vec : Learning Word Embeddings using Lexical Dictionaries
Published 1 January 2017Open access
Julien Tissier, Christopher Gravier, Amaury Habrard
Citations104
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Dict2vec builds new word pairs from dictionary entries so that semantically-related words are moved closer, and negative sampling filters out pairs whose words are unrelated in dictionaries.
Abstract
Learning word embeddings on large unlabeled corpus has been shown to be successful in improving many natural language tasks. The most efficient and popular approaches learn or retrofit such representations using additional external data. Resulting embeddings are generally better than their corpus-only counterparts, although such resources cover a fraction of words in the vocabulary. In this paper, we propose a new approach, Dict2vec, based on one of the largest yet refined datasource for describing words – natural language dictionaries. Dict2vec builds new word pairs from dictionary entries so that semantically-related words are moved closer, and negative sampling filters out pairs whose words are unrelated in dictionaries. We evaluate the word representations obtained using Dict2vec on eleven datasets for the word similarity task and on four datasets for a text classification task.
Keywords
Computer ScienceArts and Humanities
Glove: Global Vectors for Word Representation
33,769 Citations2014Jeffrey Pennington, Richard Socher +1 more
A new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods and produces a vector space with meaningful substructure.
arXiv (Cornell University)Efficient Estimation of Word Representations in Vector Space
18,050 Citations2013Tomáš Mikolov, Kai Chen +2 more
Two novel model architectures for computing continuous vector representations of words from very large data sets are proposed and it is shown that these vectors provide state-of-the-art performance on the authors' test set for measuring syntactic and semantic word similarities.
arXiv (Cornell University)Efficient Estimation of Word Representations in Vector Space
11,710 Citations2013Tomáš Mikolov, Kai Chen +2 more
Transactions of the Association for Computational LinguisticsEnriching Word Vectors with Subword Information
9,784 Citations2017Piotr Bojanowski, Édouard Grave +2 more
A new approach based on the skipgram model, where each word is represented as a bag of character n-grams, with words being represented as the sum of these representations, which achieves state-of-the-art performance on word similarity and analogy tasks.
Recurrent neural network based language model
5,349 Citations2010Tomáš Mikolov, Martin Karafiát +3 more
Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model.
A unified architecture for natural language processing
5,204 Citations2008Ronan Collobert, Jason Weston
This work describes a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense using a language model.
arXiv (Cornell University)Natural Language Processing (almost) from Scratch
5,172 Citations2011Ronan Collobert, Jason Weston +4 more
Lecture notes in computer scienceDBpedia: A Nucleus for a Web of Open Data
4,675 Citations2007Sören Auer, Christian Bizer +4 more
The extraction of the DBpedia datasets is described, and how the resulting information is published on the Web for human-andmachine-consumption and how DBpedia could serve as a nucleus for an emerging Web of open data.
Natural Language Processing (almost) from Scratch
3,987 Citations2011Ronan Collobert, Jason Weston +4 more
Language and Cognitive ProcessesContextual correlates of semantic similarity
1,584 Citations1991George A. Miller, Walter G. Charles
Communications of the ACMContextual correlates of synonymy
1,449 Citations1965Herbert Rubenstein, John B. Goodenough
The shapes of the functions indicate that similarity of context is reliable as criterion only for detecting pairs of words that are very similar in meaning.
Generating Text with Recurrent Neural Networks
1,166 Citations2011Ilya Sutskever, James Martens +1 more
The power of RNNs trained with the new Hessian-Free optimizer by applying them to character-level language modeling tasks is demonstrated, and a new RNN variant that uses multiplicative connections which allow the current input character to determine the transition matrix from one hidden state vector to the next is introduced.
Journal of Artificial Intelligence ResearchMultimodal Distributional Semantics
939 Citations2014Elia Bruni, Nam K. Tran +1 more
This work proposes a flexible architecture to integrate text- and image-based distributional information, and shows in a set of empirical tests that the integrated model is superior to the purely text-based approach, and it provides somewhat complementary semantic information with respect to the latter.
Placing search in context: the concept revisited.
931 Citations2002Lev Finkelstein, Evgeniy Gabrilovich +5 more
Placing search in context
861 Citations2001Lev Finkelstein, Evgeniy Gabrilovich +5 more
A new conceptual paradigm for performing search in context is presented, that largely automates the search process, providing even non-professional users with highly relevant results.
Conference on Computational Natural Language LearningBetter Word Representations with Recursive Neural Networks for Morphology
820 Citations2013Thang Luong, Richard Socher +1 more
This paper combines recursive neural networks, where each morpheme is a basic unit, with neural language models to consider contextual information in learning morphologicallyaware word representations and proposes a novel model capable of building representations for morphologically complex words from their morphemes.
Retrofitting Word Vectors to Semantic Lexicons
709 Citations2015Manaal Faruqui, Jesse Dodge +4 more
Fast and Robust Neural Network Joint Models for Statistical Machine Translation
489 Citations2014Jacob Devlin, Rabih Zbib +4 more
A novel formulation for a neural network joint model (NNJM), which augments the NNLM with a source context window, which is purely lexicalized and can be integrated into any MT decoder.
A word at a time
413 Citations2011Kira Radinsky, Eugene Agichtein +2 more
This paper proposes a new semantic relatedness model, Temporal Semantic Analysis (TSA), which captures this temporal information in word semantics as a vector of concepts over a corpus of temporally-ordered documents.
Knowledge Graph and Text Jointly Embedding
412 Citations2014Zhen Wang, Jianwen Zhang +2 more
Large scale experiments on Freebase and a Wikipedia/NY Times corpus show that jointly embedding brings promising improvement in the accuracy of predicting facts, compared to separately embedding knowledge graphs and text.
End-to-end learning of semantic role labeling using recurrent neural networks
366 Citations2015Jie Zhou, Wei Xu
This work proposes to use deep bi-directional recurrent network as an end-to-end system for SRL, which takes only original text information as input feature, without using any syntactic knowledge.
Improving Lexical Embeddings with Semantic Knowledge
286 Citations2014Mo Yu, Mark Dredze
This work proposes a new learning objective that incorporates both a neural language model objective (Mikolov et al., 2013) and prior knowledge from semantic resources to learn improved lexical semantic embeddings.
Embeddings for Word Sense Disambiguation: An Evaluation Study
257 Citations2016Ignacio Iacobacci, Mohammad Taher Pilehvar +1 more
This work proposes different methods through which word embeddings can be leveraged in a state-of-the-art supervised WSD system architecture, and performs a deep analysis of how different parameters affect performance.
Large-scale learning of word relatedness with constraints
243 Citations2012Guy Halawi, Gideon Dror +2 more
A large-scale data mining approach to learning word-word relatedness, where known pairs of related words impose constraints on the learning process, and learns for each word a low-dimensional representation, which strives to maximize the likelihood of a word given the contexts in which it appears.
Lecture notes in computer scienceKnowledge-Powered Deep Learning for Word Embedding
163 Citations2014Jiang Bian, Bin Gao +1 more
This study explores the capacity of leveraging morphological, syntactic, and semantic knowledge to achieve high-quality word embeddings, and explores these types of knowledge to define new basis for word representation, provide additional input information, and serve as auxiliary supervision in deep learning.
Not All Contexts Are Created Equal: Better Word Representations with Variable Attention
147 Citations2015Ling Wang, Yulia Tsvetkov +6 more
An extension to the bag-ofwords model for learning words representations that take into account both syntactic and semantic properties within language is introduced by employing an attention model that finds within the contextual words, the words that are relevant for each prediction.
Verb similarity on the taxonomy of WordNet
140 Citations2006Dongqiang Yang, David Powers
Two kinds of word similarity algorithms are introduced, SHE and RHE, to investigate the capability of WordNet in measuring verb similarity, in the absence of a standard verb set.
Specializing Word Embeddings for Similarity or Relatedness
138 Citations2015Douwe Kiela, Felix Hill +1 more
It is shown that using specialized spaces in NLP tasks and applications leads to clear improvements, for document classification and synonym selection, which rely on either similarity or relatedness but not both.
WikiReading: A Novel Large-scale Language Understanding Task over Wikipedia
130 Citations2016Daniel Hewlett, Alexandre Lacoste +6 more
This work presents WIKIREADING, a large-scale natural language understanding task and publicly-available dataset with 18 million instances, and compares various state-of-the-art DNNbased architectures for document classification, information extraction, and question answering.