Language Models as Knowledge Bases?
Published 1 January 2019Open access
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu
Citations1,692
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
An in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of state-of-the-art pretrained language models finds that BERT contains relational knowledge competitive with traditional NLP methods that have some access to oracle knowledge.
Abstract
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Keywords
Computer ScienceDecision Sciences
Neural ComputationLong Short-Term Memory
98,079 Citations1997Sepp Hochreiter, Jürgen Schmidhuber
A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)MizAR 60 for Mizar 50
76,311 Citations2023Jakubův, Jan, Chvalovský, Karel +7 more
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)AI-Assisted Pipeline for Dynamic Generation of Trustworthy Health Supplement Content at Scale
45,681 Citations2018Kefallinos, Dionysios, Alexandris, Georgios +6 more
32,525 Citations2019Jacob Devlin, Ming‐Wei Chang +2 more
A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.
SQuAD: 100,000+ Questions for Machine Comprehension of Text
6,328 Citations2016Pranav Rajpurkar, Jian Zhang +2 more
A strong logistic regression model is built, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%).
Translating embeddings for modeling multi-relational data
5,178 Citations2015Antoine Bordes, Nicolas Usunier +2 more
TransE is proposed, a method which models relationships by interpreting them as translations operating on the low-dimensional embeddings of the entities, which proves to be powerful since extensive experiments show that TransE significantly outperforms state-of-the-art methods in link prediction on two knowledge bases.
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
3,817 Citations2018Alex Wang, Amanpreet Singh +4 more
A benchmark of nine diverse NLU tasks, an auxiliary dataset for probing models for understanding of specific linguistic phenomena, and an online platform for evaluating and comparing models, which favors models that can represent linguistic knowledge in a way that facilitates sample-efficient learning and effective knowledge-transfer across tasks.
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context
3,149 Citations2019Zihang Dai, Zhilin Yang +4 more
This work proposes a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence, which consists of a segment-level recurrence mechanism and a novel positional encoding scheme.
Journal of Machine Learning ResearchA neural probabilistic language model
2,663 Citations2003BengioYoshua, DucharmeRéjean +2 more
This work reports on experiments using neural networks for the probability function, showing on two text corpora that the proposed approach very significantly improves on a state-of-the-art trigram model.
arXiv (Cornell University)Recurrent Neural Network Regularization
2,274 Citations2014Wojciech Zaremba, Ilya Sutskever +1 more
This paper shows how to correctly apply dropout to LSTMs, and shows that it substantially reduces overfitting on a variety of tasks.
Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books
2,035 Citations2015Yukun Zhu, Ryan Kiros +5 more
To align movies and books, a neural sentence embedding that is trained in an unsupervised way from a large corpus of books, as well as a video-text neural embedding for computing similarities between movie clips and sentences in the book are proposed.
Transactions of the Association for Computational LinguisticsNatural Questions: A Benchmark for Question Answering Research
1,994 Citations2019Tom Kwiatkowski, Jennimaria Palomaki +16 more
The Natural Questions corpus, a question answering data set, is presented, introducing robust metrics for the purposes of evaluating question answering systems; demonstrating high human upper bounds on these metrics; and establishing baseline results using competitive methods drawn from related literature.
Proceedings of the IEEEA Review of Relational Machine Learning for Knowledge Graphs
1,594 Citations2015Maximilian Nickel, Kevin Murphy +2 more
This paper provides a review of how statistical models can be “trained” on large knowledge graphs, and then used to predict new facts about the world (which is equivalent to predicting new edges in the graph) and how such statistical models of graphs can be combined with text-based information extraction methods for automatically constructing knowledge graphs from the Web.
Reading Wikipedia to Answer Open-Domain Questions
1,435 Citations2017Danqi Chen, Adam Fisch +2 more
This approach combines a search component based on bigram hashing and TF-IDF matching with a multi-layer recurrent neural network model trained to detect answers in Wikipedia paragraphs, indicating that both modules are highly competitive with respect to existing counterparts.
Don't count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors
1,379 Citations2014Marco Baroni, Georgiana Dinu +1 more
An extensive evaluation of context-predicting models with classic, count-vector-based distributional semantic approaches, on a wide range of lexical semantics tasks and across many parameter settings shows that the buzz around these models is fully justified.
Computational LinguisticsSimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation
1,197 Citations2015Felix Hill, Roi Reichart +1 more
SimLex-999 is presented, a gold standard resource for evaluating distributional semantic models that improves on existing resources in several important ways, and explicitly quantifies similarity rather than association or relatedness so that pairs of entities that are associated but not actually similar have a low rating.
arXiv (Cornell University)Language Modeling with Gated Convolutional Networks
1,129 Citations2016Yann Dauphin, Angela Fan +2 more
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
901 Citations2019Tom McCoy, Ellie Pavlick +1 more
There is substantial room for improvement in NLI systems, and the HANS dataset can motivate and measure progress in this area, which contains many examples where the heuristics fail.
International Conference on Machine LearningLanguage modeling with gated convolutional networks
884 Citations2017Yann Dauphin, Angela Fan +2 more
Context dependent recurrent neural network language model
574 Citations2012Tomáš Mikolov, Geoffrey Zweig
This paper improves recurrent neural network language models performance by providing a contextual real-valued input vector in association with each word to convey contextual information about the sentence being modeled by performing Latent Dirichlet Allocation using a block of preceding text.
Dissecting Contextual Word Embeddings: Architecture and Representation
421 Citations2018Matthew E. Peters, Mark E Neumann +2 more
There is a tradeoff between speed and accuracy, but all architectures learn high quality contextual representations that outperform word embeddings for four challenging NLP tasks, suggesting that unsupervised biLMs, independent of architecture, are learning much more about the structure of language than previously appreciated.
Representing General Relational Knowledge in ConceptNet 5
419 Citations2012Robert E. Speer, Catherine Havasi
The latest iteration of ConceptNet 5 is presented, including its fundamental design decisions, ways to use it, and evaluations of its coverage and accuracy.
arXiv (Cornell University)What do you learn from context? Probing for sentence structure in\n contextualized word representations
360 Citations2019Ian Tenney, Patrick Xia +9 more
Targeted Syntactic Evaluation of Language Models
351 Citations2018Rebecca Marvin, Tal Linzen
There is considerable room for improvement over LSTMs in capturing syntax in a language model, and a large gap remained between its performance and the accuracy of human participants recruited online in an experiment using this data set.
arXiv (Cornell University)CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
310 Citations2018Alon Talmor, Jonathan Herzig +2 more
arXiv (Cornell University)Assessing BERT's Syntactic Abilities
294 Citations2019Yoav Goldberg
The extent to which the recently introduced BERT model captures English syntactic phenomena is assessed, using naturally-occurring subject-verb agreement stimuli; "coloreless green ideas" subject- Verb Agreement stimuli; and manually crafted stimuli for subject- verb agreement and reflexive anaphora phenomena.
T-REx: A Large Scale Alignment of Natural Language with Knowledge Base Triples
174 Citations2018Hady Elsahar, Pavlos Vougiouklis +5 more
T-REx, a dataset of large scale alignments between Wikipedia abstracts and Wikidata triples, is presented, which is two orders of magnitude larger than the largest available alignments dataset and covers 2.5 times more predicates.
Context-Aware Representations for Knowledge Base Relation Extraction
150 Citations2017Daniil Sorokin, Iryna Gurevych
It is demonstrated that for sentence-level relation extraction it is beneficial to consider other relations in the sentential context while predicting the target relation and to combine the context representations with an attention mechanism to make the final prediction.
arXiv (Cornell University)BabyAI: First Steps Towards Grounded Language Learning With a Human In the Loop.
89 Citations2018Maxime Chevalier-Boisvert, Dzmitry Bahdanau +5 more
The BabyAI research platform is introduced to support investigations towards including humans in the loop for grounded language learning and puts forward strong evidence that current deep learning methods are not yet sufficiently sample efficient when it comes to learning a language with compositional properties.
arXiv (Cornell University)Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
88 Citations2019R. Thomas McCoy, Ellie Pavlick +1 more
A Survey of Reinforcement Learning Informed by Natural Language
73 Citations2019Jelena Luketina, Nantas Nardelli +6 more
The time is right to investigate a tight integration of natural language understanding into Reinforcement Learning in particular, and the state of the field is surveyed, including work on instruction following, text games, and learning from textual domain knowledge.