DeepDive: Web-scale Knowledge-base Construction using Statistical Learning and Inference
Published 1 January 2012
Feng Niu, Ce Zhang, R Christopher, Jude Shavlik
Citations167
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
An end-to-end (live) demonstration system called DeepDive is presented that performs knowledge-base construction (KBC) from hundreds of millions of web pages and addresses the scalability challenges to achieve web-scale KBC.
Abstract
We present an end-to-end (live) demonstration system called DeepDive that performs knowledge-base construction (KBC) from hundreds of millions of web pages. DeepDive employs statistical learning and inference to combine diverse data resources and best-of-breed algorithms. A key challenge of this approach is scalability, i.e., how to deal with terabytes of imperfect data efficiently. We describe how we address the scalability challenges to achieve web-scale KBC and the lessons we have learned from building DeepDive. 1.
Keywords
Computer Science
Journal of the Royal Statistical Society Series B (Statistical Methodology)Regression Shrinkage and Selection Via the Lasso
51,790 Citations1996Robert Tibshirani
A new method for estimation in linear models called the lasso, which minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant, is proposed.
Distant supervision for relation extraction without labeled data
2,913 Citations2009Mike D. Mintz, Steven Bills +2 more
This work investigates an alternative paradigm that does not require labeled corpora, avoiding the domain dependence of ACE-style algorithms, and allowing the use of corpora of any size.
Machine LearningMarkov logic networks
2,678 Citations2006Matthew Richardson, Pedro Domingos
Experiments with a real-world database and knowledge base in a university domain illustrate the promise of this approach to combining first-order logic and probabilistic graphical models in a single representation.
Proceedings of the AAAI Conference on Artificial IntelligenceToward an Architecture for Never-Ending Language Learning
1,991 Citations2010J. Andrew Carlson, Justin Betteridge +4 more
AI MagazineBuilding Watson: An Overview of the DeepQA Project
1,517 Citations2010David Ferrucci, Eric W. Brown +10 more
The results strongly suggest that DeepQA is an effective and extensible architecture that may be used as a foundation for combining, deploying, evaluating and advancing a wide range of algorithmic techniques to rapidly advance the field of QA.
Overview of the TAC 2010 Knowledge Base Population Track
419 Citations2010Heng Ji, Ralph Grishman +3 more
An overview of the task definition and annotation challenges associated with KBP2010 is provided and the evaluation results and lessons that are learned are discussed based on detailed analysis.
Autonomously semantifying wikipedia
381 Citations2007Fei Wu, Daniel S. Weld
This paper argues that autonomously "Semantifying Wikipedia" is the best way to solve the chicken-and-egg problem, and describes a prototype implementation of a self-supervised, machine learning system which realizes the vision.
StatSnowball
259 Citations2009Jun Zhu, Zaiqing Nie +3 more
A statistical extraction framework called Statistical Snowball (StatSnowball), which is a bootstrapping system and can perform both traditional relation extraction and Open IE, is proposed and a working entity relation search engine called Renlifang is developed based on it.
Proceedings of the VLDB EndowmentTuffy
257 Citations2011Feng Niu, Christopher Ré +2 more
This work presents Tuffy, a scalable Markov Logic Networks framework that achieves scalability via three novel contributions: a bottom-up approach to grounding, a novel hybrid architecture that allows to perform AI-style local search efficiently using an RDBMS, and a theoretical insight that shows when one can improve the efficiency of stochastic local search.
Scalable knowledge harvesting with high precision and high recall
198 Citations2011Ndapa Nakashole, Martin Theobald +1 more
A new notion of ngram-itemsets for richer patterns is proposed, and MaxSat-based constraint reasoning is used on both the quality of patterns and the validity of fact candidates, to use in a scalable system for high-quality knowledge harvesting.
Towards a unified architecture for in-RDBMS analytics
178 Citations2012Xixuan Feng, Arun Kumar +2 more
This work proposes a unified architecture for in-database analytics that requires changes to only a few dozen lines of code to integrate a new statistical technique, and demonstrates the feasibility of this architecture by integrating several popular analytics techniques into two commercial and one open-source RDBMS.
From information to knowledge
143 Citations2010Gerhard Weikum, Martin Theobald
This tutorial discusses state-of-the-art methods, research opportunities, and open challenges along this avenue of knowledge harvesting, to automatically construct and maintain a comprehensive knowledge base of facts about named entities, their semantic classes, and their mutual relations as well as temporal contexts, with high precision and high recall.
ACM SIGMOD RecordThe YAGO-NAGA approach to knowledge discovery
104 Citations2009Gjergji Kasneci, Maya Ramanath +2 more
The architecture of the YAGO extractor toolkit, its distinctive approach to consistency checking, its provisions for maintenance and further growth, and the query engine for YAGA, coined NAGA are presented.
Proceedings of the VLDB EndowmentREX
90 Citations2011Lujun Fang, Anish Das Sarma +2 more
REX is presented, a system that takes a pair of entities in a given knowledge base as input and efficiently identifies a ranked list of relationship explanations, which formally define relationship explanations and analyze their desirable properties.
Big Data versus the Crowd: Looking for Relationships in All the Right Places
46 Citations2012Ce Zhang, Feng Niu +2 more
The experiments show that increasing the corpus size for distant supervision has a statistically significant, positive impact on quality (F1 score), and human feedback has a positive and statistically significant; but lower, impact on precision and recall.