The Missing Link - A Probabilistic Model of Document Content and Hypertext Connectivity

Published 1 January 2000

David Cohn, Thomas Hofmann

Citations439

TL;DR

A joint probabilistic model for modeling the contents and inter-connectivity of document collections such as sets of web pages or research paper archives is described, based on a Probabilistic factor decomposition.

Abstract

We describe a joint probabilistic model for modeling the contents and inter-connectivity of document collections such as sets of web pages or research paper archives. The model is based on a probabilistic factor decomposition and allows identifying principal topics of the collection as well as authoritative documents within those topics. Furthermore, the relationships between topics is mapped out in order to build a predictive model of link content. Among the many applications of this approach are information retrieval and search, topic identification, query disambiguation, focused web crawling, web authoring, and bibliometric analysis.

Keywords

Computer SciencePhysics and Astronomy

Computer Networks and ISDN SystemsThe anatomy of a large-scale hypertextual Web search engine

15,828 Citations1998Sergey Brin, Lawrence M. Page

This paper provides an in-depth description of Google, a prototype of a large-scale search engine which makes heavy use of the structure present in hypertext and looks at the problem of how to effectively deal with uncontrolled hypertext collections where anyone can publish anything they want.

NatureLearning the parts of objects by non-negative matrix factorization

14,055 Citations1999Daniel D. Lee, H. Sebastian Seung

An algorithm for non-negative matrix factorization is demonstrated that is able to learn parts of faces and semantic features of text and is in contrast to other methods that learn holistic, not parts-based, representations.

Journal of the American Society for Information ScienceIndexing by latent semantic analysis

12,677 Citations1990Scott Deerwester, Susan Dumais +3 more

arXiv (Cornell University)Probabilistic Latent Semantic Analysis

2,092 Citations2013Thomas Hofmann

This work proposes a widely applicable generalization of maximum likelihood model fitting by tempered EM, based on a mixture decomposition derived from a latent class model which results in a more principled approach which has a solid foundation in statistics.

View PDF

Symposium on Discrete AlgorithmsAuthoritative sources in a hyperlinked environment

1,836 Citations1998Jon Kleinberg

Information RetrievalAutomating the Construction of Internet Portals with Machine Learning

1,265 Citations2000Andrew Kachites McCallum, Kamal Nigam +2 more

New research in reinforcement learning, information extraction and text classification that enables efficient spidering, the identification of informative text segments, and the population of topic hierarchies are described.

Learning Probabilistic Relational Models

941 Citations2001Lise Getoor, Nir Friedman +2 more

ACM SIGIR ForumImproved Algorithms for Topic Distillation in a Hyperlinked Environment

677 Citations2017Krishna Bharat, Monika Henzinger

This paper addresses the problem of topic distillation on the World Wide Web, namely, given a typical user query to find quality documents related to the query topic, by augmenting a previous connectivity analysis based algorithm with content analysis.

Multi-paragraph segmentation of expository text

558 Citations1994Marti A. Hearst

TextTiling, an algorithm for partitioning expository texts into coherent multi-paragraph discourse units which reflect the subtopic structure of the texts, is described and shown to produce segmentation that corresponds well to human judgments of the major subtopic boundaries of thirteen lengthy texts.

View PDF

arXiv (Cornell University)Multi-Paragraph Segmentation of Expository Text

392 Citations1994Marti A. Hearst

View PDF