A probabilistic description-oriented approach for categorizing web documents

Published 1 November 1999Open access

Norbert Gövert, Mounia Lalmas, Norbert Fuhr

DOI10.1145/319950.320053

Citations61

TL;DR

This work provides an enhanced document representation that incorporates the structural and heterogeneous nature of web documents and provides a theoretical sound justification for the various parameters of the k-nearest neighbour classifier.

Abstract

The automatic categorisation of web documents is becoming crucial for organising the huge amount of information available in the Internet. We are facing a new challenge due to the fact that web documents have a rich structure and are highly heterogeneous. Two ways to respond to this challenge are (1) to use a representation of the content of web documents that captures these two characteristics and (2) to use more effective classifiers. Our categorisation approach is based on a probabilistic description-oriented representation of web documents, and a probabilistic interpretation of the k-nearest neighbour classifier. With the former, we provide an enhanced document representation that incorporates the structural and heterogeneous nature of web documents. With the latter, we provide a theoretical sound justification for the various parameters of k-nearest neighbour classifier. Experimental results show that (1) using an enhanced representation of web documents is crucial for an effective categorisation of web documents, and (2) a theoretical interpretation of the k-nearest neighbour classifier gives us improvement over the standard k-nearest neighbour classifier.

Keywords

Computer Science

Information Processing & ManagementTerm-weighting approaches in automatic text retrieval

9,532 Citations1988Gerard Salton, Chris Buckley

This paper summarizes the insights gained in automatic term weighting, and provides baseline single term indexing models with which other more elaborate content analysis procedures can be compared.

Information RetrievalAn Evaluation of Statistical Approaches to Text Categorization

1,946 Citations1999Yiming Yang

Analysis and empirical evidence suggest that the evaluation results on some versions of Reuters were significantly affected by the inclusion of a large portion of unlabelled documents, mading those results difficult to interpret and leading to considerable confusions in the literature.

Enhanced hypertext categorization using hyperlinks

775 Citations1998Soumen Chakrabarti, Byron Dom +1 more

This work has developed a text classifier that misclassified only 13% of the documents in the well-known Reuters benchmark; this was comparable to the best results ever obtained and its technique also adapts gracefully to the fraction of neighboring documents having known topics.

View PDF

ACM Transactions on Information SystemsA probabilistic learning approach for document indexing

202 Citations1991Norbert Fuhr, Chris Buckley

A method for probabilistic document indexing using relevance feedback data that has been collected from a set of queries based on three new concepts, which allows the integration of new text analysis and knowledge-based methods in this approach as well as the consideration of document structures or different types of terms.

View PDF

ACM Transactions on Information SystemsOn modeling information retrieval with probabilistic inference

185 Citations1995S. K. M. Wong, Yiyu Yao

This article examines and extends the logical models of information retrieval in the context of probability theory, and the fundamental notions of term weights and relevance are given probabilistic interpretations.

View PDF

Information Processing & ManagementModels for retrieval with probabilistic indexing

177 Citations1989Norbert Fuhr

Three retrieval models for probabilistic indexing are described along with evaluation results for each, including the binary independence indexing (BII) model, which is a generalized version of the Maron and Kuhns indexing model.

Towards an information logic

127 Citations1989C. J. van Rijsbergen

This paper argues that much of the research work in the past has been steps in the direction of a logic for IR, and presents a picture of IR, describing bits of a puzzle which may fit together to point to a new framework within which a computational model or logic could be described.

View PDF

Optimizing document indexing and search term weighting based on probabilistic models

17 Citations1992Norbert Fuhr, Chris Buckley

This work describes the application of probabilistic indexing and retrieval methods to the TREC material and applies a description-oriented approach which uses relevance feedback information from previous queries run on the same collection.