STOCHASTIC MODELS FOR THE DISTRIBUTION OF INDEX TERMS

A probability model of the occurrence of index terms is used to derive discrete distributions which are mixtures of Poisson and negative binomial distributions, which have the advantage of being more explanatory and can incorporate a time parameter if necessary.

Abstract

Distributions of index terms have been used in modelling information retrieval systems and databases. Most previous models used some form of the Zipf distribution. This work uses a probability model of the occurrence of index terms to derive discrete distributions which are mixtures of Poisson and negative binomial distributions. These distributions, the generalised inverse Gaussian‐Poisson and the Generalised Waring give better fits than the simpler Zipf distribution, particularly in the tails of the distribution where the high frequency terms are found. They have the advantage of being more explanatory and can incorporate a time parameter if necessary.