A novel corpus-based stemming algorithm using co-occurrence statistics

A stemming algorithm for text retrieval using a very simple co-occurrence measure that reflects how often a pair of word variants occurs in a document as well as in the whole corpus and a partition algorithm that groups the word variants based on their strongest neighbors, that is, the neighbors with largest strengths.

Abstract

We present a stemming algorithm for text retrieval. The algorithm uses the statistics collected on the basis of certain corpus analysis based on the co-occurrence between two word variants. We use a very simple co-occurrence measure that reflects how often a pair of word variants occurs in a document as well as in the whole corpus. A graph is formed where the word variants are the nodes and two word variants form an edge if they co-occur. On the basis of the co-occurrence measure, a certain edge strength is defined for each of the edges. Finally, on the basis of the edge strengths, we propose a partition algorithm that groups the word variants based on their strongest neighbors, that is, the neighbors with largest strengths.