An analysis of statistical and syntactic phrases

It is discovered that once a good basic ranking scheme is being used, the use of phrases does not have a major effect on precision at high ranks, and phrases are more useful at lower ranks where the connection between documents and relevance is more tenuous.

Abstract

As the amount of textual information available through the World Wide Web grows, there is a growing need for high-precision IR systems that enable a user to find useful information from the masses of available textual data. Phrases have traditionally been regarded as precision-enhancing devices and have proved useful as content-identifiers in representing documents. In this study, we compare the usefulness of phrases recognized using linguistic methods and those recognized by statistical techniques. We focus in particular on high-precision retrieval. We discover that once a good basic ranking scheme is being used, the use of phrases does not have a major effect on precision at high ranks. Phrases are more useful at lower ranks where the connection between documents and relevance is more tenuous. Also, we find that the syntactic and statistical methods for recognizing phrases yield comparable performance. 1 Introduction The amount of textual information available through the World Wide...