Viewing sentence boundary detection as collocation identification

A flexible, languageindependent and accurate method based on the idea that an abbreviation can be viewed as a collocation is described, which shows that abbreviations can be detected with high accuracy and shows that inaccurate tokenization leads to a considerably higher error rate during tagging.

Abstract

The detection of abbreviations is an important step in the process of sentence boundary detection. We describe a flexible, languageindependent and accurate method based on the idea that an abbreviation can be viewed as a collocation. As such, it can be identified by using methods for collocation detection such as the log likelihood ratio. Although the log likelihood ratio is known to show a good recall, its precision is poor. We employ scaling factors that lead to a strong improvement of precision. Experiments with English and German corpora show that abbreviations can be detected with high accuracy. We also show that inaccurate tokenization leads to a considerably higher error rate during tagging.