Evaluating Cross-Language Text Filtering Effectiveness

This chapter describes an evaluation methodology for cross-language text filtering systems which exploits existing test collections that were designed for monolingual evaluations, and concludes that the additional investment required to produce a truly multilingual test collection would be well justified.

Abstract

In this chapter we describe an evaluation methodology for cross-language text filtering systems which exploits existing test collections that were designed for monolingual evaluations. Our methodology, based on normative relevance assessments by expert users, is well suited for comparing the effect of different cross-language mapping techniques on filtering accuracy. By measuring the degradation introduced by the use of existing test collections, we are able to qualify the broader applicability of our results and to quantify the improvement in evaluation accuracy that would result from development of a test collection tailored to the evaluation of multilingual text filtering systems. From our experiments we conclude that the additional investment required to produce a truly multilingual test collection would be well justified because evaluation of multilingual text filtering techniques appears to be both practical and productive.