Test-driving tanka: Evaluating a semi-automatic system of text analysis for knowledge acquisition

The evaluation of a large implemented natural language processing system involves more than its application to a common performance task, and a detailed evaluation must take into account the specificity of the system.

Abstract

The evaluation of a large implemented natural language processing system involves more than its application to a common performance task. Such tasks have been used in the message understanding conferences (MUCs), text retrieval conferences (TRECs) as well as in speech technology and machine translation workshops. It is useful to compare the performance of different systems in a predefined application, but a detailed evaluation must take into account the specificity of the system. We have carried out a systematic performance evaluation of our text analysis system tanka. Since it is a semi-automatic, trainable system, we had to measure the user's participation (with a view to decreasing it gradually) and the rate at which the system learns from preceding analyses. This paper discusses the premises, the design and the execution of an evaluation of tanka. The results confirm the basic assumptions of our supervised text analysis procedures, namely, that the system learns to make better analyses, that knowledge acquisition is possible even from erroneous or fragmentary parses and that the process is not too onerous for the user.