Using a randomised controlled clinical trial to evaluate an NLG system
Published 1 January 2001Open access
Ehud Reiter, Roma Robertson, A Scott Lennox, Liesl M. Osman
Citations22
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The STOP system, which generates personalised smoking-cessation letters, was evaluated by a randomised controlled clinical trial, which is believed to be the largest and perhaps most rigorous task effectiveness evaluation ever performed on an NLG system.
Abstract
The STOP system, which generates personalised smoking-cessation letters, was evaluated by a randomised controlled clinical trial. We believe this is the largest and perhaps most rigorous task effectiveness evaluation ever performed on an NLG system. The detailed results of the clinical trial have been presented elsewhere, in the medical literature. In this paper we discuss the clinical trial itself: its structure and cost, what we did and did not learn from it (especially considering that the trial showed that STOP was not effective), and how it compares to other NLG evaluation techniques.
Keywords
Computer ScienceMedicineHealth Professions
Archives of Internal MedicineAn Analysis of the Effectiveness of Interventions Intended to Help People Stop Smoking
462 Citations1995Malcolm Law
Following personal advice and encouragement to stop smoking given by physicians during a single routine consultation, an estimated 2% of all smokers stopped smoking and did not relapse up to 1 year as a direct consequence of the advice.
Developing and empirically evaluating robust explanation generators: the KNIGHT experiments
84 Citations1997James C. Lester, Bruce Porter
This paper describes KNIGHT, a robust explanation system that constructs multisentential and multi-paragraph explanations from the Biology Knowledge Base, a large-scale knowledge base in the domain of botanical anatomy, physiology, and development, and introduces the Two-Panel evaluation methodology.
BMJCost effectiveness of computer tailored and non-tailored smoking cessation letters in general practice: randomised controlled trial
81 Citations2001A Scott Lennox, Liesl M Osman +6 more
In a large general practice, a brief non-tailored letter effectively increased cessation rates among smokers and was not effective in increasing cessation rates but promoted shift in movement towards cessation in heavy smokers.
Computer Speech & LanguageEvaluation in the context of natural language generation
67 Citations1998Chris Mellish, Robert Dale
What is involved in natural language generation, and how evaluation has figured in work in this area to date is described; a particular text generation application is examined and the issues that are raised in assessing its performance on a variety of dimensions are looked at.
Artificial IntelligenceUsing Grice's maxim of Quantity to select the content of plan descriptions
59 Citations1999R. Michael Young
The Cooperative Plan Identification (CPI) architecture is presented, a computational model that generates concise, effective textual descriptions of plans that speakers and hearers cooperate with one another in their communication about a plan.
Knowledge acquisition for natural language generation
51 Citations2000Ehud Reiter, Roma Robertson +1 more
The knowledge acquisition techniques used to build the STOP system, especially sorting and think-aloud protocols, are described and informal evaluations suggest that they had some benefit, but perhaps were most successful as a source of insight and hypotheses.
Evaluating and comparing three text-production techniques
38 Citations1996José Coch
An assessment of three techniques for producing multisentential text: semi-automatic fill-in-the-blank interfacing, automatic linguistic-and-templates hybrid generation, and human writing using a black-box methodology.
An empirical study of the influence of argument conciseness on argument effectiveness
34 Citations2000Giuseppe Carenini, Johanna D. Moore
The results of a formal experiment performed in an evaluation framework in which the effectiveness of evaluative arguments can be measured with real users are presented to verify the influence of argument conciseness on argument effectiveness.
Computational LinguisticsPipelines and Size Constraints
33 Citations2000Ehud Reiter
Experimental data shows that a multiple-solution pipeline does much better than a single-s solution pipeline, and that a revision-based system does best of all.
Lecture notes in computer scienceTypes of Knowledge Required to Personalise Smoking Cessation Letters
24 Citations1999Ehud Reiter, Roma Robertson +1 more
The STOP system generates personalised smoking-cessation letters, using as input responses to a smoking questionnaire, largely based on knowledge of psychology, empathy, and readability.
An empirical study on the generation of anaphora in Chinese
24 Citations1997Ching-Long Yeh, Chris Mellish
The results of the comparison show that the rules are fairly effective in dealing with the generation of anaphora in Chinese, and they are implemented in a Chinese natural language generation system that is able to generate descriptive texts.
Evaluating and comparing three text-production techniques
7 Citations2002Ios Coch