Text-to-text semantic similarity for automatic short answer grading
Published 1 January 2009Open access
Michael Mohler, Rada Mihalcea
Citations293
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper compares a number of knowledge-based and corpus-based measures of text similarity, evaluates the effect of domain and size on the corpus- based measures, and introduces a novel technique to improve the performance of the system by integrating automatic feedback from the student answers.
Abstract
In this paper, we explore unsupervised techniques for the task of automatic short answer grading. We compare a number of knowledge-based and corpus-based measures of text similarity, evaluate the effect of domain and size on the corpus-based measures, and also introduce a novel technique to improve the performance of the system by integrating automatic feedback from the student answers. Overall, our system significantly and consistently outperforms other unsupervised methods for short answer grading that have been proposed in the past.
Keywords
Computer Science
Communications of the ACMA vector space model for automatic indexing
7,434 Citations1975Gerard Salton, Anita M.-Y. Wong +1 more
An approach based on space density computations is used to choose an optimum indexing vocabulary for a collection of documents, demonstating the usefulness of the model.
Psychological ReviewA solution to Plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge.
6,039 Citations1997Thomas K. Landauer, Susan Dumais
A new general theory of acquired similarity and knowledge representation, latent semantic analysis (LSA), is presented and used to successfully simulate such learning and several other psycholinguistic phenomena.
An Information-Theoretic Definition of Similarity
3,662 Citations1998Dekang Lin
This work presents an informationtheoretic definition of similarity that is applicable as long as there is a probabilistic model and demonstrates how this definition can be used to measure the similarity in a number of different domains.
Verbs semantics and lexical selection
3,086 Citations1994Zhibiao Wu, Martha Palmer
This paper will focus on the semantic representation of verbs in computer systems and its impact on lexical selection problems in machine translation (MT), and sees the approach as closely aligned with knowledge-based MT approaches (KBMT), and as a separate component that could be incorporated into existing systems.
arXiv (Cornell University)Semantic Similarity Based on Corpus Statistics and Lexical Taxonomy
2,224 Citations1997Jay J. Jiang, David W. Conrath
Computing semantic relatedness using Wikipedia-based explicit semantic analysis
1,990 Citations2007Evgeniy Gabrilovich, Shaul Markovitch
This work proposes Explicit Semantic Analysis (ESA), a novel method that represents the meaning of texts in a high-dimensional space of concepts derived from Wikipedia that results in substantial improvements in correlation of computed relatedness scores with human judgments.
arXiv (Cornell University)Using Information Content to Evaluate Semantic Similarity in a Taxonomy
1,760 Citations1995Philip Resnik
This paper presents a new measure of semantic similarity in an IS-A taxonomy, based on the notion of information content, which performs encouragingly well and is significantly better than the traditional edge counting approach.
Automatic sense disambiguation using machine readable dictionaries
1,756 Citations1986Michael Lesk
This procedure uses available dictionaries, so that it will process any text; and uses solely the immediate context to decide which sense of a word is intended (in written English).
Corpus-based and knowledge-based measures of text semantic similarity
1,189 Citations2006Rada Mihalcea, Courtney D. Corley +1 more
This paper shows that the semantic similarity method out-performs methods based on simple lexical matching, resulting in up to 13% error rate reduction with respect to the traditional vector-based similarity metric.
WordNet: similarity - measuring the relatedness of concepts
866 Citations2004Ted Pedersen, Siddharth Patwardhan +1 more
Psychological ReviewA solution to Plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge.
853 Citations1997Thomas K. Landauer, Susan Dumais
The MIT Press eBooksLexical Chains as Representations of Context for the Detection and Correction of Malapropisms
840 Citations1998Graeme Hirst
How lexical chains can be constructed by means of WordNet, and how they can be applied in one particularlinguistic task: the detection and correction of malapropisms is shown.
Lecture notes in computer scienceUsing Measures of Semantic Relatedness for Word Sense Disambiguation
490 Citations2003Siddharth Patwardhan, Satanjeev Banerjee +1 more
This paper generalizes the Adapted Lesk Algorithm to a method of word sense disambiguation based on semantic relatedness and finds that the gloss overlaps of AdaptedLesk and the semantic distance measure of Jiang and Conrath (1997) result in the highest accuracy.
arXiv (Cornell University)Verb Semantics and Lexical Selection
460 Citations1994Zhibiao Wu, Martha Palmer
Overcoming the brittleness bottleneck using wikipedia: enhancing text categorization with encyclopedic knowledge
404 Citations2006Evgeniy Gabrilovich, Shaul Markovitch
It is proposed to enrich document representation through automatic use of a vast compendium of human knowledge--an encyclopedia, and empirical results confirm that this knowledge-intensive representation brings text categorization to a qualitatively new level of performance across a diverse collection of datasets.
Computers and the HumanitiesC-rater: Automated Scoring of Short-Answer Questions
372 Citations2003Claudia Leacock, Martin Chodorow
C-rater is an automated scoring engine that has been developed to score responses to content-based short answer questions using predicateargument structure, pronominal reference, morphological analysis and synonyms to assign full or partial credit to a short answer question.
White Rose Research Online (University of Leeds, The University of Sheffield, University of York)Exploiting Syntactic and Shallow Semantic Kernels for Question Answer Classification
191 Citations2007Alessandro Moschitti, Silvia Quarteroni +2 more
The experiments suggest that syntactic information helps tasks such as question/answer classification and that shallow semantics gives remarkable contribution when a reliable set of PASs can be extracted, e.g. from answers.
Adelaide Research & Scholarship (AR&S) (University of Adelaide)An Empirical Evaluation of Models of Text Document Similarity
185 Citations2005Michael Lee, Brandon Pincombe +1 more
An attempt to establish a ‘psychological ground truth’ for evaluating models of the ability of word-based, n-gram and Latent Semantic Analysis approaches to model human judgments of text document similarity is reported.
Detecting Text Similarity over Short Passages: Exploring Linguistic Feature Combinations via Machine Learning
178 Citations1999Vasileios Hatzivassiloglou, Judith L. Klavans +1 more
A new composite similarity metric is presented that combines information from multiple linguistic indicators to measure semantic distance between pairs of small textual units and is evaluated against standard information retrieval techniques, establishing that the new method is more effective in identifying closely related textual units.
Loughborough University Institutional Repository (Loughborough University)Towards robust computerised marking of free-text responses
150 Citations2002Tom M. Mitchell, Terry Russell +2 more
The aim of this exploration is to reveal some of the issues which need to be addressed if computerised marking is to play any kind of reliable role in the future development of such test regimes.
North American Chapter of the Association for Computational LinguisticsEvaluating Multiple Aspects of Coherence in Student Essays
128 Citations2004Derrick Higgins, Jill Burstein +2 more
A new system is described that enhances Criterion’s capability, by evaluating multiple aspects of coherence in essays, by identifying features of sentences based on semantic similarity measures and discourse structure.
Automatic evaluation of students' answers using syntactically enhanced LSA
84 Citations2003Dharmendra Kanejiya, Arun Kumar +1 more
Syntactically Enhanced LSA is presented here an approach which generalizes LSA by considering a word along with its syntactic neighborhood given by the part-of-speech tag of its preceding word, as a unit of knowledge representation, which provides better discrimination of syntactic-semantic knowledge representation than LSA.
Loughborough University Institutional Repository (Loughborough University)CAA of Short Non-MCQ Answers
64 Citations2001David Callear, Jenny Jerrams-Smith +1 more
The expected results from prototyping using ATM are obtained, indicating the reliability and feasibility of this new approach for the detailed assessment of text contents incorporating word order.
Lecture notes in computer scienceBeyond the Short Answer Question with Research Methods Tutor
12 Citations2002Malatesta Kalliopi-Irini, Peter Hastings +1 more
Research Methods Tutor is a new intelligent tutoring system created by porting the existing implementation of the AutoTutor system to a new domain, Research Methods in Behavioural Sciences, which allows more interactive dialogues.