Identifying Word Relations in Software: A Comparative Study of Semantic Similarity Tools
Published 1 June 2008
Giriprasad Sridhara, Emily Hill, Lori Pollock, K. Vijay‐Shanker
Citations95
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A comparative study of six state of the art, English-based semantic similarity techniques and their effectiveness on words from the comments and identifiers in software suggests that applying English- based semantic similarity Techniques to software without any customization could be detrimental to the performance of the client software tools.
Abstract
Modern software systems are typically large and complex, making comprehension of these systems extremely difficult. Experienced programmers comprehend code by seamlessly processing synonyms and other word relations. Thus, we believe that automated comprehension and software tools can be significantly improved by leveraging word relations in software. In this paper, we perform a comparative study of six state of the art, English-based semantic similarity techniques and evaluate their effectiveness on words from the comments and identifiers in software. Our results suggest that applying English-based semantic similarity techniques to software without any customization could be detrimental to the performance of the client software tools. We propose strategies to customize the existing semantic similarity techniques to software, and describe how various program comprehension tools can benefit from word relation information.
Keywords
Computer Science
Language<b>WordNet: An electronic lexical database</b> . Ed. by Christiane Fellbaum. Cambridge, MA: MIT Press, 1998. Pp. xxii, 423.
11,687 Citations2000Adam Kilgarriff
The lexical database: nouns in WordNet, George A. Miller modifiers in WordNet, Katherine J. Miller a semantic network of English verbs, and applications of WordNet: building semantic concordances are presented.
Foundations of statistical natural language processing
9,996 Citations1999Christopher D. Manning, Hinrich Schütze
Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition
4,165 Citations2000Daniel Jurafsky, James Martin
This book takes an empirical approach to language processing, based on applying statistical and other machine-learning algorithms to large corpora, to demonstrate how the same algorithm can be used for speech recognition and word-sense disambiguation.
An Information-Theoretic Definition of Similarity
3,662 Citations1998Dekang Lin
This work presents an informationtheoretic definition of similarity that is applicable as long as there is a probabilistic model and demonstrates how this definition can be used to measure the similarity in a number of different domains.
Verbs semantics and lexical selection
3,086 Citations1994Zhibiao Wu, Martha Palmer
This paper will focus on the semantic representation of verbs in computer systems and its impact on lexical selection problems in machine translation (MT), and sees the approach as closely aligned with knowledge-based MT approaches (KBMT), and as a separate component that could be incorporated into existing systems.
arXiv (Cornell University)Semantic Similarity Based on Corpus Statistics and Lexical Taxonomy
2,224 Citations1997Jay J. Jiang, David W. Conrath
arXiv (Cornell University)Using Information Content to Evaluate Semantic Similarity in a Taxonomy
2,147 Citations1995Philip Resnik
arXiv (Cornell University)Using Information Content to Evaluate Semantic Similarity in a Taxonomy
1,760 Citations1995Philip Resnik
This paper presents a new measure of semantic similarity in an IS-A taxonomy, based on the notion of information content, which performs encouragingly well and is significantly better than the traditional edge counting approach.
Automatic sense disambiguation using machine readable dictionaries
1,756 Citations1986Michael Lesk
This procedure uses available dictionaries, so that it will process any text; and uses solely the immediate context to decide which sense of a word is intended (in written English).
The MIT Press eBooksCombining Local Context and WordNet Similarity for Word Sense Identification
1,638 Citations1998Claudia Leacock
This chapter contains sections titled: Introducfion, Training and Testing Data, Experiment 1: The Local Context Classifier, Experiment 2: Measuring Word Similarity In Wordnet, and Combining Local Context and Wordnet Similarity Measures.
Computational LinguisticsEvaluating WordNet-based Measures of Lexical Semantic Relatedness
1,414 Citations2006Alexander Budanitsky, Graeme Hirst
An information-content-based measure proposed by Jiang and Conrath is found superior to those proposed by Hirst and St-Onge, Leacock and Chodorow, Lin, and Resnik, and why distributional similarity is not an adequate proxy for lexical semantic relatedness.
IEEE Transactions on Software EngineeringRecovering traceability links between code and documentation
949 Citations2002Giuliano Antoniol, Gerardo Canfora +3 more
A probabilistic and a vector space information retrieval model is applied in two case studies to trace C++ source code onto manual pages and Java code to functional requirements to recover traceability links between source code and free text documents.
Who should fix this bug?
921 Citations2006John Anvik, Lyndon Hiew +1 more
This paper applies a machine learning algorithm to the open bug repository to learn the kinds of reports each developer resolves and reaches precision levels of 57% and 64% on the Eclipse and Firefox development projects respectively.
Lecture notes in computer scienceAn Adapted Lesk Algorithm for Word Sense Disambiguation Using WordNet
850 Citations2002Satanjeev Banerjee, Ted Pedersen
This paper presents an adaptation of Lesk's dictionary-based word sense disambiguation algorithm that uses the lexical database WordNet as the source of glosses for this approach, and attains an overall accuracy of 32%.
Extended gloss overlaps as a measure of semantic relatedness
719 Citations2003Satanjeev Banerjee, Ted Pedersen
A new measure of semantic relatedness between concepts that is based on the number of shared words (overlaps) in their definitions (glosses) and reasonably correlates to human judgments is presented.
Identifying similar code with program dependence graphs
614 Citations2002Jens Krinke
This approach to identifying similar code in programs based on finding similar subgraphs in attributed directed graphs considers not only the syntactic structure of programs but also the data flow within (as an abstraction of the semantics).
CERN Document Server (European Organization for Nuclear Research)Aspectj in Action: Practical Aspect-Oriented Programming
460 Citations2003Ramnivas Laddad
This is a practical guide to AOP and Aspect J, which enables AOP programming in Java by adding a few new language constructs and re-usable code examples will enable quick implementation of functionality.
An information retrieval approach to concept location in source code
448 Citations2005Andrian Marcus, Andrey Sergeyev +2 more
This work addresses the problem of concept location using an advanced information retrieval method, Latent Semantic Indexing (LSI), used to map concepts expressed in natural language by the programmer to the relevant parts of the source code.
25th International Conference on Software Engineering, 2003. Proceedings.Recovering documentation-to-source-code traceability links using latent semantic indexing
435 Citations2003Andrian Marcus, Jonathan I. Maletic
The method presented proves to give good results by comparison and additionally it is a low cost, highly flexible method to apply with regards to preprocessing and/or parsing of the source code and documentation.
Using structural context to recommend source code examples
385 Citations2005Reid Holmes, Gail C. Murphy
An approach for locating relevant code in an example repository that is based on heuristically matching the structure of the code under development to the example code, and the structural context needed to query the repository is extracted automatically from the code.
International Conference on Software EngineeringThe concept assignment problem in program understanding
326 Citations1993Ted J. Biggerstaff, Bharat G. Mitbander +1 more
/*icomment
268 Citations2007Lin Tan, Ding Yuan +2 more
iComment automatically extracts 1832 rules from comments with 90.8-100% accuracy and detects 60 comment-code inconsistencies, 33 newbugs and 27 bad comments, in the latest versions of the four programs.
Proceedings - Working Conference on Reverse EngineeringDo Code and Comments Co-Evolve? On the Relation between Source Code and Comment Changes
229 Citations2007Beat Fluri, Michael Würsch +1 more
An approach to map code and comments to observe their co-evolution over multiple versions is described and shows that newly added code - despite its growth rate - barely gets commented.
Sourcerer
223 Citations2006Sushil Bajracharya, Trung Chi Ngo +5 more
The authors present Sourcerer, a search engine for open-source code that extracts fine-grained structural information from the code and stores it in a relational model to implement a basic notion of CodeRank and to enable search forms that go beyond conventional keyword-based searches.
Using natural language program analysis to locate and understand action-oriented concerns
208 Citations2007David Shepherd, Zachary P. Fry +3 more
A semi-automated concern location and comprehension tool designed to reduce the time developers spend on maintenance tasks and to increase their confidence in the results of these tasks, which is effective because it searches a unique natural language-based representation of source code.
Advanced clone-analysis to support object-oriented system refactoring
177 Citations2002Magdalena Bałazińska, Ettore Merlo +3 more
A novel approach for computer-aided clone-based object-oriented system refactoring based on an advanced clone analysis which focuses on the extraction of clone differences and their interpretation in terms of programming language entities.
Maximizing Semantic Relatedness to Perform Word Sense Disambiguation
143 Citations2005Ted Pedersen, Satanjeev Banerjee +1 more
This article presents a method of word sense disambiguation that assigns a target word the sense that is most related to the senses of its neighboring words and explores the use of measures of similarity and relatedness that are based on finding paths in a concept network, information content derived from a large corpus, and word sense glosses.
Automatic generation of suggestions for program investigation
126 Citations2005Martin P. Robillard
A technique to automatically propose and rank program elements that are potentially interesting to a developer investigating source code based on an analysis of the topology of structural dependencies in a program is proposed.
A differencing algorithm for object-oriented programs
126 Citations2004Taweesup Apiwattanapong, Alessandro Orso +1 more
This work presents a technique for comparing object-oriented programs that identifies both differences and correspondences between two versions of a program and presents empirical results that show the efficiency and effectiveness of the technique on a real program.
Exploring the neighborhood with dora to expedite software maintenance
117 Citations2007Emily Hill, Lori Pollock +1 more
This approach uses structural information to focus automated program exploration and lexical information to prune irrelevant structure edges from consideration and demonstrates that an integrated lexical-and structural-based approach is significantly more effective than a state-of-the-art structural program exploration technique.
The concept assignment problem in program understanding
98 Citations2002Ted J. Biggerstaff, Bharat G. Mitbander +1 more
The authors argue that the solution to the problem of discovering individual human oriented concepts and assigning them to their implementation oriented counterparts for a given program is the concept assignment problem and requires methods that have a strong plausible reasoning component.
Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005.Using structural context to recommend source code examples
77 Citations2005Reid Holmes, Gail C. Murphy
A differencing algorithm for object-oriented programs
69 Citations2004Taweesup Apiwattanapong, Alessandro Orso +1 more
Recommending random walks
63 Citations2007Zachary M. Saul, Vladimir Filkov +2 more
A random-walk approach, mimicking the more focused behavior of a developer, who browses the caller-callee links in the callgraph of a large program, seeking routines that are likely to be related to a function of interest, finds that this algorithm dramatically improves upon Suade in this setting.
Leveraged Quality Assessment using Information Retrieval Techniques
51 Citations2006Dawn Lawrie, H. Feild +1 more
Statistically significant results from this experiment validate the use of the QALP tool in human leverage quality assessment.
ACM SIGSOFT Software Engineering NotesAutomatic generation of suggestions for program investigation
43 Citations2005Martin P. Robillard
Lecture notes in computer scienceAutomatic detection of thesaurus relations for information retrieval applications
43 Citations1997Gerda Ruge
This article explains how implicit semantic knowledge can be transformed to an explicit one and evaluates of quality and performance of the approach are very encouraging.
Journal of PragmaticsThe language of comments in computer software: A sublanguage of English
38 Citations2001Letha H. Etzkorn, Carl G. Davis +1 more
Whether English-language comments in object-oriented software can be considered to be a sublanguage of English, using standard criteria for sublanguage determination is explored, including the grammatical content of comments, and the use of tense, mood, and voice in sentence-style comments.
Case study
12 Citations2007David Shepherd, Lori Pollock +1 more