Comparing case-based reasoning classifiers for predicting high risk software components

Evaluating the performance of a CBR classifier with different parameters, namely: different distance measures, different standardization techniques, use or non-use of weights, and the number of nearest neighbors to use for the prediction indicates that there is no difference in prediction performance when using any combination of parameters.

Abstract

Case-based reasoning (CBR) has been proposed for predicting the risk class of software components. Risky components can be defined as those that are fault-prone, or those that require a large amount of effort to maintain. Thus far evaluative studies of CBR classifiers have been promising, showing that their predictive performance is as good as or better than other types of classifiers. However, a CBR classifier can be instantiated in different ways by varying its parameters, and it is not clear which combination of parameters provides the best performance. In this paper we evaluate the performance of a CBR classifier with different parameters, namely: (a) different distance measures, (b) different standardization techniques, (c) use or non-use of weights, and (d) the number of nearest neighbors to use for the prediction. In total, we compared 30 different CBR classifiers. The study was conducted with a data set from a large real-time system, and the objective was to predict the fault-proneness of its components. Our results indicate that there is no difference in prediction performance when using any combination of parameters. Based on these results, we recommend using a simple CBR classifier with Euclidean distance, z-score standardization, no weighting scheme, and selecting the single nearest neighbor for prediction. The advantage of such a classifier is its intuitive appeal to nonspecialists, and the fact that it performs as well as more complex classifiers.