Using Simulation to Evaluate Prediction Techniques

There are significant differences depending upon the characteristics of the dataset in regression, rule induction and nearest neighbour (a form of case based reasoning) and the results suggest that researchers should consider prediction context when evaluating competing prediction systems.

Abstract

The need for accurate software prediction systems increases as software becomes much larger and more complex. A variety of techniques have been proposed, however, none has proved consistently accurate and there is still much uncertainty as to what technique suits which type of prediction problem. We believe that the underlying characteristics -- size, number of features, type of distribution, etc. -- of the dataset influence the choice of the prediction system to be used. In previous work, it has proved difficult to obtain significant results over small datasets. Consequently we required large validation datasets, moreover, we wished to control the characteristics of such datasets in order to systematically explore the relationship between accuracy, choice of prediction system and dataset characteristic. Our solution has been to simulate data allowing both control and the possibility of large (1000) validation cases. In this paper we compared regression, rule induction and nearest neighbour (a form of...