Evidence, inference, and the “rejection” of the significance test

Abstract

Hammond (1996) reiterates Cohen's (1994) "attack" on simple-minded interpretations of significance tests and recommends the use of other statistical methods (including effect size measures and confidence intervals) in their place. Hammond's laudable aim is to inform the Australian psychology community of the resurgence of this debate in the US, and to open these issues to overdue debate here. In this paper we take the stand that the issues underlying some of the criticisms in this debate have not been well drawn. In particular, we believe that the fundamental distinction between the interpretation of probability as relative frequency and its interpretation as evidentiary-belief—a distinction underlying the history of confusion about statistical inference in psychology and elsewhere—is still not receiving the major focus it requires in this debate. We argue that these interpretive issues are just as relevant for confidence intervals as for significance tests and that the problem of inference—that of specifying how sample data provide evidence about unknown population parameters—is not a purely mathematical one. As a result, such issues should not be left to the "statistical types" among us; rather, psychologists who wish to perform or evaluate research and the conclusions drawn from it need to understand the different approaches that have been taken to the problem. Indeed, we see the encouragement—and liberty—to re-think the role of data analysis in the interpretation of our research findings as the most positive aspect of the debate.