The fallacy of the null-hypothesis significance test.

To the experimental scientist, statistical inference is a research instrument, a processing device by which unwieldy masses of raw data may be refined into a product more suitable for assimilation into the corpus of science, and in this lies both strength and weakness.

Abstract

The theory of probability and statistical inference is various things to various people. To the mathematician, it is an intricate formal calculus, to be explored and developed with little professional concern for any empirical significance that might attach to the terms and propositions involved. This chapter examines a dogma of inferential procedure which, for psychologists at least, has attained the status of a religious conviction. The null hypothesis significance test treats acceptance or rejection of a hypothesis as though these were decisions one makes on the basis of the experimental data. The traditional null hypothesis decision test unrealistically limits the significance of an experimental outcome to a mere two alternatives, confirmation or disconfirmation of the null hypothesis. Acceptance or rejection of a hypothesis is a cognitive process, a degree of believing or disbelieving which, if rational, is not a matter of choice but determined solely by how likely it is, given the evidence, that the hypothesis is true.