The Appropriateness of Analysis of Variance and Multiple-Comparison Procedures

Medical research is likely to fare poorly among reviewers when the statistical analysis is judged to be naive, and the purpose of this work is to clarify the approach taken in virtually all textbooks.

Abstract

Consider the situation where samples have been obtained randomly from each of five populations A, B, C, D and E. The question arises, 'How should the data be analyzed?'. A common answer to this question is 'Do one-way analysis of variance and, if P < .05, make pairwise comparisons using an appropriate multiple-comparison algorithm'. This is the approach presented in virtually all textbooks, subscribed to by most statisticians, and expected (if not required) by most journals. Conversely, the alternative answer, 'Do 10 two-sample t tests', would typically be considered naive and likely to elicit the response, 'You should consult a statistician'. Perhaps most importantly, medical research is likely to fare poorly among reviewers when the statistical analysis is judged to be naive. Thus the purpose of this