You Can't Beat Good Experiments With Correlational Evidence: Mullen, Johnson, and Salas's Meta-Analytic Misinterpretations

Abstract

This article addresses the inconsistency in the conclusions of Mullen, Johnson, and Salas's meta-analysis with the findings of a series of experiments conducted by Diehl and Stroebe (1987). We first question whether metaanalytic procedures can be profitably applied to an area in which a relatively small number of studies manipulated a limited number of variables and yielded highly consistent results. We then contrast the meta-analytic approach of Mullen et al. with the experimental approach of Diehl and Stroebe to argue that Mullen et al.'s conclusion that the influence of procedural and economic mechanisms on the productivity loss in brainstorming is erroneous. It is proposed that Mullen et al.'s analysis does not allow such a conclusion because they merely examine the effects of four rather arbitrarily selected variables on the difference between real and nominal groups. Because real groups always work under blocking conditions-whereas blocking does not occur in nominal groups-their findings merely reflect the extent to which these variables modulate the central procedural mechanism.