The Problems in Using Fixed-Effects Models of Meta-Analysis on Real-World Data

Abstract

There are 2 approaches to meta-analysis: One assumes that studies in a meta-analysis are sampled from populations with the same effect size (the fixed-effects case), the other assumes that studies are taken from populations that have varying effect sizes (the random-effects case). As such, 2 corresponding meta-analytic frameworks have been developed: fixed- and random-effects methods. Recent evidence suggests that the assumption of fixed population effect sizes is not tenable for virtually all real-world data (e.g., Hunter & Schmidt, 2000), and yet fixed-effects methods of meta-analysis are routinely applied to real-world data (see National Research Council, 1992). This article describes some of the problems in using fixed-effects models on random-effects data by presenting 2 Monte Carlo simulations. In keeping with statistical theory (e.g., Hunter & Schmidt, 2000) results show a radical inflation of the significance tests of the mean effect sizes (above and beyond theoretical predictions). These results are discussed in terms of the implications for previously published meta-analytic reviews and those yet to be done.