Diagnostic Procedures for Research Synthesis Models

This chapter describes a few methods for discovering potential sources of poor fit to fixed effects models by providing methods for recognizing one or more estimates that deviate greatly from their expected values if the model were correct.

Abstract

This chapter describes a few methods for discovering potential sources of poor fit to fixed effects models. These diagnostic procedures provide methods for recognizing one or more estimates that deviate greatly from their expected values if the model were correct. These procedures often point to studies that differ from others in ways that are remediable; for example, they may represent mistakes in coding or calculation. Sometimes the diagnostic procedures point to sets of studies that differ in a collective way that suggests a new explanatory variable. Sometimes a study is an outlier that cannot be explained by an obvious characteristic of the study. The analysis of data containing a few observations that are outliers is a complicated task. It invariably requires the use of good judgment and decisions that are, in some sense, compromises. There are two extreme positions on dealing with outliers: (1) data are sacred, and no datum point (study) should ever be set aside for any reason and (2) data should be tested for outliers, and data points (studies) that fail to conform to the hypothesized model should be removed.