Common Methodological, Analytical, and Interpretational Errorsin Published Educational Studies: An Analysis of the 1998 Volume of the British Journal of Educational Psychology

Abstract

The purpose of the present study was to identify and to discuss the major analytical and interpretational errors that occur regularly in quantitative and qualitative educational research. To this end, all 36 research articles published in the 1998 volume of the British Journal of Educational Psychology were examined Findings revealed that the univariate analysis of variance (ANOVA) F test was the most commonly used technique-- represented in 38.9% of research articles. The multiple analysis of variance (MANOVA) test was the next most popular technique utilized (22.2%), followed closely by exploratory factor analysis (19.4%). A content analysis revealed the following eight errors made with respect to statistical analyses: (1) no evidence provided that statistical assumptions were checked; (2) no power/sample size considerations discussed, (3) a MANO VA followed by an ANO VA; (4) no effect sizes reported, (5) effect sizes not reported for some inferential analyses; (6) no previous reliability indices provided; (7) no reliability indices provided for the present sample; and (8) no control for Type I error made. Implications are discussed. There is little doubt that educational research worldwide has influenced and informed educational practice. The last decade has seen a proliferation in the number of articles published in educational research journals. Unfortunately, some researchers have found that the majority of published studies and dissertations are seriously flawed, containing analytical and interpretational errors (Hall, Ward, & Comer, 1988; Thompson, 1998a; Vockell & Asher, 1974; Ward, Hall, & Schramm, 1975). Some of these flaws have arisen from graduate-level instruction in which research methodology and statistics are taught as a series of routine steps, rather than as a holistic, reflective, integrative process (Newman & Benz, 1998); from graduate-level curricula that minimize students' exposure to quantitative and qualitative content (Thompson, 1998a); from increasing numbers of research methodology instructors teaching out of their specialty areas; and from a failure, unwillingness, or even refusal to recognize that analytical and interpretational techniques that were popular in previous decades no longer reflect best practices and, moreover, may now be deemed inappropriate, invalid, or obsolete (Schmidt & Hunter, 1997). At the highest level, these errors include creating a false dichotomy between quantitative and qualitative research methodologies; that is, failing to treat quantitative and qualitative research strategies as lying on an interactive continuum, with theory as the driving force. This practice tends to prevent researchers from taking a holistic and comprehensive approach to research (Newman & Benz, 1998). With respect to qualitative research methodologies, analytical errors include a failure, often for philosophical reasons, to legitimize research findings and interpretations through documentation of validity (e.g., credibility, relativism, external criticism) and reliability (e.g., interrater reliability, internal criticism). Interpretative errors include the tendency to generalize findings rather than to use qualitative techniques to obtain insights into particular educational, social, and familial processes and practices that exist within a specific location (Connolly, 1998). With respect to quantitative research methodologies, perhaps the most common analytical/interpretational error stems from a failure to realize that all parametric analyses (i.e., univariate and multivariate techniques) are subsumed by a general linear model, and that, consequently, all analyses are correlational (Cohen, 1968; Knapp, 1978; Onwuegbuzie & Daniel, in press; Thompson, 1998a). Lacking this knowledge, many researchers inappropriately categorize variables in non-- experimental designs using analysis of variance (ANOVA) or other OVA analyses, in an attempt to justify making causal-inferences, when all that occurred typically is a discarding of relevant variance (Cliff, 1987; Pedhazur, 1982; Thompson, 1992). …