Unmixing Mixed‐Methods Research

Abstract

Over the last decade, mixed-methods research has become a growth industry. Legions of instructional texts on mixed-methods research (e.g., Bergman, 2008; Greene, 2007; Tashakkori & Teddlie, 2010) are now available and their numbers are rapidly increasing. First published in 2007, the Journal of Mixed Methods Research was ranked in 2012 4th out of 92 journals in impact in the Social Sciences/Interdisciplinary category (http://mmr.sagepub.com/). The National Institutes of Health (NIH; Creswell, Klassen, Plano Clark, & Smith for the Office of Behavioral and Social Sciences Research, 2011) and a host of scholars have published criteria or standards for appraising, conducting, and/or reporting mixed-methods research (e.g., Bryman, Becker, & Sempik, 2008; Heyvaert, Hannes, Maes, & Onghena, 2013). And, the Mixed Methods International Research Association (MMIRA; http://mmira.wildapricot.org/) was recently launched to promote the development of an international and interdisciplinary mixed methods research community, with an inaugural conference scheduled for 2014. Mixed-methods research has become an academic discipline and movement with its own experts, discourse, culture, and even "nationality," to which ever-growing numbers of researchers identifying with both qualitative and quantitative research are claiming "dual citizenship" (Cooper, Glaesser, Gomm, & Hammersley, 2012, p. 2). Mixed-methods research is also arguably yet another in a long line of methodological trends in vogue in the guise of methodological innovation. Claims to innovation are keys to success in the academic marketplace (Taylor & Coffey, 2009; Travers, 2009; Wiles, Crow, & Pain, 2011), whether addressing innovation in NIH grant proposals or claiming it in publications reporting the use or development of methods. After all is said, written, and read, however, mixed-methods research is better understood less as a new mode of inquiry than as a discursive re-packaging of the combinations of theoretical perspectives, methodologies, sampling strategies, and/or data collection, analysis, interpretation, and/or representation techniques encompassed in or constituting any empirical study. In this essay, I unmix mixed-methods research with a view toward "putting the mixed back into" (Onwuegbuzie, 2012) the understanding, conduct, and teaching of all health sciences research. As depicted in the ever-growing literature addressing it, the defining features of mixed-methods research are the (a) mixing of (b) qualitative (QL) and quantitative (QN) research conceived as two categorically different entities. Mixed-methods research is said to represent an approach to inquiry different from research that is conceived of as mono-method research, or as exclusively either QL or QN. Mixes of two or more research elements classified as QL (e.g., maximum variation and theoretical sampling, open-ended interviewing and participant observation, and content and discourse analyses), and mixes of two or more elements classified as QN (e.g., simple random and random cluster sampling, standardized questionnaires on anxiety and depression, and multivariate regression and path analyses), are usually not referred to as mixed-methods research, a term that is typically confined to mixes of at least one QL and one QN element typically in the same category (e.g., sampling, data collection, or data analysis). Mixed-methods research thus not only establishes a new binary—that is, mixed-/mono-method research—but also reinforces an old one, namely, QL/QN research. Beyond the foundational assumption that mixed-methods research represents a new approach to inquiry are several persistent and arguably misleading ideas about mixed-methods research. One is that only mixed-methods research can address diversity and complexity—for example, in phenomena, problems of concern, intervention programs, and stakeholders—in the behavioral, health, and social sciences (Creswell et al., 2011; Mertens, 2007). Yet mono-method QL or QN studies may be just as and even more complex than mixed-methods studies in the sense of being multi-faceted and requiring sophisticated skills, and more (e.g., data collection measures, data analysis techniques) is not necessarily better in the matter of research design. As Greene and Caracelli (1997, p. 5) noted, "multiple and diverse methods is a good idea, but is not automatically good science." A second persistent idea is that QL and QN research approaches offer different understandings of targeted problems and phenomena. Yet because philosophical/theoretical orientations to inquiry are not uniformly tied to any specific methodologies, techniques, or research designs, and because data may be conceived the same way regardless of how they were generated, mixed-methods research will not necessarily yield these different understandings. For example, as I proposed in more detail elsewhere (Sandelowski, 2011), a study might include both standardized closed-ended and highly structured questionnaires (typically conceived as QN data collection) and open-ended minimally structured interviews (typically conceived as QL data collection) addressing illness management. But if participants' responses to both of these data collection modalities are treated the same way—that is, as more or less accurate indices of participants' feelings, thoughts, and actions regarding illness management, as opposed to, for example, as scripted cultural performances—the result may be more information but not necessarily different perspectives on illness management. A third persistent assumption about mixed-methods research is that the respective strengths and weaknesses of QL or QN research approaches will offset each other. Yet strength and weakness are not attributes of research approaches but rather judgments researchers make about them (Sandelowski, 2012). The choice of combination of elements to use must be defended as meeting the specific objectives of a study, and it is the choice researchers make that will be judged as strong or weak. The term mixed in mixed-methods research is used variously to refer to combinations of research elements, often with no clear delineation of the type of mix and no acknowledgment that mono-method research is also characterized by mixes of elements or that research elements may be used in the same study without mixing them. The mixed in mixed-methods research signals one of three basic types of mixes, including the mere use together, linking, or actual integration of one or more QL and one or more QN elements at one or more time points in one study or in a program of research encompassing more than one study. The most common meeting points, or points of interface (Guest, 2013), between elements conceived as either QL or QN are at the methodological (e.g., randomized controlled trial and narrative study) and operational levels of inquiry, the latter including combinations of sampling (e.g., purposeful and probability) and/or data collection (e.g., standardized questionnaire and ethnographic interview), analysis (e.g., multiple regression and narrative analyses), interpretation (e.g., structural equation model and grounded theory), and representational (e.g., words, numbers, visual displays) techniques typically classified as either QL or QN. In the use-together type of mixed-methods research, QL and QN components remain separate, as when open-ended and minimally structured ethnographic interviews and closed-ended highly structured standardized questionnaires are used to generate/collect data. In the linking type of mixed-methods research, one or more QL and QN elements are placed in juxtaposition to each other but not merged, as when the results from the ethnographic interviews and standardized questionnaires are compared to ascertain their relationship to each other: that is, to determine whether they confirm, refute, or otherwise extend or modify each other. In the integration type of mixed methods research, QL and QN elements are assimilated into each other, as when data in one form are converted to another form to create a new data set. Qualitizing and quantitizing are the terms typically used to refer to the transformation or conversion of QN into QL data, and QL into QN data, respectively (Greene, 2007; Teddlie & Tashakkori, 2006). For example, the verbal themes or conceptual syntheses constituting the results derived from ethnographic interviews may be converted into variables to which numbers are assigned for statistical description or inference; or verbal summaries, profiles, or categories may be created from the statistical results derived from standardized questionnaires. The integration type of mixed-methods research is evident also when findings derived from QL or QN analyses are used as placeholders for each other. In this form of integration, a grounded theory derived from ethnographic interviews and observations is used to hold in place, or organize, house, or configure findings from a statistical analysis of responses to standardized questionnaires, or the findings derived from the statistical analysis of these responses are used to delineate further the grounded theory derived from the ethnographic interviews and observations. Any one mixed-methods study or program of research may entail mixes of these types of mixes (use-together, linking, or integration) at one or more time points. These mixes may be planned ahead of time or decided on as a result of data analysis. This is why the many typologies advanced for mixed-methods research designs (Nastasi, Hitchcock, & Brown, 2010) are somewhat misleading as they tend to treat the mixes of research elements as more deliberate, static, and mutually exclusive than they are in actual use (Guest, 2013). Even more misleading is that studies not referred to as mixed-methods typically entail such mixes. Ethnographies often entail mixes of diverse QL and/or QN data collection and analysis approaches, yet they are not typically conceived as mixed-methods research. Instrument development research typically entails cycling back and forth between QL and QN elements (e.g., narrative and cognitive interviewing, psychometric testing). In short, mixes of research elements and the challenges they pose are inherent to all empirical research design, not just to research designated as mixed-methods. For example, issues related to the timing (concurrent vs. sequential) and "priming effects" (Vitale, Armenakis, & Field, 2008) of mixing different research elements are not unique to mixed-methods research designs. In an ostensibly mono-method longitudinal QN study, for example, researchers still have to decide whether to administer the depression or anxiety inventory first or second and the likely influence of one inventory on the other on participant responses. Researchers still have to decide on the most appropriate time points for different data collection instruments. In short, researchers still have to address what Symonds and Gorard (2010, p. 132) described as the core elements of any empirical research design, namely, the "construction" of elements of the research process, the "transformation" of data from one form into another, the "influence" of one research element on another element, the "conceptual level" whereby different elements are used to address the research purposes, the "weight" given to the elements of the research process, and the "timing" of these elements in relation to each other. Although mixed-methods research originated in large part as an ecumenical move to transcend the often contentious QL/QN divide (Onwuegbuzie, 2012), it has served to reinstate it. The logo for the newly launched Mixed Methods International Research Association (MMIRA) depicts QL research on the left (suggesting a political stance?) with a steno pad, pen, and three pieces of a jigsaw puzzle, and QN research on the right (suggesting an opposing political stance?) with a bar graph and four numbers. Yet bar graphs and numbers also may be effectively used to represent QL elements, and steno pads, pens, and jigsaw puzzle pieces QN elements (Harvey, 2009; Onwuegbuzie & Dickinson, 2008; Slone, 2009). Indeed, the default to the QL/QN binary has been singled out as one (if not the) major impediment to mindful, inventive, and well-crafted inquiry (e.g., Allwood, 2012; Ercikan & Roth, 2006; Gorard, 2004; Sandelowski, Voils, & Knafl, 2009; Vogt, 2008). Both QL and QN research are still presented in instructional texts as either/or with sharp lines drawn between QL subjective/QN objective, QL inductive/QN deductive, QL word/QN number, and the like. These distinctions elide the commonalities between elements designated as QL or QN, and the enormous variations within the QL and QN research domains, respectively. The designations QL and QN are at best umbrella terms signifying a range of different and even conflicting elements, from research paradigms to research designs to techniques for sampling, data collection, and data analysis, and to modes of representation. Contributing to the within-QL and within-QN methodological diversity is the diversity in researchers from the practice, behavioral, social science, and arts and humanities disciplines, with highly varied and often conflicting ideas about the purpose and practice of QL and/or QN research. The QL/QN binary masks the reality that no ostensibly QN study escapes qualitizing, and few ostensibly QL studies escape quantitizing (Gorard, 2004; Martin, 2004; Maxwell, 2010; Sandelowski et al., 2009). Moreover, data are neither QL nor QN, but rather aspects of experiences or phenomena transformed into words, numbers, visual forms, and the like, each of which may in turn be transformed again into other forms (Valsiner, 2000). The idea, for example, that the closed-ended and highly structured questionnaire constitutes a QN element and the open-ended minimally structured interview a QL element effaces the countless variations in how questionnaires and interviews may be conceived, developed, conducted, or administered, in the purposes they are intended to fulfill, and in the way questionnaire and interview data may be analyzed, interpreted, and represented. Questionnaires may become more like interviews and the social interaction characterizing interviews when they are administered orally and in person, regardless of the training the person administering the questionnaire received to avoid such practices as helping respondents with or indicating (dis)approval of their answers. Interviews may become more like questionnaires and the reader/text interaction characterizing questionnaires when participants are asked to respond to questions without an interviewer present. Both questionnaires and interviews are typically conceived as ways to collect or generate data about a target phenomenon, but the questionnaire may also be conceived as a communication vehicle between an imagined researcher and imagined participant (Smith, 2008) and analyzed as a social interaction (Potter, 2003). The interview in turn may also be conceived as a cultural performance and vehicle for impression management (Rapley, 2001; Riessman, 2008). All behavioral and social science questionnaires may be viewed as QL because respondents interpret both verbal anchors and the numerical values attached to them (Sandelowski et al., 2009). Closed-ended highly structured questionnaires are typically used to draw formal generalizations from probability samples to populations, but they may also be used to draw particularizing generalizations from and about purposefully selected cases. Questionnaire items may be used as elicitation devices in narrative interviewing. Responses to both open- and closed-ended data collection devices may be analyzed via a range of diverse approaches, including varieties of content, narrative, conversation, and statistical analyses. In short, neither questionnaire nor interview can be easily categorized as QL or QN. Moreover, in variously encompassing QL and QN elements, each one may be seen as itself a mixed modality. The tendency to align the QL side of the QL/QN binary with such descriptors as subjective, inductive, verbal, and constructivist and the QN side with such descriptors as objective, deductive, numerical, and realist mistakes foreground/background features for categorical differences lying on one or the other side of the QL/QN divide. The subjective/objective binary, for example, signifies differences in emphasis in QL and QN discourses, not in the actual conduct of inquiry; that is, the disciplined subjectivity characterizing all empirical inquiry tends to be center stage in discourses about QL (both favorable and unfavorable to it), while it remains in the background in QN discourses (Sandelowski, 2011). To characterize QL research as inductive and QN research as deductive is to move to the background the iterative cycling between induction and deduction that characterizes all inquiry. The tendency to align QN but not QL with hypothesis testing, the interest in causation, the ability to generalize, and the emphasis on validity overlooks the varied conceptualizations and/or operationalizations of inquiry objectives common to all empirical inquiry. Hypothesis testing is integral to forms of grounded theory and analytic induction (Hammersley, 2012). QL approaches are often directed toward examining causal processes (Maxwell, 2004). Generalization is not defined exclusively by the ability to draw inferences from samples to populations (formal generalization), but also by the ability to draw inferences from and about individual cases (idiographic generalization) and from theoretical concepts and typologies (analytical generalization), and to apply concepts derived from the study of a phenomenon in one group to the understanding of a different phenomenon in a different group (transferability; Halkier, 2011; Larsson, 2009; Polit & Beck, 2010). The varied forms of validity (e.g., content, construct, descriptive, interpretive, theoretical; Maxwell, 2012) that are objectives in all manner of empirical inquiry are achieved by a variety of practices, the aspects of which often overlap. For example, there are aspects of cognitive interviewing that overlap with member checking and of reliability coding that overlap with negotiated consensus. At its best and most useful, mixed-methods research represents a novel discursive repackaging of the artful assemblages of research elements characterizing empirical inquiry in general. Indeed, the singular contribution of the mixed-methods research movement may be to remind researchers and teachers of research methods that mixing methods may just be the "natural way to conduct high quality relevant research" (Gorard, 2010, p. 64). The mixed-methods research movement may be said to have generated the "rebirth of research as a craft" (Symonds & Gorard, 2010): that is, the playful discipline and disciplined play whereby researchers creatively combine theories, methodologies, and/or materials to accommodate their specific research purposes. There are no "best practices" (Creswell et al., 2011) for designing, conducting, and evaluating mixed-methods research that do not apply also to mono-method research. Indeed, as mono-method research typically entails the mixing of elements, the distinction is itself called into question. In my effort to unmix mixed-methods research here, I am not positioning myself against mixed-methods research qua research, or proposing that theories and methodologies cannot or should not be distinguished from each other. Rather, I am encouraging the adoption of a mindset less focused on the "mixing" of methods conceived to lie on the QL/QN divide and more focused on which method and design assemblages make sense for answering significant research questions. I am urging an "organic" view of empirical research whereby the unity of a study or program of study is emphasized over its "methodological parts"(Song, Sandelowski, & Happ, 2010, p. 730). Regrettably, the emphasis on these methodological parts—especially evident in the "song and dance" (Symonds & Gorard, 2010, p. 131) of mixed-methods language and typologies researchers feel obliged to perform in their proposals and reports to establish their mixed-methods research bona fides—has increasingly taken priority over communicating clearly the significance of the research problem and what was or will be done to address it.