The weakness of the strong/weak comparison of modes of inquiry

Abstract

A quick perusal of the methodological literature in print and on the web addressing qualitative, quantitative, and mixed-methods primary research and research synthesis will show how commonplace it still is to compare modes of inquiry on their strengths and weaknesses (e.g., Chronic Poverty Center, n.d.; Dixon-Woods, Agarwal, Jones, Young, & Sutton, 2005; Grant & Booth, 2009; Johnson, Onwuegbuzie, & Turner, 2007; Pope, Mays, & Popay, 2007; Speech Therapy Information and Resources, 2009). The strong/weak binary is central to the idea of a "hierarchy of evidence" (Petticrew & Roberts, 2006, p. 57), and it is used most often to compare qualitative and quantitative methods and to legitimize the need for and use of mixed-methods research, which is typically conceived as the combined use of qualitative and quantitative modes of inquiry (e.g., Creswell & Plano Clark, 2007; Lieber, 2009; Petros, 2011). The essence of the strong/weak comparison is that the weaknesses of one inquiry tradition or individual methodology or technique are offset by the strengths of another inquiry tradition or individual methodology or technique (hereafter all referred to as modes of inquiry). The strong/weak comparison has become such a familiar and taken-for-granted refrain in the methods literature that its use is hardly ever questioned. Whether conceived as entities such as qualitative and quantitative inquiry, experimental and ethnographic study, or as specific techniques for sampling, data collection, and data analysis, modes of inquiry have defining (albeit not fixed) attributes, but they have no inherent strengths or weaknesses. For example, to state that individual interviews allow more intensive exploration of individual experience than focus group interviews is to state something about the nature of these data collection techniques and thereby to signal their appropriate use; it is not to state that individual interviews are stronger in any absolute sense than focus group interviews. Individual interviews are different from focus group interviews by virtue of allowing the elicitation of more detailed individual narratives. Individual interviews are not stronger than focus group interviews because individual interviews are what they are, just as focus groups are not weaker than individual interviews because they are what they are. Accordingly, the choice to conduct both individual and focus group interviews in a study is not because the so-called weakness of one is offset by the so-called strength of the other, but rather because each of them permits different kinds of information to be elicited or generated and because researchers deem these different kinds of information to be necessary to achieve the aims of their studies. Moreover, a study is not stronger simply by virtue of including both kinds of data collection modes. A study is not stronger simply by virtue of being mixed-method (whether that is defined in terms of combinations of theoretical perspectives, methodologies, techniques of sampling, data collection, or analysis, or other such entities) as opposed to mono-method (however that is defined) unless strength is defined as the use of more than one theoretical perspective, methodology, technique of sampling, data collection, or analysis, or other such entities. Indeed, mixed-methods research has itself been described as weak in comparison to mono-method research by virtue of the diverse expertise and time it takes to conduct such studies (Youngs & Piggot-Irvine, 2011). This particular depiction of mixed-methods versus mono-method research entails the use of feasibility as an appropriate parameter for a strong/weak comparison. Indeed, to designate any one entity as stronger or weaker than another is to assume an a priori definition of what constitutes strong and weak and to make a judgment about the appropriate parameters for comparing the strengths and weaknesses of two or more entities. The view that a highly structured closed-ended questionnaire yields less information about the experiences of research participants than a minimally structured open-ended interview tends to be based on the assumption that fixed closed-ended data collection is weaker than emergent open-ended data collection in accessing the authentic experiences of research participants because the former confines respondents to answering certain questions in a certain way, while the latter allows interviewees to say anything they like in any way with respect to any aspect of the experiences targeted for study. In this example the notions of authenticity (more in minimally structured open-ended interviews/less in highly structured closed-ended questionnaires) and freedom (more in open-ended interviews/less in closed-ended questionnaires) are in play as standards against which interviews are judged to be stronger than questionnaires. Neither authenticity nor freedom, however, are attributes of these modes of data collection, rather they are parameters selected to compare these modes. Moreover, the fact that strong is equated with more not less authenticity and more not less freedom is also a judgment call. Equally arguable, for example, is that neither data collection mode yields more nor less authentic information about experience than the other (i.e., it is neither an appropriate nor useful parameter for comparison) and that having less freedom is stronger than having too much freedom in conveying information about a target experience. Here, the view that having less freedom to respond is a strength of the fixed closed-ended questionnaire in comparison to the emergent open-ended interview might in turn derive from the value placed on having a comparable data set whereby the (high versus low) comparability of data sets becomes yet another parameter of comparison contributing to the strong/weak judgment, this time favoring the fixed closed-ended questionnaire. Similarly, neither transparency nor objectivity—two commonly floated parameters of strong/weak comparisons of modes of inquiry (e.g., Dixon-Woods et al., 2005; Hiles, 2008; Miller, 2008; Silverman, 2001)—is a characteristic of any method but rather each is a judgment on the part of someone that transparency and objectivity ought to be standards against which to compare modes of inquiry and of how these terms ought to be defined. The regrettably persistent notion that qualitative research is weaker than quantitative research—because it is less transparent and objective—rests on a priori definitions of these terms and adherence to selected inquiry norms. As in the examples of authenticity and freedom described previously, whether one mode of inquiry is presented as stronger or weaker than another on the parameters of transparency or objectivity depends entirely on whether these are judged to be appropriate parameters of comparison at all, how these terms are defined, and whether more or less of either are considered to be indices of strength or weakness. Depending on these judgments, what is considered methodologically strong in one comparison might be considered weak in another. The assumption that one mode of inquiry is stronger or weaker than another is based also on the view that modes of inquiry have existences independent from each other and from the researchers who use them. Yet, as I have proposed elsewhere (Sandelowski, 2010), modes of inquiry have no clear boundary lines in research practice; for example, no quantitative studies escape qualitizing and few qualitative studies escape quantitizing (Sandelowski, Voils, & Knafl, 2009). Modes of inquiry also have no existence apart from users and user contexts as they are always situated, that is, accommodated (within boundaries acceptable to current communities of research practice) to individual studies and re-invented (also within boundaries acceptable to current communities of research practice) every time they are used. Researchers decide what modes of inquiry to use to accomplish their aims and how they will execute them. There are therefore no individual modes of inquiry per se, but rather modes of inquiry-in-use. In conclusion, the strong/weak comparison—whether explicitly deployed or implied—is yet another binary that like the qualitative/quantitative binary itself continues to bind. The terms strong and weak designate idiosyncratic judgments about modes of inquiry, not attributes of them. Indeed, the use of these terms to compare modes of inquiry unfortunately transforms judgments about modes of inquiry into fixed attributes of these modes. This transformation reproduces the idea that modes of inquiry are limited simply by being what they are. A limitation in the use of focus group interviews in a study might have been the failure to generate the full group interaction that communities of research practice currently see as characterizing the appropriate use of focus groups, not the generation of data in group interaction itself. Limitations of individual modes of inquiry ought to be conceived against prevailing standards for their proper use and execution, not against standards for the use and execution of other modes of inquiry. Researchers and instructors of research methods should abandon the strong/weak binary for the weak comparative device that it is.