Editorial: Empirical elephants—Why multiple methods are essential to quality research in operations and supply chain management

Abstract

Vinod Singhal I agree with Kenneth Boyer and Morgan Swink that multiple methods are critical for doing high quality research in operations and supply chain management (OSCM). Ken and Morgan have clearly described the various empirical methods used in OSCM, the strengths and weaknesses of these methods, and examples where these method, individually or collectively, have been effectively used. The debate generated by this article will be useful in advancing empirical research in OSCM. I have two thoughts about empirical research in OSCM. First, although all four methods mentioned by Ken and Morgan have been used in empirical research in OSCM, the majority of empirical papers use survey-based methodologies. Its seems to me that researchers as well as respondents are suffering from survey fatigue, which has a direct impact on the quality of data received as well as the response rate. As a profession we need to consider how we can organize ourselves to develop a common database based on data from surveys and make it available to our community of researchers. This will make the data collection process more efficient, more reliable, allow researchers to more effectively build upon previous work, and our papers do not have to spend time justifying the reliability and robustness of data collection methods or the data collected. Our colleagues in Finance and Accounting have done this. Also, there are organizations that are in the business of developing, maintaining, and providing specialized data to academics. We could work with them to develop databases that are tailored to our research needs. Second, secondary data and survey-based methodologies complement each other. An appealing characteristic of secondary data is that it is more objective than survey-based data, particularly with respect to performance data. On the other hand secondary data is often not available at the level of detail that the researcher can use to measure the construct of interest. For example, it is not possible to measure constructs like leadership using secondary data and hence, one has to use perceptual data collected via surveys or interviews to measure this construct. Most of our colleagues understand the limitations of secondary data and are likely to be comfortable with the use of perceptual data to measure constructs such as leadership. However, many of our colleagues are uncomfortable and uneasy with the use of surveys and perceptual data to measure performance outcomes. Empirical researchers need to seriously consider how to combine perceptual data collected through surveys with secondary data on performance. This will require creativity as researchers will face constraints on the target population and the unit of analysis. Such efforts would be an excellent example where use of multiple methods can improve the quality and applicability of our research. Barbara B. Flynn It is heartening to think back to the time when we wrote the Flynn et al. (1990) article and to reflect on the evolution of research in operations and supply chain management over the past eighteen years. Empirical research has emerged as the dominant paradigm, something that I believe is completely appropriate for our field, which is, by definition, applied in its perspective. The state of knowledge in the field has advanced tremendously, in large part due to the application of what were considered non-traditional methodologies in the early 1990s. At the time that we wrote that article, we made several points that are perhaps even more valid today. First, it is important to avoid becoming method-bound and to apply the most appropriate methodology to suit the issue that is being studied. Rather than seeking issues that lend themselves to being studied by our favorite methodology, developing expertise in a variety of methodologies will allow us to study the most interesting and important problems in a way that will reveal the most insights. Second, we didn't invent any of the approaches that we described in the Flynn et al. (1990) article; we simply reported on the methodological approaches and development that had been done in other fields and contemplated how they could be applied to operations and supply chain management issues. As our research continues to develop, it is important that we continue to learn from what our colleagues in other areas are doing methodologically. I hope that we will see much more research that uses secondary data, case, experimental and other methodologies in the future. Third, like the blind men and the elephant, triangulation of approaches will give us a more realistic and thorough perspective. Although it is true that doctoral students are often cautioned against conducting case study research early in their careers, a dissertation provides the ideal opportunity to use several methodologies to build the most compelling case for the student's arguments and to demonstrate that the student is proficient in a variety of approaches. Using case study methodology will set the stage and help to develop stronger measures for a subsequent survey. Combined with a controlled experiment to hone into some of the findings from the survey or perhaps a mathematical model to analyze some of the findings, the doctoral student can present a strong case for both the research findings as well as his or her methodological expertise. Because of the variety of approaches that are relevant to research in cutting-edge topics, it is critical that doctoral students become bilingual (or perhaps even multilingual) in their methodologies. As we move into the future, many important issues relate to the global operations and supply chain management environment. It is important that we adapt our methods to a global context. Measures that were developed for use in a Western context may not be relevant in very different national cultures, both in terms of their wording and in terms of the underlying constructs that they represent. Farh et al. (2006) provide an excellent reference on this issue, comparing the translation, adaptation, contextualization and de-contextualization approaches to cross-cultural research. It is exciting to think about what the next 18 years of empirical research in operations and supply chain management research will hold as we apply new and combined methodological approaches to examine ever more interesting issues. Peter T. Ward As someone who has perceived the evidentiary elephant through touching primary data for many years, I feel obliged to comment. It seems clear to me that the editors' argument holds: our field benefits from the insights provided by different empirical approaches to evidence. It seems equally clear to me that a mostly unarticulated divide exists between researchers who typically use primary and those who use secondary data, collected by others, usually for another purpose. In my view, the divide is not really about the source of the data but rather about the orthodoxies and research traditions from which the researchers themselves have emerged. During the 1980s calls by leading researchers for empirical approaches in operations were common. For example, Miller and Graham (1981) synthesized the thoughts of a number of OM faculty in setting an agenda that called for OM researchers to leave their offices more often to observe the world around them and to address broader issues. Observing that OM researchers had little experience with such an approach, the authors suggested collaboration with researchers from other areas, particularly organizational behavior and business policy (p. 568). Later in the 1980s OM researchers were beginning to heed this advice as evidenced by the work of Swamidass and Newell (1987) and others that addressed strategic issues in operations empirically. Borrowing tools from researchers in the "management department" also meant that Ph.D. students would take courses that inevitably led to adopting a research tradition that focused on construct validity and linking the measures in question to a larger nomological network. In particular, the focus on using primary data also meant that researchers often were faced with devilish questions about the value of self reported performance data. Despite the apparent hazards of an empirical approach that came from the traditions of behavioral research, a fairly large number of OSCM researchers adopted it during the 1990s. Also during the 1990s, OSCM was greatly influenced by theories and analytic models that originated in economics. It seems natural that when OSCM economic modelers and their students developed empirical questions, they would turn to empirical methods from economics. The research traditions surrounding econometrics are quite different from those of behavioral research, with the focus on uncovering extant data to address pressing questions that emerge from theory. Good reasons exist for the different orthodoxies that have emerged in pursuing empirical questions in OSCM. The fact remains that these differences seem to prevent us from taking full advantage of the work of our colleagues in moving our field forward. Perhaps the first step is to acknowledge that either approach, when applied correctly, results in good research and that both are needed to adequately address the many managerially important issues that remain in our field. That is the elephant in the room. Aleda V. Roth Good reasons exist for the different orthodoxies that have emerged in pursuing empirical questions in operations and supply chain management. The fact remains that these differences seem to prevent us from taking full advantage of the work of our colleagues in moving our field forward. Perhaps the first step is to acknowledge that many approaches, when applied correctly, results in good research. Multiple methods are needed to adequately address the many managerially important issues that remain in our field. That is the elephant in the room. Among the primary purposes of empirical research in OSCM are descriptions and hypotheses pertaining to phenomena of interest to our discipline; and ultimately, to build, test, and refine theory (Roth, 2007). The underlying premise of the Ken Boyer's and Morgan Swink's "Empirical Elephants" paper is this: methods triangulation can improve our research quality—an idea derived from social sciences (Campbell and Fiske, 1959). The aims of triangulation are to offer convergence and completeness in the researchers' knowledge of a complex phenomenon. In conceptualizing triangulation in OSCM empirical research, there are several important issues that need further elaboration. First, scholars should be aware of the multiple types of triangulation. Take for example, Denzin's (1970) conceptualization, which includes the following categories: (1) data (e.g., sampling different entities and at different times), (2) investigator as research designer and interpreter (e.g., data collection and observations by different researchers), (3) theoretical (e.g., use of competing theories and research lenses), and (4) methodological (e.g., the use of different approaches). Others have expanded on Denzin's typology (Jick, 1983; Deacon et al., 1989). In contrast, the "Empirical Elephants" paper is presents a narrower focus 'among' methods triangulation and builds on the most prevalent approaches found in our literature (e.g., survey research, cases, and experiments), methods sources (i.e. primary vs. secondary collection), and hybrids thereof. The logic is this: OSCM empirical research would benefit from a convergence on the underlying phenomena (e.g., the elephant). Or is it this easy? Others have argued the contrary. My second point addresses the diversity of opinion regarding the implied benefits. Blaikie (1991) proposes that "triangulation means many things to many people." In social sciences, there is much philosophical debate about the value of multiple methods (e.g., positivists, who assume a single reality that can be captured by multiple mappings, vs. strict interpretivists, who view the scientific method as inadequate to study social reality that can only be hinted on through people's construction) (Roth, 2007). Interpretivists claim that there is a misplaced trust in the application of multiple methods (Massey, 1999). Not surprisingly, the methods of strict interpretivists are currently rare in empirical OSCM research. Even if positivist perspectives are held, which are common in our field, researchers should take special caution about being too strong in assertions when comparing the results among methods. OSCM scholars interested in methods triangulation would be advised to review the list of common error types given by Massey (1999). Care must be taken regarding interpretation and generalization of comparative results across different contexts and units of analysis. Further, in judging quality empirical research, arguably, a keener awareness of the differences within and between the various research traditions is needed. Having presented these two caveats, the future of OSCM empirical research is bright. Some unique opportunities for future research are now offered. First, there is a need for better metrics with good psychometric properties (Roth et al., 2008b). In moving the field forward, these authors make the case for a second generation of item and measurement scale development that are based on well-developed constructs, which form the nomological basis for OSCM theory development. Second, research that uses secondary data sources, which has not been collected directly by the researcher or is derived from archival data, often falls short on offering sufficient evidence of the validity and reliability (Roth et al., 2008a). Studies would benefit from a better understanding of the measurement properties and biases in commonly employed secondary sources as they pertain to OSCM. Third, many operations problems represent complex phenomena and algorithms, and insights from behavioral experiments can be informative, yet the researchers need to be vigilant regarding potential threats to validity (Campbell and Stanley, 1966). Research is needed, for example, on which types of problems may preclude the use of students as subjects, in favor of knowledgeable practitioners. Finally, I argue that the field would benefit enormously from studying phenomena based on combined analytical and empirical methods. In summary, I am most enthusiastic about the new OSM forum in JOM and the scholarly debates it will generate. Vishal Gaur Boyer and Swink, in their editorial article, encourage researchers to apply multiple methods to study operations and supply chain management phenomenon. The article focuses on empirical research: it shows examples of papers that combine methodological approaches in a single study as well as those that apply different methodologies to study a common topic. The messages are that all methodologies have something to offer, and furthermore, multiple methods are required to develop a holistic understanding. The ideas expressed by Boyer and Swink are invaluable to me as a researcher. There are many opportunities of research at the intersection of methods. For example, a researcher well versed in the methods of data collection is well placed to identify biases in secondary data and determine what types of secondary data should be collected. Conversely, researchers working with secondary data can identify questions that are worthy of detailed examination through surveys and case studies. Theoretical and empirical research can also inform each other. Theoretical research in operations generally deals with decision points, but empirical research tends to aggregate data at the level of an individual manager or firm. These gaps can be addressed by researchers who seek to use different methods in their work either individually or through collaborations. Boyer and Swink cite a few examples that can be used to develop research agendas. For example, case-based research on supplier–supplier relationships could be complemented by using secondary data for subject companies to examine the performance of different supplier–supplier relationships. The six blind men in the poem did not have a journal to go to. In our field, journals can take initiatives (and are most likely taking them) to reduce publication risk as well as create opportunities for synergy. For example, they can encourage papers that apply different methods to study questions researched before. Going further, they may also encourage papers that repeat previous studies using the same methodology but different data. One example of such research comes from marketing: many papers have investigated price and cross-price elasticities of demand for different types of products. Journals may also encourage interconnections between methods by selecting reviewers both on the basis of topic of paper as well as research method. For example, a researcher who used surveys to study supplier relationships may review a paper that uses secondary data for the same topic. This forum article is a useful intent of JOM. Many new types of data are becoming available in operations management. Thus, for many more questions than in the past, we can combine different methodologies to improve our understanding of operations phenomena. Massey and Walford (1999), Deacon et al. (1998).