Abstract
Mixed methods research (MMR) has become increasingly popular in recent years. Yet, methodological challenges of mixing qualitative and quantitative data remain. Understanding how MMR is approached in qualitative research journals provides insights into lingering mixing issues. In this article, we content analyzed five leading qualitative research journals from 2003 to 2014, which represents the reflective period of MMR. Of the 5,254 articles published, 94, or 1.79%, were mixed methods in nature, comprising 44 theoretically oriented articles and 50 empirical articles. In terms of theoretical articles, five content-based themes were identified: (a) MMR advocacy, (b) philosophy issues, (c) procedural suggestions, (d) practical issues and best practices, and (e) future directions. In terms of empirical articles, 36% used exploratory sequential designs, primarily to develop instruments, and 52% explicitly identified as MMR. None of the studies included MMR questions, and development (21%) and complementarity (14%) were the primary rationales for mixing. In virtually all studies (98%), mixing occurred at the data interpretation stage through some comparison of qualitative and quantitative research. Qualitative data were prioritized in 86% of the studies. Based on these findings, it appears that MMR affects qualitative research most directly by influencing study design and study purpose; however, there is a strong tendency to conduct and publish qualitative and quantitative studies separately. Recommendations for publishing future MMR are discussed.
Keywords
Mixed methods research (MMR) has become increasingly popular over the last 25 years (Creswell, 2015). However, collecting qualitative and quantitative data was commonplace in many social sciences throughout the first 60 years of the 20th Century. During the 1980's, MMR re-emerged as a distinct approach, inducing a second wave of popularity (Creswell, 2015; Guest, 2013; Johnson, Onwuegbuzie & Turner, 2007). This period often is regarded as the beginning of contemporary MMR practice and the third research paradigm (Tashakkori & Teddlie, 2010). After gaining traction through this formative period and the ensuing era of paradigm debate, MMR expanded into a distinct methodology equipped with procedures and nomenclature (Creswell & Plano Clark, 2011). There is now little question of the legitimacy and utility of MMR in various sectors of inquiry (e.g., health and social sciences).
Still, challenges remain (Bazeley, 2002). Such challenges exist at multiple levels, including procedural and methodological levels (Archibald, 2015; Bryman, 2006b). For instance, to what extent can qualitative and quantitative research be mixed without violating paradigmatic assumptions of each approach? To what extent should paradigmatic considerations infiltrate the procedural aspects of mixing? Questions of how mixing actually occurs within studies might trigger methodological concerns for certain practitioners while triggering procedural concerns for others. How MMR is understood and defined is fundamental to these considerations.
Indeed, defining MMR is a debated topic in the field. Diverse definitions have been forwarded and refined as MMR studies and publications have emerged, and its procedural aspects established (Johnson et al., 2007; Tashakkori & Teddlie, 1998). Some purport that MMR involves the collection and integration of qualitative and quantitative data in a single or a series of studies (Johnson et al., 2007). Within this view, mixing can occur at the level of methods (e.g., Creswell, 2015), methodology (e.g., Tashakkori & Teddlie, 1998), or across disciplines (Bazeley, 2002). Others regard the mixing of diverse qualitative approaches as MMR (e.g., Morse & Niehaus, 2009), whereas a third group notes that the mixing metaphor can be a misnomer (Bazeley & Kemp, 2012; Johnson et al., 2007). Multiple quantitative research approaches in a single study have historically been termed multi-method research (Campbell & Fiske, 1959), whereas Tashakkori and Teddlie (1998) preferred the term mixed model research when integrating beyond the level of methods. In light of this diversity, it is imperative to clarify what is actually being mixed within MMR studies, and how this mixing has occurred (Bazeley, 2002).
The variability in MMR definitions mirrors the diversity in its application. How MMR is applied across disciplines, geographic contexts, and within particular research traditions varies extensively. To date, analyses of actual MMR procedures across disciplines are scant (Hesse-Biber, 2010, p. 15). Recent work has uncovered practices that illuminate state-of-the-science of MMR in the reflective period, circa 2003 to the present (e.g., Bryman, 2006c; Creswell & Plano Clark, 2011; Zhang & Creswell, 2013). Although important strides towards delineating high-quality MMR studies have been made, there are additional gains to be made in understanding how MMR is approached in journals aligned specifically with one research tradition (e.g., qualitative). Despite this, there has been no empirical study of MMR practices within qualitatively oriented research journals. As such, we conducted the present study on MMR practices in qualitative research journals.
Qualitative and quantitative research can inform one another. For example, qualitatively driven MMR designs, such as those of an exploratory sequential nature, help identify variables for testing within quantitative research designs, contribute to modifying existing tools or to developing new tools or measures, and provide context and meaning to numerical values in a self-report measure (e.g., Mason, 2006). However, investigating precisely how MMR is approached in qualitative research journals is necessary to provoke reflection upon beneficial and problematic practices that aid or impede the fields' development and evolution. Understanding the actual practices of data integration, reporting of mixed methods sampling procedures, and using visual displays to illustrate methodological complexities provides insights into the state-of-the-science of MMR. This information is important to investigators seeking publishing venues, to researchers seeking understanding of MMR procedures, and to MMR educators identifying domains where MMR skills-gaps persist and additional training might be required. Such inquiry also might facilitate understanding of pertinent issues to the field of MMR, such as those related to typological designs and integration, which are perennial and generate considerable discussion.
Analyzing how MMR is approached in qualitative research journals is also relevant because MMR is increasingly published in non-mixed methods specific journals (i.e., Journal of Mixed Methods Research, International Journal of Multiple Research Approaches). Moreover, MMR is published in a wide array of venues, such as those with a largely qualitative or quantitative research orientation. Trainees and novice researchers who are conducting MMR might look to published articles for guidance, particularly because the formal MMR training might not be available (Archibald, 2015); however, whether such articles are worthy exemplars is not yet established. Therefore, we believe that this work provides the literature with a preliminary description of theoretical and empirical MMR being published in primarily qualitative research journals.
Three research questions guided this study and set the stage for the results and discussion: Is MMR published in leading qualitative research journals? If so, how is it approached? How has MMR influenced qualitative research, more broadly, if at all? Additionally, we posed the following eight sub-questions: Do authors of published articles identify them as MMR? How is MMR defined? How are research questions approached? How are research designs approached? At the procedural level, what is the function of integrating? How does ‘mixing’ occur? What recommendations regarding MMR are made? And finally, to what extent are political dimensions discussed? We explored these questions differently, depending on whether articles were theoretical or empirical in nature. We integrated our findings with our interpretations under each associated question heading.
Method
Researcher-as-Instrument: Our Research Team
Our research team consisted of three female doctoral candidates and one male faculty member. The graduate students were in nursing, school and child clinical psychology, and measurement and cognition. The faculty member was in counselling psychology. Two of us self-identified as White European Canadian, one of us self-identified as Chinese, and one other member self-identified as White European American. All four of us were, at the time of the study, involved actively in various qualitative, quantitative, and MM research studies, as well as a campus-based Mixed Methods Interest Group (MMIG). Generally speaking, as a team, we value each methodology equally and do not adhere to a single ontology or epistemology, although one of us had statistically significantly more experience and training in quantitative methods. Before conducting the study, we discussed our research experiences, values, and biases as part of the MMIG. We also discussed community of practice issues regarding MMR at length. Further, prior to coding published studies, we discussed the rating sheet and potential challenges.
Throughout, we employed a mixed methods (MM) approach to reviewing published articles (see Harden & Thomas, 2010). Ours constituted a MM approach in two ways: first, we included studies of various methodologies in the review (e.g., MMR of various typologies; qualitative research studies included in an overarching MM program of research; theoretical writings on MMR). Second, we approached the synthesis of different data sources using separate, and then combined, synthesis methods. Unlike Harden and Thomas (2010), who model multiple syntheses including statistical meta-analysis of quantitative findings, the nature of our review was not amendable to such statistical rendering. However, we still conducted three distinct syntheses, comprising coding and descriptively analyzing domains from MMR empirical studies, conducting thematic analyses of recommendations and definitions provided in empirical and theoretical articles, and, when possible, integrating the two aforementioned syntheses at a comparative level.
Sample / Published Articles
The six qualitative research journals reviewed were identified through team discussion and comprised the following: International Journal of Qualitative Methods (IJQM), Qualitative Health Research (QHR), Qualitative Research (QR), Qualitative Inquiry (QI), The Qualitative Report (TQR), and Forum: Qualitative Social Research (FQS). These journals were selected based on the team's awareness of their influence, their broad international audience (e.g., FQS; IJQM; QR), comparatively high Journal Impact Factors (e.g., QHR; QR) and generally, for their non-disciplinary orientations. Although we were aware that QHR is a health-oriented journal, it is not specific to one health discipline and was, therefore, included.
A total of 5,254 articles were hand searched and reviewed. Initially, 105 articles were identified as eligible (n = 63 empirical and n = 54 theoretical, conceptual, methodological or empirical). Following primary and secondary screening procedures, 44 theoretical and 50 empirical articles met the inclusion criteria and were included in the content analysis (CA). Collectively, these journals published 94 MMR empirical studies or related theoretical articles between January 2003 and January 2014 (inclusive), representing 1.79% of the entire population of published articles (N = 5,254). A flow diagram of study inclusion is provided in Figure 1. During this screening process, we located other articles relevant to MMR that did not meet our inclusion criteria (e.g., Bryman, 2006b; O'Cathain, Murphy, & Nicholl, 2008). For instance, research syntheses that did not use a MM approach to reviewing the literature, articles where multiple qualitative methods or multiple quantitative approaches were used, and articles where the integration of qualitative and quantitative approaches were not discussed or demonstrated were all excluded from our review.
Flow chart of article inclusion.
Procedure
We hand-searched journals for keywords of “mixed” and “qualitative and quantitative” published between January 2003 and January 2014, inclusive. Inclusivity was the guiding principle at this stage of article selection. Once potential articles were identified, they were reviewed by a second independent reviewer for preliminary eligibility. To be included at this stage, empirical articles needed to include both a qualitative and quantitative component or self-identify as MMR. Theoretical, conceptual, methodological, or editorial articles that mentioned MMR in any capacity, and book reviews about MMR also were included. This latter class of manuscripts are broadly referred to as theoretical throughout the remainder of the manuscript.
Results and Discussion
Is MMR Published in Leading Qualitative Research Journals?
Yes, but very infrequently. In this sample of studies, less than 2% of published articles were MMR. A total of 44 theoretical articles and 50 empirical articles were identified. The vast majority of theoretical articles (82%) were methodological or conceptual in nature. Four editorials (9%) and four reviews (9%) also were identified. The distribution of theoretical and editorial articles was not statistically significant among journals (χ 2 = 5.09, p = 0.41); there was no association between journal and article type (theoretical or editorial). TQR was the most common venue, publishing 11 articles (nine theoretical and two reviews). QI published nine articles (eight theoretical and one editorial); FQS published eight articles (six theoretical and two reviews); QHR published seven articles (five theoretical and two editorials); QR published six articles (five theoretical and one editorial) and IJQM published three articles (all theoretical), respectively.
Distinct peaks were identified when MMR articles were examined by year of publication. There was an overall increase in MMR published since 2003. Publishing of theoretical articles peaked in 2006 and, five years later, empirical articles peaked (see Figure 2).
Mixed methods articles by year of publication.
Separate trend analyses were conducted to reveal patterns in the two types of articles (i.e., empirical and theoretical), with the number of articles as dependent variable and year (year; year and year square; year, year square and year cubic) as the independent variable. There was a statistically significant positive linear trend across the years among empirical articles (t = 2.86, p < .05; Cohen's [1988] d = 2.02), indicating that frequencies for the empirical articles increase in a linear fashion over the time period of analysis (2003–2014). No statistically significant trend in any of the linear (t = −0.22, p = 0.83), quadratic (t = −1.84, p = 0.10) or cubic (t = −0.24, p = .82) patterns were present among theoretical articles, indicating that there is no trend pattern for publishing theoretical articles among these journals. The results of these analyses are presented in Table 1.
Summary of trend analyses of numbers of articles across the years (N = 22)
Note. * represents a statistically significant trend pattern; p < .05
How is MMR Approached in Leading Qualitative Research Journals?
In order to be included in this CA, MMR needed to be mentioned in theoretical articles. However, during our initial article screening, it became evident that authors varied in the extent to which they discussed MMR. Theoretical articles, therefore, were coded according to their relative focus on MMR. We wanted to capture whether articles had an explicit focus on MMR versus merely mentioning MMR within the context of a larger discussion or, finally, not referring explicitly to MMR at all. Of the MMR articles published in QI, 100% had an exclusive focus on MMR, followed by QHR at 71%, TQR at 64%, QR at 50%, IJQM at 33% and FQS at 13% (χ 2 = 12.31, p < 0.05; V = 0.31), indicating a statistically significant difference in whether journals focused exclusively on MMR. Similarly, we found the empirical works published in FQS to be among the least likely to self-identify as MMR or be among the least likely to include “mixed methods” in the research title. Perhaps this is a function of journal editorialship.
Comparative focus of mixed methods definitions.
The discrepancy in how MMR is defined across empirical and theoretical manuscripts was interesting in so much as it mirrors longstanding debates around MMR definitions (Johnson et al., 2007). Similar to Creswell's (2009) insight that probing of definitions seemed to have ceased in conference reports, MM definitions were not a dominant topic of inquiry in our review. Also, we considered that the longstanding history of MM, particularly within the discipline of sociology (Gilbert, 2006; Greene, Caracelli, & Graham, 1989; Plano Clark, 2010) might have contributed to a persistent regard for MMR as a combination of methods, techniques, or procedures, rather than as a methodology. Alternatively, a methods-based approach to MMR might be regarded as being particularly useful as an introduction to the field because it provides a more concrete and accessible entry-point than a methodological orientation (Creswell, 2015).
Leaders in the field have historically overlooked the procedures and implications of devising a well-formed MMR question (Onwuegbuzie & Leech, 2006; Plano Clark & Badiee, 2010). Recent attention to developing and identifying MMR questions (e.g., Creswell, 2015; Plano Clark & Badiee, 2010) has provided excellent guidance in this regard. Including how MMR questions in published reports would advantage the reader by acting as a signpost, providing important hints and direction regarding the likelihood of a particular MMR design, the associated sequence and timing of data collection, and probable types of data analysis and integration procedures utilized (Creswell, 2015; Onwuegbuzie & Leech, 2006). Researchers are encouraged to attend closely to MMR questions and to familiarize themselves with available resources.
The overwhelming majority (92%) of the 26 studies that self-identified as MMR explicitly utilized a MMR design typology. This high occurrence might reflect the proliferation of such typologies in the MMR literature (e.g., Creswell & Plano Clark, 2011; Greene et al., 1989; Morse & Niehaus, 2009). Typologies, as tools for research design and pedagogy, might offer the additional advantages of a common language, structure, and legitimization to the field of MMR (Guest, 2013). Yet, typologies also might obscure temporal order and render the point of integration unclear to the reader, depending on the point of emphasis (e.g., data collection or analysis) (Guest, 2013).
Although we did encounter a lack of clarity related to the point of integration and temporal order, we were also challenged at times to understand the context of particular manuscripts, because 46% of the articles were identified as being part of a larger study, or program of study. This typically meant that quantitative results from the study were reviewed briefly and then qualitative findings were explored more extensively. When MM studies are published separately, it is more difficult to assess, or even identify, the actual point of integration and, in turn, the study's typology.
Our impression of this sample of studies was that, generally, researchers emphasized a fixed approach to study design. Only 6% of the empirical works appeared emergent in nature and that, overwhelmingly, manuscripts conveyed a message of design fidelity or adherence to the initial research plan. We were at times unclear whether this tendency reflected a social desirability for cleanliness in research reporting, considering that the process and outcomes of MMR are not always predictable (Bryman, 2006a), and qualitatively driven MMR may be particularly constructivist and organic in nature.
Approximately one half (52%) of the empirical articles provided a rationale for using and integrating qualitative and quantitative research methods. However, these were often implicit or embedded within the study's purpose. The rationales provided are displayed in order of occurrence by primary rationale, in Table 2.
Rationales for Integrating Qualitative and Quantitative Research
Development was the most common rationale provided, evidenced in 21% of empirical studies, followed by complementarity at 14%. The high occurrence of development as a rationale corresponded with the large volume of instrument development models. Triangulation was more common than expansion and initiation. Because previous analyses have reported that initiation is uncommon, this was somewhat of an expected finding. We did find the low occurrence of expansion to be an unexpected result, particularly considering that 66% of our study sample reflected sequential designs using distinct methods for separate, but related, inquiry components.
This finding prompted us to question whether there was a potential misalignment between the stated rationales and the actual conduct of mixing qualitative and quantitative research. We were less concerned with studies claiming development as the study rationale; indeed, there was some internal consistency between this finding and the occurrence of instrument development models. Further, as Bryman (2006b) previously illustrated, “when ‘instrument development’ and ‘sampling’ are the rationales, they are nearly always used in this way” (p. 110). Other rationales, particularly triangulation, complementarity, and expansion, might be less clear because overlap exists between their operational components. This might lead to ambiguity in interpreting these functions—one need only refer to the extensive literature on the ambiguity surrounding triangulation to substantiate this insight (e.g., Denzin, 2012; Wolf, 2010). Although it is beyond the scope of this study, future work that examines the correspondence between stated rationales, such as triangulation and expansion, with actual integration procedures, would contribute further insights to this domain.
Because our CA focuses on how MMR is approached in qualitative research journals, it is perhaps not surprising that the included manuscripts favoured publication of the qualitative aspects of MMR. Indeed, drawing integrated inferences, or referring to how qualitative and quantitative components related, was the most common level of mixing, occurring in 98% of studies. Publishing practices might have influenced how authors approached integration because qualitative and quantitative components often were contained in separate publications, particularly because of the dominance of sequential designs. This is aligned with the historical focus on component designs and the relative lack of quality examples illustrating fully integrated designs in the MMR literature (Bazeley, 2002; Bazeley & Kemp, 2012). The emphasis on component designs has prompted some to question whether “these really do constitute a mixed methods study or rather, are two separate studies which happen to be about the same topic” (Bazeley, 2002, p. 3). We also noted at times, studies of other design typologies (e.g., embedded) that self-identified as MMR but demonstrated the same disconnected publishing practices.
For example, in their study of burnout and coping of occupational therapists, Gupta, Paterson, von Zweck, and Lysaght (2012) utilized a “mixed methods approach to gain a comprehensive view of stressors” (p. 4). The authors used a concurrent embedded model with a predominantly quantitative focus and reported their quantitative and qualitative results in separate manuscripts. Focus group data were analysed using a hermeneutic approach; the analysis and findings of which were included in the present review. Although the authors use a comprehensiveness rationale for employing MMR, how qualitative and quantitative components contributed to this objective was insufficiently described. Quantitative findings are “presented elsewhere” (p. 6) and data are not presented in a manner that facilitates comparison (e.g., side-by-side). In fact, only one reference is made to the accompanying quantitative research manuscript.
Mixing at the level of data, including crossover data transformation procedures was evident in 22% of studies. This most often occurred within convergent designs. Exploratory designs (28%) occasionally mixed their data, typically through transforming qualitative data into quantitative data. Integrating at the level of methodology also was fairly common, evident in 10 studies. An interesting example of this approach included the interface of phenomenology within a controlled clinical trial (Bishop, Jacobson, Shaw, & Kaptchuk. 2012).
The weighting of qualitative and quantitative study components is another relevant dimension of MMR (Bryman, 2006b). Not surprisingly, the vast majority of studies (86%) prioritized qualitative data. This occurred across research designs. Only 10% of studies provided equal weighting to quantitative and qualitative data, as evidenced by equal credence being paid to each data strand in the article as well as both data strands influencing inferences. The remaining 4% of studies provided inadequate reporting details to ascertain weighting or, in one case, prioritized quantitative data. The proportions among these three weighting types were statistically significantly different (χ 2 = 100.08, p < 0.001; V = 0.82), indicating a contrast between the expected proportions and what we observed.
Where integration occurred (i.e., the point of integration) varied extensively across studies. Almost one quarter of studies (22%) integrated by way of developing an instrument or data collection tool, meaning that the research findings of one strand contributed to creating a new measure or improving an existing one. The extent to which qualitative findings were discussed in relation to the tools development varied but generally was well described. For instance, Lutz, Kneipp, and Means (2009) reported on the first of three phases of a community-based participatory research study designed to develop a culturally appropriate health-screening questionnaire. They conducted three rounds of focus groups that iteratively informed the development of constructs and process components related to their screening measure, and described this process in detail. Subsequent stages of the study (i.e., assessing the questionnaire's acceptability through a survey methodology and eventual use in a randomized controlled trial) were outlined as successive stages of study. Interestingly, the authors did not identify this as a MM program of study.
Data collection was a common point of integration, present in 14% of articles, a finding consistent with previous reports (e.g., Hesse-Biber, 2010). For instance, Capezza (2003) examined the cultural-psychological foundations of violence and non-violence by collecting questionnaire, narrative, and observational data simultaneously. Integrating during data analysis was common and documented in 30% of studies. At this level, data were cross-compared (for example) in relation to convergent or divergent findings. Overall, convergent designs tended to demonstrate integration earlier in the research process and 71% of these designs began integrating at the point of data collection.
Given the abundance of sequential designs and the presence of multiphase designs, it is not surprising that 86% of studies collected data in a sequential manner. As a result, most studies in this sample used a connected approach to data analysis wherein one data strand leads to the other. Concurrent data collection was less common and was noted in 14% of studies, all of which were convergent designs (χ 2 = 49.48, p < 0.001; V = 0.70); these differences were statistically significant. Merged (12%) and embedded (4%) approaches to data analysis were infrequent in this sample, consistent with the high representation of sequential approaches to data analysis observed. Merged data analysis occurred within convergent designs only. In one article, the analysis procedures were impossible to decipher because of insufficient reporting of the quantitative aspect of the study.
In contrast to the empirical studies, recommendations were made in a greater proportion (45%) of theoretical articles and many authors forwarded multiple recommendations. Five themes were identified through open coding and comprised: (a) advocacy (i.e., advocating for the benefits of MMR or related approaches); (b) foundational principles (i.e., those aspects or beliefs necessary to or underlying MMR); (c) procedures (i.e., recommendations related to the “how-to” or technical aspects of MMR); (d) reflexivity (i.e., reflecting on some aspect of MMR or practice); or (e) future directions (i.e., possibilities for enhancing MM research or practice).
MMR procedures were the most common of the five themes identified through the analysis of recommendations in theoretical articles, forwarded by 11 authors. Four subcategories were identified under the MMR procedure theme and included: (a) MMR designs; (b) research problems or questions; (c) language or reporting; and (d) analysis procedures. Multiple recommendations were common within this theme. Of these, language and reporting were most common, present in five recommendations. Design related recommendations and those related to research problems or questions were present in four recommendations each. Three recommendations were specific to data analysis, only one of which pertained to data integration. For instance, Moran-Ellis et al. (2006) make a number of arguments and recommendations pertaining to the meaning of integration, terminology, and justifications. In total, an additional five recommendations were made speaking to sampling, team research, validity, utility, and clarity in understanding, respectively.
In the theoretical articles, 10 recommendations were made related to foundational principles of MMR. Most often, these related to theoretical or methodological considerations (n = 6 recommendations). For example, Shaw (2003) identified that epistemological integrity is necessary for meaningful and well-conducted research. Similarly, Miller and Fredericks (2006) recommended that, “methodological, structural, and substantive concerns must be recognized and addressed before MMR can be integrated into evaluation research” (p. 574). Ensuring that researchers have adequate training or skills to conduct MMR were present in three recommendations. Awareness of research processes or current issues in the field of MMR and ethical considerations were present in three and two recommendations, respectively. A visual representation of these recommendations is provided in Figure 4.
Pie chart illustrating the recommendations in theoretical articles
Ethical considerations of MMR were raised in 9% of theoretical articles. Ethics were discussed in diverse manners in these sources. For instance, Mertens (2010) refers to the ethical dimensions of social justice and human rights in MMR, discussing how the transformative paradigm has enabled her to clarify the ethical dimensions of her practice. Conversely, Christ (2014) discusses how ethical considerations invariably infiltrate traditional randomized controlled trials (RCT), thereby advocating for the inclusion of MMR approaches into the RCT design.
When Creswell (2009) provided a map of the MMR terrain, he anticipated increased attention to the politicized dimensions of MMR. Our findings suggest that the politicized dimensions, particularly deconstructing MMR, are indeed a focal aspect of inquiry into politicization. Although subtle, we were able to detect an approximately 30% increase in the proportion of articles attending to deconstructing MMR (and to a lesser degree, ethics) beginning in 2009. Indeed, 2009/2010 seem to be a tipping point for theoretical work relating to deconstructing MMR and the ethics of MMR, with seven articles about these topics published over these 2 years; five were published between 2003 and 2008 and four were published between 2011 and 2014. However, quality articles attending to these concepts also were noted prior to this point. Useful examples of such articles include Onwuegbuzie and Leech's (2004) exploration of the meanings of significance in relation to MMR, and Moran-Ellis et al.'s (2006) critique of integration and mixing in the MMR context.
How Has MMR Influenced Qualitative Research, More Broadly, If At All?
This CA has highlighted numerous ways in which MMR influences qualitative research, and also stimulated further hypotheses in this regard. We question whether the rise in popularity of the generic qualitative research method and the tendency to mix within and between qualitative research methodologies coincided with the re-emergence and increasing popularity of MMR as a methodological approach (e.g., Scott, Archibald, Pullishy, & Chambers, in press). Debates surrounding paradigmatic compatibility also exist within the qualitative domain, enabling alternatives to methodological muddling, methodological purity, and divisive methodological distinctions (Kahlke, 2014). How MMR has shaped the questions asked by qualitative researchers is also a point of reflection. For instance, questions about the participant experience during clinical trials; exploring the meanings of quantitative measures, and using qualitative data to inform the development of quantitative measures in instrument development models were common in this CA. Such questions might be inspired by MMR.
We believe that the growing popularity and acceptance of MMR as a legitimate methodology has influenced researchers to re-examine the previously distinct divide between qualitative and quantitative research. This re-examining has foreseeably softened some of the claims previously made about research methodologies—for instance, in notions that the researcher can be bracketed out of the research process; claims related to absolute objectivity; or that numbers and qualitative research cannot coexist in a qualitative research study. For instance, authors frequently discuss majorities and proportions, and often provide numerical representations of these through counts or percentages (Maxwell, 2010). Regardless of one's involvement in MMR, the role of numbers and proportions, while contested at times, can be meaningful to the practice of qualitative research (Sandelowski, Voils, & Knaft, 2009).
Concluding Remarks and Recommendations
In this article, we explored how MMR has been approached in six qualitatively oriented journals. We identified predominant publishing and research practices and examined several pressing issues, such as the extent to which MMR studies are published in qualitative research journals; the types of studies published; how qualitative and quantitative data strands are used and mixed; and how MMR is discussed by researchers, editors, and theorists publishing in leading qualitatively oriented journals. We suggested that the publishing trends of empirical manuscripts favor publishing component pieces of designs as opposed to the design as a whole, and argued that this practice can obscure assessment of integration.
The common practice of publishing qualitative and quantitative results as separate manuscripts presented some unique challenges to assessing the meaningfulness and extent of integration. The dominant tendency was to publish qualitative study components in qualitative research journals and quantitative findings in quantitative research journals, consequently making little use of side-by-side display or other approaches to facilitate mixing. We recognize there is not one formulaic path to integration in MMR (Bazeley & Kemp, 2012). The extent of integration can be influenced by the sequence of data collection, the overall MMR design, the type of data collected, and “can occur throughout or at different stages of the research process” (Wooley, 2009, p. 7). Regardless of these nuances, mutual illumination or additive benefit is a necessary outcome of bringing together qualitative and quantitative study components in a meaningful way (Wooley, 2009). Although integration takes different forms, we found that these publishing practices presented a challenge to assessing, and potentially to conducting, meaningful integration.
A number of factors likely influence this practice. First, carry-over from the paradigm wars has, for some, led to a preference towards maintaining the epistemological integrity of seemingly separate research strands by keeping research components independent and integrating when drawing conclusions (Bazeley, 2009). Second, the space limitations of publishing are challenging for MMR; dual publications alleviate the need for absolute parsimony and enable a more thorough description of procedures and analysis for each strand than might be possible if published together. Third, the pressure to publish that is pervasive across academic circles should not be overlooked (Schraagen, 2012). Dividing findings for publication into journals representing each respective research tradition might be seen as desirable because it enables a higher number of research outputs. Finally, despite the emergence of excellent resources pertaining to integration, a skills barrier still might exist.
Important lessons for research, practice, and teaching can be gleaned from these findings. Sequential designs were most frequently encountered and were often reported in separate publications. This was particularly true instrument development research; many of these manuscripts only presented the qualitative portion of their work while making reference to the quantitative work as opposed to including the final product (the instrument) or validation information with the qualitative phase in one manuscript. Publishing practices demonstrating a preference for conveying results in separate manuscripts and authors inadequately demonstrating methods of integration were both problematic. Reporting findings in separate manuscripts makes identifying the point of integration between strands difficult and additionally, might encourage less rigorous forms of integration than otherwise would be expected.
In response to these challenges, researchers who integrate qualitative and quantitative data need to be attentive to how integration is represented in their manuscripts, irrespective of whether the substantive qualitative and quantitative research components are published separately. Information about both qualitative and quantitative strands and strong evidence of integration need to be provided to qualitative research journals editors if they are to publish MMR in these arenas. Signposts of MMR, such as using mixed methods in article titles, including a MM purpose statement, and having clear rationale(s) for mixing should be included regardless of publication venue. Moreover, instructors of MMR and those mentoring trainees conducting MMR need to emphasize integration as a hallmark of MMR and find creative ways, such as the use of visual diagrams, to illustrate how and where integration is occurring. This is especially relevant because empirical articles are increasingly published in qualitatively oriented journals, and because sequential designs, especially those of an explanatory nature, are particularly popular with trainees and novice researchers (Creswell, 2015).
Novice MM researchers are encouraged to work within specific, well-established designs, or typologies, which serve as road maps for data collection, analysis, integration, and interpretation. Becoming familiar with design typologies and learning to integrate qualitative and quantitative data at various points in the study increases one's flexibility and opens the door to more advanced MMR designs. Furthermore, given the popularity of sequential designs, it is prudent that qualitative researchers become particularly familiar with exploratory types of designs and model variants.
It is advisable that researchers at all levels aim for design coherence, particularly, how the research purpose and rationale for mixing influence design, and what extent of integration is enabled through the design and research procedures. This coherence should be represented through the appropriate use of metaphors to describe variations of mixing (Bazeley & Kemp, 2012). Adopting the common variations of merging, connecting, and embedding can accurately reflect the points of integration within a given study.
There are a few limitations to this CA. First, we recognize that the extent to which a given study's rated elements represent overarching categories matters greatly. In our study, we rated specific, well-defined MMR characteristics (e.g., study purpose, research questions) to answer the research questions and took a consensus-based rating approach, whereby we fully discussed and reconciled any discrepancies to address potential reliability and validity issues. Other informative methods to enhance content validity, such as using a Table of Specifications (ToS), may have been adopted (Newman, Lim & Pineda, 2013). Indeed, using a ToS might be a fruitful avenue for future research particularly when broader conceptual study categories are rated (e.g., methodological coherence or quality).
Second, we faced challenges during article coding. These were compounded by different publishing practices of the journals that we reviewed. The tendency to publish qualitative and quantitative research separately, the often deficient reporting of related quantitative research, and the inconsistent presentation and location of important information across journals contributed to these challenges. Third, we included articles published between 2003 and 2014. As such, a similar CA of previously published articles might reveal important trends not captured in the present research. The six qualitative research journals provided a broad sample of non-disciplinary specific articles; however, an alternative sample of qualitative research journals might have yielded different results. Including leading quantitative journals also might have yielded different findings; would publishing practices be similar in these journals and would similar designs be utilized? Future research in this arena might provide fruitful data for cross comparison and further understandings of MMR practices.
Numerous areas for future research were identified through this work, enabling systematic study of emerging MMR practices over the past decade. For example, we have begun examining how stated rationales for mixing corresponded (or not) with actual integration procedures, and whether features of visual displays differ by research design. These works are in progress and contribute to mapping the field of MMR during this reflexive era of practice (Creswell, 2009; Creswell & Plano Clark, 2011). Overall, findings from this CA contribute to longstanding efforts to delineate data mixing procedures (Bryman, 2007; Tashakkori & Teddlie, 2010), to illuminate influential practices in qualitative research, quantitative research, and MMR traditions, and to assist in understanding MMR practices and perspectives across diverse communities.
Footnotes
Appendix A
References for studies included in this review. N = 94