Abstract
The purpose of this article is to explore investigator triangulation (IT), a collaborative strategy with potential for mixed methods research (MMR). A critical review of the literature was conducted to identify IT’s core elements and its use in MMR. Five databases, 2 MMR journals, and 13 MMR texts were searched for evidence of IT according to preestablished inclusion criteria. IT descriptions and applications were inconsistent and lacked detailed reporting. Incongruence between IT procedures and associated claims were present. IT was generally limited to single-strand data analysis and was used predominantly to reduce researcher bias. IT’s potential as a generative and pragmatic research strategy used to engage with tensions emerging through diversity in MMR is explored and reporting guidelines are presented.
There is little doubt that mixed methods research (MMR) has profound capacity to contribute understandings about complexity and to “facilitate coming at things differently” (Hesse-Biber & Johnson, 2013, p. 103). As a boundary spanning practice, MMR can challenge paradigmatic and methodological distinctions, offer alternative approaches to understanding phenomenon, and foster diversity between multiple ways of knowing, traditions, and investigators. Yet investigators conducting MMR are confronted with developing the requisite expertise in qualitative, quantitative, and MMR traditions and grappling with emerging technical or methodological tensions. Given these challenges and potentials, collaboration is increasingly regarded as a desirable approach to MMR, mitigating exclusive reliance on the lone investigator (Creswell & Plano Clark, 2011; Hemmings, Beckett, Kennerly, & Yap, 2013; O’Cathain, Murphy, & Nicholl, 2008; Onwuegbuzie, 2012). As such, it is time to identify collaborative strategies that simultaneously attend to these challenges while promoting the spirit of diversity characteristic of MMR (Greene, 2007).
Investigator Triangulation (IT) is one such collaborative strategy that has potential for MMR practice. In this article, it is argued that IT, originally defined by Denzin (1970) as the use of multiple observers/investigators in a single study, could potentially attend to the multilevel challenges present in MMR while promoting a “mixed methods way of thinking” (Greene, 2007, p. 17). Exploring IT as a collaborative strategy is particularly timely for several reasons. First, as MMR expands in popularity and complexity, meaningful collaboration may foster opportunities to approach intricate research problems from new and synergistic angles (Hemmings et al., 2013; Hesse-Biber, 2010; Youngs & Piggot-Irvine, 2012). Second, at its best, collaboration may promote pluralistic orientations to research and attend to persistent challenges in the MMR field, such as those related to legitimation and integration (Bergman, 2008; Bryman, 2007; Morse & Niehaus, 2009). Third, collaboration can alleviate exclusive reliance on individual investigators who are challenged to develop needed expertise across research traditions amidst a MMR “skills gap . . . and a paucity of MM course offerings” (Hesse-Biber & Johnson, 2013, p. 104).
Yet collaborative strategies such as IT are not without their own challenges. Beyond the technical level, philosophical and methodological considerations are often entangled with those at the more procedural level. Historically centering on the compatibility and meaning of research paradigms (e.g., Greene, 2007; Morgan, 2007), paradigmatic considerations continue to infiltrate technical issues vis-à-vis researcher’s ontological and epistemological orientations. Whether acknowledged or unrecognized, such preferences may manifest as affinities for particular MMR designs or contribute to nonintegration of qualitative and quantitative data through methodological favoritism (Bryman, 2007), thereby acting as a barrier to effective collaboration.
Recent inquiries into collaboration, namely, the nuances of team dynamics (Hemmings et al., 2013), communities of research practice (Denscombe, 2008), and the social interplay between collaborating investigators (Lunde, Heggen, & Strand, 2013), provide examples of existing complexities. Individual investigator’s idiosyncrasies and personalities, whether researchers recognize their biases, and the level of methodological respect and investigative openness all exert considerable influence on the success of such collaborative ventures (Bryman, 2007; Lunde et al., 2013; Youngs & Piggot-Irvine, 2012). Understanding how collaboration has occurred in MMR and identifying collaborative methods that could attend to these important nuances are essential to developing usable strategies to assist in MM inquiry.
Therefore, to advance understanding of the potentials of IT as a collaborative approach in MMR, the aims of this article are to (a) identify how IT has been defined in the theoretical literature and applied in MMR, (b) explore and initiate dialogue regarding IT’s potential as a collaborative strategy with multilevel impact in MMR, (c) offer guidelines for reporting of IT, and (d) identify challenges and future directions. To these ends, an overview of triangulation is provided, results from a critical review of theoretical and MMR literature on IT are presented, preliminary reporting guidelines are offered, and the potential benefits of IT in MMR are explored.
Overview of Triangulation
Triangulation: Classical Perspectives
This overview of triangulation and its relationship to validity is brief as comprehensive historical accounts of both concepts are available elsewhere (e.g., Dellinger & Leech, 2007; Johnson, Onwuegbuzie, & Turner, 2007; Onwuegbuzie & Johnson, 2006; Teddlie & Tashakkori, 2009). Webb, Campbell, Schwartz, and Sechrest are often credited with coining the term triangulation in 1966 (Johnson et al., 2007, p. 114); however, original discussions may be traced back to Campbell and Fiske (1959), who explored triangulation in relation to measurement validation (Hammersley, 2008). The triangulation metaphor itself has its origins in navigation wherein two points are used to ascertain the position of a third (Halcomb & Andrew, 2005). While the term itself may link to a single root, triangulation has come to mean many things over its almost 60-year history (Morgan, 1998; Sandelowski, 1995).
The Sage Dictionary of Social Research Methods refers to triangulation as the observation of a research issue from a minimum of two points (Flick, 2006). Yet how this is understood and applied differs greatly within qualitative (e.g., completeness) and quantitative (e.g., confirmatory) traditions. Within the quantitative tradition, confirming findings is a common objective of triangulation, often manifesting as convergent validation or involving multiple measures to arrive at the same point (Shih, 1998). Literal interpretation requires the existence of a single reality wherein two points may converge (Sandelowski, 1995); however, this postpositivist orientation is problematic as measuring the same phenomenon twice is theoretically implausible and a singular truth, unattainable (Denzin & Lincoln, 1994; Fielding, 2012). Confirmation may more accurately denote consistency checking, attained through validity checking vis-à-vis interrater reliability or intercoder agreement procedures (Patton, 1999). However, viewing triangulation as a confirmatory strategy alone, although dominant in the mid to late 20th century, limits its potential (Halcomb & Andrew, 2005; Hammersley, 2008). As such, conceptions of triangulation have broadened beyond measurement validation to assume less positivist and more postmodern forms (Denzin, 2012; Torrance, 2012).
The qualitative origins of triangulation favor completeness and cohesiveness over confirmation (Greene, 2007) and reside within more comprehensive explanatory or holistic frameworks (Howe, 2012; Jick, 1979). As such, triangulation within the qualitative tradition generally involves using multiple methods (e.g., interviews, observations) or diverse analytic perspectives (e.g., Patton, 2002). Seeking complementary information or synthesizing divergent views to overcome strengths, weaknesses, and associated biases of a particular approach are consistent with this perspective (Bergman, 2008). The capacity to identify these strengths, weaknesses and biases is therefore imperative when using triangulation for completeness.
Interpretations of triangulation extend beyond qualitative and quantitative distinctions. Triangulation is frequently synonymized with MMR, a conflation that raises flags of caution for many researchers (Denzin, 2012; Wolf, 2010). Triangulation can refer to a MMR design, the intent of which is often corroboration or confirmation (Greene, 2007; Hesse-Biber, 2010). It may also denote bringing together methodologies, data, methods, or investigators (Denzin, 1970). Generic uses of triangulation do not capture its potentials and contribute to imprecision and obfuscation in the term’s meaning. The diverse meanings of triangulation and their proposed relationships to validity also underscore the intimate relationship between philosophical and methodological assumptions and triangulation procedures (Hammersley, 2008).
Triangulation in MMR: Beyond Confirmation?
Although confirmation and completeness remain two broad purposes by which triangulation is frequently classified (Creswell & Plano Clark, 2011; Hesse-Biber, 2010), using triangulation as a confirmatory strategy alone may inadvertently hinder development of the MMR field. An important assumption underlying confirmatory approaches to triangulation is that “validating (rather than developing) interpretations is what is most important in research” (Hammersley, 2008, p. 24). Yet, as Bryman (2007) argues, MMR involves “forging an overall or negotiated account of the findings that brings together components of the conversation or debate” (p. 21), rather than focusing exclusively on how results reinforce or confirm one another. An overemphasis on confirmation and corroboration may “mute the potential value of divergence and dissonance” (Greene, 2007, p. 45); undercutting the potential of triangulation to generate further questions and understandings. As divergence is gaining attention as a valuable opportunity for exploring empirical puzzles in MMR (Greene, 2007; Tashakkori & Teddlie, 2010), there is an opportunity for triangulation, and specifically IT, to extend beyond confirmation within the context of interdisciplinary collaboration in MMR.
Hammersley (2008) identifies indefinite triangulation and epistemological dialogue as two additional and interdependent purposes of triangulation. Within the former, triangulation is a “devise for generating divergent interpretations” (Hammersley, 2008, p. 26), or uncovering multiple perspectives (Fielding & Fielding, 1986), encouraging reflecting on how one arrives at interpretation and for which function. Epistemological dialogue is transactional, constructivist, and aligned with metaphorical alternatives of the prism, crystal or “multi-genre crystallization” (Ellingson, 2009; Leech, Dellinger, Brannagan, & Tanaka, 2010; Mertens, 2010). Such alternatives provide opportunities to engage with epistemological tensions and question understandings and assumptions. It is within this spirit of engaging with, and furthering discussions on triangulation that the present inquiry on IT resides.
Method
A critical review of the theoretical and MMR literature was conducted. Critical reviews are useful when creating an inventory of research in a particular area and when examining how these works contribute to the development of a concept, such as IT. Systematic methods are employed and directions for future work are identified (Grant & Booth, 2009). The electronic databases of CINAHL, MEDLINE, EMBASE, ERIC, and PsychInfo, selected for their breath and interdisciplinarity, were systematically searched for investigator OR researcher triangulation, using no limiters. Two prominent MMR journals (i.e., The Journal of Mixed Methods Research and International Journal of Multiple Research Approaches) were searched for “investigator OR researcher triangulation,” to ensure relevant MMR articles on IT were identified.
To be included in this review, authors needed to either describe IT in a theoretical or discussion paper or explicitly state that IT was used in MMR. MMR is defined here as the collection, analysis, and integration of qualitative and quantitative data in a single study, recognizing that dialogue surrounding a MMR definition remains open (Creswell, 2009; Johnson et al., 2007). For IT descriptions/criteria, authors mentioning IT but providing no additional information were excluded (e.g., IT is a type of triangulation) as there were no data to extract. Gray literature was excluded. Thirteen MMR textbooks identified through library searching and expert consultation were reviewed for evidence of IT. Ancestry searching was conducted to capture highly cited articles or texts on IT. The foundational texts by Denzin (1970, 1978, 1989) on IT were used as a starting point and were counted as one resource.
Data were extracted according to consistent categories, which differed for IT criteria and empirical examples. Categories were based on Denzin’s initial iterations of IT (1970, 1978, 1989). For all articles, data on first author and discipline, year of publication, and authors cited in support of IT were extracted. For IT criteria, the following additional data were extracted: number of researchers, researcher attributes, stage and extent of researcher collaboration, and IT-associated claims or justifications. Descriptions of IT procedures included in empirical examples were reviewed to uncover methods, benefits, and shortcomings of using IT in MMR.
Results
Database searching yielded 239 articles (CINAHL n = 138; Medline n = 64; EMBASE n = 22; PsychInfo n = 12; ERIC n = 3). Forty-three articles were identified through journal searching (Journal of Mixed Methods Research [n = 38]; International Journal of Multiple Research Approaches [n = 5]) for a total of 282 articles. Sixty-two duplicate articles and 34 gray literatures (e.g., dissertations) were removed.
Of the remaining 186 articles, 166 were excluded based on the inclusion criteria. Most often, these articles included both keywords of investigator and triangulation in ways consistent with the following example: “Investigators triangulated data/methods or triangulation of data were conducted by investigators . . .” Of the remaining 20 articles, 13 were included as IT Criteria (Table 1) and 7 were included as Empirical Examples of IT in MMR (Table 2). Six additional resources (one textbook; five articles) were identified through ancestry searching and were included for a total of 18 articles and 2 textbooks under IT criteria.
IT Criteria.
Note. IT = investigator triangulation; MMR = mixed methods research.
Empirical Examples of IT in MMR.
Note. IT = investigator triangulation; MMR = mixed methods research.
Criteria for IT
IT descriptions were discrepant and ambiguous. Discrepancies begin with Denzin’s (1970) initial conceptualization of IT, which included two or more observers in a research study. Later, he conceived of IT as involving two or more skilled researchers collecting and analyzing data (Denzin, 1989). Here, equal influence of each researcher is suggested. Extending this description, Kimchi et al. (1991) included role prominence, diverse expertise, and disciplines as additional criteria. Subsequent authors (e.g., Begley, 1996; Im & Chee, 2002) reiterated these descriptions with slight modifications. Most often, these modifications were made without discussing how they departed from original conceptualizations of IT. Claims regarding IT were questionable in light of these modifications. Claims associated with IT favored perspectives of triangulation as a confirmation or validation strategy.
Areas of consensus included the use of multiple investigators and generally that some diversity of skills, training, or disciplinary backgrounds should exist. However, descriptions of researcher attributes provided, such as skills and expertise, were nondescriptive and ambiguous. For instance, 7 of 18 references did not specify which researcher attributes (e.g., skill, expertise, disciplinary background) are necessary in IT (Al-Hamdan & Anthony, 2010; Azulai & Rankin, 2012; Patton, 1999, 2002; Schroepfer et al., 2009; Torrance, 2012; Williamson, 2005). Of the 13 resources that did describe attributes, 5 generically referred to skilled or expert researchers (Begley, 1996; Cowman, 1993; Denzin, 1970, 1978, 1989; Halcomb & Andrew, 2005; Thurmond, 2001).
Authors of six resources iterated the need to report investigators disciplinary or training backgrounds (Flick et al., 2012; Halcomb & Andrew, 2005; Im & Chee 2002; Kimchi et al., 1991; Shih, 1998; Thurston et al., 2008). Only Thurmond (2001) and Flick et al. (2012) identified that epistemological preferences or theoretical underpinnings of each investigator warrants acknowledgement. The remaining authors did not attend to epistemology despite claims associated with bias reduction or complementarity.
Stage, extent of collaboration, and degree of investigator independence were discrepantly described. Regarding stage (i.e., the point of the research process during which IT occurred), data collection and analysis (Al-Hamdan & Anthony, 2010; Azulai & Rankin, 2012; Cowman, 1993; Denzin, 1970, 1989; Thurmond, 2001; Williamson, 2005) or data analysis alone (Begley, 1996; Mitchell, 1986; Patton, 1999, 2002; Redfern & Norman, 1994; Schroepfer et al., 2009) were the most common points of collaboration. Authors of four articles did not specify the stage of IT collaboration (Flick et al., 2012; Kimchi et al., 1991; Shih, 1998; Torrance, 2012). Infrequently, IT was indicated for use throughout the entire study (Thurmond, 2001), during evaluation (Cameron, 2009) or during data collection alone (Thurston et al., 2008). Authors of two articles postulated that the “problem or phenomenon should be examined” through IT but provided no further description (Halcomb & Andrew, 2005; Im & Chee, 2002). The extent of researcher collaboration, frequently expressed as “equal role prominence,” was explicitly discussed in four articles (Begley, 1996; Flick et al., 2012; Kimchi et al., 1991; Shih, 1998) and eluded to by Denzin (1978, p. 303). Azulai and Rankin (2012), Halcomb and Andrew (2005), and Patton (1999) attended to investigator independence, stating that no discussion or collaboration should occur prior to examining the problem, collecting, or analyzing the data.
Despite these discrepancies, claims associated with IT were extensive and diverse. Authors of 11 resources credited IT with reducing or neutralizing researcher bias (Cowman, 1993; Denzin, 1970, 1978, 1989; Halcomb & Andrew, 2005; Kimchi et al., 1991; Mitchell, 1986; Patton, 1999, 2002; Redfern & Norman, 1994; Schroepfer et al., 2009; Thurmond, 2001; Thurston et al., 2008), while eight endorsed IT with enhancing trustworthiness, credibility, reliability, or validity (Azulai & Rankin, 2012; Cowman, 1993; Halcomb & Andrew, 2005; Mitchell, 1986; Patton, 1999, 2002; Schroepfer et al., 2009; Thurmond, 2001). Authors of six resources discussed how triangulation can enhance understanding and richness of data (Flick et al., 2012; Halcomb & Andrew, 2005; Patton, 2002; Shih, 1998; Thurmond, 2001; Torrance, 2012); with Flick et al. (2012), Patton (2002), Shih (1998), and Thurmond (2001) discussing this specific to IT procedures.
IT was indexed in three of the 13 MMR textbooks reviewed (Johnson & Christensen, 2012; Plano Clark & Creswell, 2008; Teddlie & Tashakkori, 2009). These authors referred to but added little to Denzin’s descriptions (e.g., 1978). General discussions of triangulation in these works were more comprehensive than those provided in included articles. When IT was not indexed, sections on triangulation were reviewed for evidence of IT. Authors of two textbooks not indexing IT defined it generally (i.e., Greene, 2007; Tashakkori & Teddlie, 1998), iterating elements of Denzin’s 1978 description. Bergman (2008), Creswell (2009), Creswell and Plano Clark (2011), Hesse-Biber (2010), Mertens (2010), Morse and Niehaus (2009), Sheperis, Young, and Daniels (2009), and Tashakkori and Teddlie (2010) did not refer to IT. In summary, IT has been conceptualized diversely and the concept itself appears underdeveloped. To explore if and how these inconsistencies contribute to variability in IT’s application, empirical examples of IT from the MMR literature are now presented.
Examples of IT in MMR
Seven articles were included as empirical examples of IT in MMR (Table 2). Five key findings emerged from analysis, including IT justifications, disciplinary variations, variability in investigator expertise, discrepant reporting and support for IT, and stage and nature of investigator collaboration. These will be discussed in sequence.
IT justifications provided mirrored the inconsistencies present in the IT descriptions. Justifications most commonly related to rigor, credibility, reliability, or validity of data analysis or interpretation (Crooks, Schuurman, Cinnamon, Castleden, & Johnston, 2011; Kulig et al., 2008; Kuusela et al., 2013; Niedermann et al., 2010). The benefits of researcher diversity in examining complex phenomenon were underscored in two studies (Flick et al., 2012; Kennett et al., 2008). Lo (2009) provided no justification for using IT.
IT was employed in various disciplines such as medicine (Kuusela et al., 2013; Niedermann et al., 2010), psychology (Flick et al., 2012; Kennett et al., 2008), nursing (Kulig et al., 2008), and health geography (Crooks et al., 2011). Lo’s (2009) disciplinary background was unclear. IT reporting was most comprehensive in studies with first authors from medicine or psychology, where the number of investigators, investigator diversity, and the stage of collaboration were generally described.
The discipline, training, and expertise of researchers involved in IT were generally unclear. Exceptions included articles by Kennett et al. (2008), Kuusela et al. (2013), Lo (2009), and Niedermann et al. (2010). Kennett et al. (2008) and Niedermann et al. (2010) provided particularly rich descriptions. Crooks et al. (2011) did not describe researcher attributes, while Flick et al. (2012) provided an incomplete description. When described, determining which investigator conducted which procedure was difficult, reducing confidence in claims associated with validity, and so on. Not one study acknowledged the philosophical or paradigmatic stances or biases of participating researchers.
Surprisingly, only one study that applied IT also described IT (Flick et al., 2012). Kennett et al. (2008), Kuusela et al. (2013), and Niedermann et al. (2010) provide no IT-specific citation to support their claims, while Crooks et al. (2011) refer to an article that mentions triangulation, not IT, once throughout its text. Denzin (1970, 1978) was the most commonly acknowledged author used to support IT.
The stage and nature of researcher collaboration varied. IT was most often used during data analysis (Kennett et al., 2008; Kuusela et al., 2013; Niedermann et al., 2010) followed by data analysis and interpretation (Crooks et al., 2011; Kulig et al., 2008). Evidence that IT was used to analyze both data strands or generate integrated inferences was sparse. The degree of researcher independence in IT procedures was generally unclear. When described, investigator independence was occasionally clear for one component of a study (e.g., qualitative analysis; Niedermann et al., 2010) but not indicated for others (e.g., integration). Methods of achieving consensus or exploring degrees of convergence and/or divergence between investigators were infrequently described.
Discussion
IT as a form of researcher collaboration applicable to multi- and interdisciplinary types of teamwork (O’Cathain et al., 2008) has been minimally explored or employed within MMR. This is problematic considering the variability in IT’s application, the lack of detailed reporting surrounding IT procedures, and the possibility that associated claims, such as those associated with robustness and rigor, are both overstated yet underexplored. There is potential to develop IT as a collaborative strategy with multilevel benefits in MMR.
Extant issues with IT are rooted in how IT is conceptualized and operationalized. Therefore, related challenges and potentials will be broadly discussed according to two dependent themes. First, the need to operationalize and conduct IT in a logically cohesive manner will be discussed, based on the core components of IT identified in the critical review. These include (a) justifications for using IT and (b) IT procedures (i.e., nature of collaboration, investigator attributes, and reporting). Second, the present and potential contributions of IT to investigators and the field of MMR will be explored.
Logical Structure of IT
Alignment between IT’s core components is needed to promote cohesiveness between procedures and associated claims. Underlying these efforts is the need for transparent and comprehensive reporting of IT efforts. As such, IT reporting details are presented in Table 3.
Reporting Details for IT.
Note. IT = investigator triangulation; MMR = mixed methods research.
Justifying IT
Just as justifying MMR provides a sense of appropriateness and context for its use, justifying triangulation adds directionality and understanding for the reader (Bryman, 2006; Greene, Caracelli, & Graham, 1989; Halcomb & Andrew, 2005). However, findings from this review illustrate that justifications and IT procedures are often misaligned and inconsistently reported. For instance, if reducing bias is the aim of IT yet no evidence of investigator diversity or declaration of bias and epistemological standpoints are provided, incongruence between claims, researcher attributes, and IT procedures is present. At times, congruence between justifications and procedures were noted (e.g., Flick et al., 2012; Kennett et al., 2008). In these studies, diverse investigator backgrounds and perspectives were sought to examine complex phenomenon. Investigator details were reported accordingly. Demonstrating congruence between justifications of validity and IT procedures would require reporting variables such as researcher independence in analysis or methods of collaboration and achieving consensus. Such omissions in reporting have been found elsewhere. For example, only 3 of 13 critical appraisal frameworks analyzed by Heyvaert, Hannes, Maes, and Onghena (2013) indicated that the “impact of the researcher on the research process and product should be clearly reported” (p. 315).
IT Procedures
Failure to operationalize the elements of IT presented by Denzin and later Kimchi et al. (1991) has contributed to discrepancy in IT procedures. Take, for example, Denzin’s (1989) statement that two or more researchers should examine data. For this definition, the degree of investigator independence during data analysis is unclear. “Examine the data” suggests analysis but is not explicit. This could denote, for instance, examination at the level of data cleaning or multiple investigators conducting content analysis. This lack of clarity was noted in the majority of included resources and it was generally unclear which “skilled investigator”“examined” which component of the data. Such nuances could be more clearly reported by adapting the concept of notation from MMR designs to capture IT endeavors (e.g., Morse & Niehaus, 2009).
IT in MMR is presently limited to using multiple investigators for one data strand, namely, qualitative data analysis (e.g., Crooks et al., 2011). Although this reflects the interpretive orientations suggestive of qualitative research and the values of objectivity commonly associated with the quantitative tradition, collaborative contributions to the design and interpretation of quantitative research may indeed be strengthened through IT efforts. For example, investigators may hold different perspectives on how to handle outliers in regression analysis or during an analysis of variance. These decisions may be influenced by the research question, degree of investigator expertise, and interpretations of the meaningfulness of the outliers for example. How investigators choose to represent data suggests “assumptions about the nature of data . . . which are not simply given but are constructed by researchers based on experiences and perceptions” (Maxwell, 2010, p. 478). As such, aspects of quantitative analysis and interpretation may be strengthened through collaboration and discussion. In qualitative analysis, IT most commonly involved multiple investigators applying an a priori established coding framework to code qualitative data. Alternatively, a second investigator “reviewed” themes or codes identified by a first investigator. Authors stated that themes or codes were confirmed and/or consensus was achieved, which enhanced the author’s confidence in results. Occasionally, consensus approaches were described (e.g., Niedermann et al., 2010) but generally occupied the lines in-between the research text. In this way, IT was applied as a consistency check but iterations and applications of IT did not reflect the potential uses of triangulation procedures (e.g., devise for generating divergence; epistemological juxtapositions; Hammersley, 2008).
In summary, IT is presently used within MMR as a single method strategy and little procedural guidance has been offered for its use. This mirrors the dearth of available procedural guidance for triangulation in general (Farmer, Robinson, Elliott, & Eyles, 2006). There is little evidence to date that IT has been applied beyond the analysis of qualitative data; its potential use in leveraging diverse perspectives of investigators to generate more comprehensive understanding and to generate and explore divergence is limited and not reflected in present reporting practices. By attending to core components of IT, such as matching investigator expertise to research procedures by use of a notation system (e.g., Morse & Niehaus, 2009), and by extending the application of IT beyond use with a single method or data strand into collaborative design and drawing of integrated inferences, IT’s potentials in MMR may be more fully realized. Toward these ends, comprehensive and transparent reporting of IT efforts is needed (Table 3). This echoes similar calls for explicit reporting of “knowledge yields” gained through data integration (O’Cathain et al., 2008), reporting of MMR generally (e.g., Creswell & Plano Clark, 2011), and for particular researcher groups (e.g., Leech & Onwuegbuzie, 2010). The potentials of IT as collaborative strategy within MMR will now be explored.
Potential Contributions of IT
IT and Quality in MMR
The indicators by which to evaluate the quality of MMR are in development. Mirroring long-standing debates on validity and/or trustworthiness in qualitative research, discussions of validity in MMR remain open, attentive to the numerous meanings (and at times, meaninglessness) of the concept more generally and the desirability of bilingual versus MMR specific nomenclature (Onwuegbuzie & Johnson, 2006; Teddlie & Tashakkori, 2009). Validity is not without baggage. Contemporary explorations of validity have broadened perspectives originally shaped by the quantitative measurement tradition (Dellinger & Leech, 2007). It is in this spirit of exploration that the present discussion resides.
Within MMR, attending to qualitative and quantitative associated quality criteria is alone insufficient as MMR specific legitimation or validity criteria need also be satisfied (e.g., Onwuegbuzie & Johnson, 2006; Teddlie & Tashakkori, 2009). This requires attending to the unique characteristics of MMR, such as meaningful data integration and drawing of integrated inferences (Leech et al., 2010). As such, differentiating between data and inference quality is useful for this discussion. As Teddlie and Tashakkori (2009) concisely state, “Data quality is limited to the goodness of qualitative and quantitative data” (p. 212), where inference quality, to the quality of conclusions drawn from data. This differentiation is important to explore two domains where IT may be of benefit.
Data quality appears to be predominantly a technical level issue, and may provide the most accessible benefits for investigators involved in IT. Technical-level efforts do not necessarily require exploration of epistemological tensions and may merely involve matching investigators to the research method with which they are most skilled to enhance the quality of data obtained within the collaborative context (Teddlie & Tashakkori, 2009). At this level, well developed strategies, such as member checking and intercoder reliability checks to verify consistency of data analysis and interpretations may be employed; complementarity may be enhanced by identifying biases, using multiple viewpoints and mitigating strengths and weaknesses associated with a singular approach (or investigator) (Hammersley, 2008; Kimchi et al., 1991; Shih, 1998). However, these represent limited uses of IT, and unnecessarily narrow views of validity and triangulation within MMR. As will be later discussed, using IT as a devise to cultivate epistemological dialogue or juxtaposition is a more generative and creative strategy for MMR.
Regarding validity, there is an intimate relationship between data quality and legitimation—the second crisis of MMR (Onwuegbuzie & Collins, 2007). The ability to draw credible inferences from integrated data is greatly influenced by data quality, but also by the views of collaborating investigators regarding what qualifies as valid or trustworthy data within a given study. The concept of validity differs between qualitative and quantitative research traditions (Dellinger & Leech, 2007), investigators conducting MMR may default to perspectives of validity most aligned with the research tradition with which they are most comfortable. Similarly, bringing together researchers who have similar perspectives on validity may obfuscate elements of quality associated with other research traditions. In this way, data quality in collaborative MMR cannot be regarded as a mirror image of data quality in stand-alone qualitative or quantitative oriented inquiries.
Inference quality is a common thread weaving through existing discussions of validity or legitimation in MMR (Dellinger & Leech, 2007; Onwuegbuzie & Johnson, 2006; Tashakkori & Teddlie, 2008). Current applications of IT in MMR inconsistently attend to this component. Inference quality may be enhanced by bridging other forms of triangulation, most visibly, theoretical triangulation, as investigators may be less bound to particular frameworks or theories when interpreting the data (Burns & Grove, 2001; Flick et al., 2012; Thurston et al., 2008). By extending beyond previously held frameworks to promote, accept and integrate diversity of perspectives, investigators may move toward the radical middle and conduct integrative MMR (Greene, 2007; Onwuegbuzie, 2012).
Integrative MMR is at the forefront of current discourse on legitimation. In a highly recognized study, Bryman (2007) identified barriers to integration, many of which are relevant to IT. To start are concerns related to the beliefs about the nature of, and relationship between, qualitative and quantitative research. How research regarded as different can be meaningfully integrated relates not only to the technicalities and practicalities of mixing, but also reflects theoretical undercurrents and associations with research traditions. Such undercurrents may go unrecognized only to surface during critical time points in MMR (e.g., data integration), reflecting Morgan’s (2007) question regarding the extent to which mixing is merely about methods rather than methodology.
Theoretical associations and assumptions may foster methodological favoritism wherein researchers emphasize findings from either qualitative or quantitative research “because they have greater faith in one rather than the other” (Bryman, 2007, p. 12), thereby affecting the seemingly technical mixing of qualitative and quantitative data. Training and experience may predispose investigators to preferentially handle one set of research findings, regardless of their meaningfulness and potential contribution to the research question, thereby threatening integration—the third crisis of MMR; Bryman, 2007; Onwuegbuzie & Collins, 2007). Although drawing meta-inferences depends on attention to both data strands (Dellinger & Leech, 2007), the interactive process of meaning making resulting from this combination is infrequently attended to (Bazeley, 2009). How this process unfolds between diverse investigators is even less understood.
Exploring processes of meaning making, emerging methodological favoritisms, and engaging with tensions arising from divergent discussions may provide meaningful insights. If triangulation is treated as a “device for generating divergent interpretations” (p. 26) as it is with indefinite triangulation (Hammersley, 2008), “opportunities for enriched explanations” (Jick, 1979, p. 609; Plano Clark & Creswell, 2008, p. 113) may be generated, thereby capitalizing on the potential values of divergence in MMR. Creating an investigator-mosaic with diverse researchers may help generate divergent interpretations, the origins of which can then be explored. This would assist in moving triangulation beyond the realm of data quality into the important domain of inference quality—supporting investigators to seek meaningful interpretations of multiple and mixed data strands (Onwuegbuzie & Johnson, 2006).
IT as a Generative Strategy
It has been argued that exploring the interpretive agreement or distinctiveness present between multiple investigators, and engaging with alternative explanations for divergence are two important elements in inference quality that may be leveraged through IT. Engaging in tensions emerging through investigative diversity within an IT framework may further assist in moving IT into the realms of juxtaposition or epistemological dialogue. This would extend IT beyond its use as a confirmation strategy largely aligned with validity checking into more generative orientations.
Triangulation as epistemological dialogue or juxtaposition is more aligned with postmodern metaphors that conjure images of multiple perspectives, such as the prism (Mertens, 2010), crystal, or kaleidoscope (Ellingson, 2009). These are more congruent with what Onwuegbuzie (2012) calls a “constructionist view of methodology” (p. 195) than postpositivist orientations to triangulation. Such contemporary perspectives of IT as an alternative to validation (Denzin, 2012; Flick et al., 2012) can be used to generate a “plurality of perspectives” (Johnstone, 2007, p. 30), “multiple ways of seeing and hearing” (Greene, 2007, p. 20), and foster researcher reflexivity by keeping epistemological issues in dialogical tension.
Such plurality of perspectives may promote investigator self-questioning, a hallmark of reflexivity (Johnstone, 2007). IT may keep researchers honest and help identify prevailing biases influencing design and analytic decisions. Indeed, the generative potential of MMR relies, in part, on investigator’s capacity to engage with and respect differences arising from multilevel diversity (e.g., methods, methodology, researcher). In these ways, IT may help leverage diversity in a manner that honors the complexities of research traditions, investigator standpoints, and the research phenomena in question (Greene, 2007).
Maintaining epistemology in tension is parallel to constructivist discussions of transactional epistemology and hermeneutical methodology (Leech et al., 2010). Knowledge production occurs through co-creative, negotiated processes, shaped in part by the experiences and biases of participating researchers. Researcher’s previous understanding of phenomenon under investigation, including reflecting on how understandings constrain or contribute to new knowledge, is a “foundational element” to this process and to MMR validity (Dellinger & Leech, 2007). Investigator flexibility is necessary to enable unique combinations of evidence and MMR validity criteria, as these may differ based on the research design and phenomenon in question (Dellinger & Leech, 2007). Reminiscent of Onwuegbuzie and Leech’s (2005) pragmatic researcher, IT may be used in this context to engage with tensions emerging from diverse standpoints and epistemologies, maximizing exposure to methodological techniques and interpretations, thereby equipping investigators with a “bi-focal lens” (p. 383) to more aptly conduct rigorous MMR and draw meaningful integrated meta-inferences.
In their unified validation framework, Dellinger and Leech (2007) provide a highly useful contribution to MMR legitimation by reexamining conceptions of construct validity emerging over time. Drawing heavily on Messick’s (1995) conceptualization, they regard construct validity as a value laden, iterative and social process that “encompasses all validity evidence” (p. 316). They carry forward two central questions related to construct validation emerging from Messick’s juxtaposition of data interpretation with data use which become central to the “foundational element” of MMR validation: (a) identifying the evidence supporting interpretations of data and (b) examining the consequences of interpreting data in a particular way (Dellinger & Leech, 2007). Within the context of IT, such reflection may be seen as the lynchpin between data quality and inference quality. During IT endeavors, these questions may assist investigators to challenge emerging methodological favoritisms regarding trustworthiness, validity, or biased perspectives toward theories, question values underpinning decisions made, and promote epistemological dialogue by using IT as generative device in MMR.
Indeed, tensions emerge when phenomenon are “approached, described, and measured” in diverse ways (Dellinger & Leech, 2007, p. 318), be it through various abstractions employed by researchers to capture or represent phenomenon (e.g., scales, interview questions) or through the diverse perspectives forwarded by individual investigators. When recognized as something to be engaged with and negotiated with rather than resolved, tensions surfacing during integration may provide meaningful insights into the interpretative nature of meaning making and result in fewer unexamined interpretive assumptions and facets of understanding. This recognition may help prevent integration stalemates arising during interdisciplinary collaboration (e.g., Bryman, 2007). Furthermore, such engagement highlights the true value of MM inquiry, namely, enabling an interactive “dialogue between different ways of seeing, interpreting, and knowing, not simply in combining different methods and data” to advance understanding (Maxwell, 2010, p. 478).
Paradigmatic Mixing, Matching, or Abandoning
Engaging with emerging tensions may disrupt the bounding of researchers to particular paradigms and frameworks (Thurston et al., 2008), and ideally begin to deconstruct and dismantle paradigmatic and or disciplinary divides should they exist. As such, paradigmatic stances and their influences on IT are necessary complements to this discussion. Risking oversimplifying these rich debates, six paradigmatic stances summarized by Greene (2007) and their influence on IT, as well as perspectives of paradigms as “shared beliefs among a community of researchers” (Morgan, 2007, p. 53) will be briefly discussed.
In a highly influential article, Morgan (2007) identified epistemological stances as one of four versions of paradigms impacting the combining of methods. This stance represents the dominant version of paradigms present in the social sciences; its influence is evidenced by barriers to integration reported in recent analyzes of collaboration in MMR (e.g., Lunde et al., 2013). Yet regarding paradigms as beliefs shared among a group of practitioners rather than a factor binding investigators to disciplinary origins may assist in moving IT into a reflexive realm where communicative exchange central to meaning making and to pragmatic approaches to research are possible (Denscombe, 2008; Morgan, 2007). Using this stance, researchers are not constrained by epistemological stances or whole disciplinary orientations but can partner over a belief that engaging with and challenging assumptions and tensions in the practice of MMR can lead to further legitimation and MMR advancement (Morgan, 2007), reflecting notions of interdisciplinary communities of research practice (Denscombe, 2008; Hemmings et al., 2013).
Engaging with epistemological tensions and generating diverse perspectives are well served through a dialectic stance on paradigmatic influence (Greene, 2007). This involves respecting the integrity of different paradigms and recognizing that while paradigm associated assumptions influence research related decisions they are themselves socially constructed.
Reminiscent of paradigms as shared belief systems and practices, the epistemological, ontological, and methodological characteristics of a paradigm are not in themselves “defining characteristics” (Morgan, 2007, p. 61) but are subject to a great deal of human agency and social influence. Thus, through respectful engagement, productive tensions and juxtapositions may be leveraged and contribute to deeper, or different, understandings. Further suggestive of the flexibility characteristic of Dellinger and Leech’s (2007) foundational element identified in the unified validation framework, this “mixed methods way of thinking” (Greene, 2007) may be promoted through IT by actively engaging multiple ways of seeing and knowing (Tashakkori & Teddlie, 2010).
Within an alternative or single paradigm stance, the emergence of new paradigms helps reconcile incommensurability and respond to demands spurred by theory and context (Greene, 2007; Tashakkori & Teddlie, 2010). Emergent paradigms characterized by, or even necessitating, the mixing of methods may embrace a strategy, such as IT, with the potential to leverage these mixings. Alternatively, IT within the substantive theory stance may involve exploring multiple theories held by investigators, which are more pertinent to MMR than are paradigms (Tashakkori & Teddlie, 2010). This stance may extend IT into the realm of theoretical triangulation, or enable deeper understanding of a singular theory employed by different investigators.
An a-paradigmatic stance promotes mixing at multiple levels, which can occur without violating presumed one-to-one relationships, for instance, between paradigms and methods (e.g., interviews are a qualitative research technique; Greene, 2007). Here, technical challenges are unburdened by paradigmatic associations, which renders ITs’ application largely procedural. Conversely, within a multiple paradigm approach, investigators may debate which paradigmatic stance is most appropriate for which MMR design and as such, using IT throughout the projects duration would be desirable.
Finally, within the complementary strengths stance, methods are separated to preserve the integrity of each distinct paradigm (e.g., constructivist, positivist) assuming a clear distinction can be drawn between these (Morgan, 2007). Paradigms are not incompatible but are different, contrasting with the purist stance where boundaries between incompatible metaphysical paradigms are predicated on perceived differences related to truth and reality (Morgan, 2007). Although the complementary strengths stance does not purport incompatibility, it does remain focused on differences; as such, communicative breakdowns, and preferential treatment of data emerging from specific methods may arise if threats to the paradigmatic and methodological integrity of either tradition are perceived (Greene, 2007; Morgan, 2007). Using IT within this stance would likely involve delegating investigators to their particular content area and challenges in collaborative meaning making may arise. Of the perspectives offered here, reflecting to identify warranted assertions, communicating to establish shared meanings, and establishing joint lines of action, as emphasized by a pragmatic research approach (Morgan, 2007) are most aligned with the potentials of IT as a collaborative, interdisciplinary strategy with generative potential in MMR.
Challenges and Future Directions
Developing IT as a collaborative strategy with multilevel benefits for investigators is both challenging but timely, given that MMR and the investigative climate more generally are trending toward collaborative research, evoking potentials for researchers capable of meaningful collaboration. Strategies that capitalize on unique aspects of MMR (e.g., integration) and attend to components of construct validity and legitimation are useful and important to MMR development. Broader perspectives on triangulation may help investigators to integrate multiple types of triangulation (e.g., investigator, theoretical; see Flick et al., 2012, for example). Future explorations of IT may expand to integrate components of respondent validation (Torrance, 2012) and further encourage strategies relating to co-constructing social knowledge. Presumably, IT and extant literature on participant involvement in MMR need not exist in isolation. Present insights may be helpful for adapted use with community-based partners. Further dialogue in this area is welcomed.
The challenges of collaboration may be compounded through the diversity idealized through IT. Teamwork can be rewarding, but also a frustrating and a costly endeavor (Creswell & Plano Clark, 2011). To resolve and manage conflict and create functional team environments may require investigator training and at a minimum, commitment and patience. Without creating an environment where investigators can freely share in an open, negotiated manner, problems related to bias and “bogus triangulation” may arise (Fielding, 2012, p. 127). Investigators’ voices must be preserved and mere echoing of perspectives held by more dominant or assertive investigators, avoided (Mertens, 2010). Furthermore, if epistemological stances of investigators are not acknowledged, researcher bias may be amplified, not reduced, through collaboration (Thurmond, 2001).
Capturing the nuances and complexities of IT negotiations and procedures in research reporting is difficult. Detailed memoing may be necessary to capture discussions and should be included in the researcher’s audit trail (Mertens, 2010). Extending the use of memoing beyond its association with qualitative research (e.g., Johnson & Christensen, 2012) to include negotiations, inference and meaning making from data integration may help bring IT into the MMR realm. Just as joint displays may assist in integrated analysis, visual diagrams may help detail IT procedures, relevant considering the space limitations faced when publishing MMR and the challenges in communicating IT complexities. Visual diagrams may enable clearer linkages between researcher attributes (e.g., training, background), role prominence, stage of IT, and negotiation processes. Adapting “mind maps” (e.g., Wheeldon, 2010) to capture integration and inference generation between multiple investigators may be useful. Considering the overlay of complexity when using IT with MMR designs, the potential benefit of using visual diagrams to depict IT in MMR is underscored.
Limitations
Despite efforts to locate articles of relevance to IT, the search methods utilized, as with any review, limited the breadth of the literature retrieved and analyzed. The terms investigator and researcher triangulation were used but broader terms, such as collaboration and interdisciplinary, were not. Considering the broad conceptualization of IT that has been forwarded here, there is space for future work to further develop this concept in relation to the growing body of literature on interdisciplinary and team collaboration in MMR.
Conclusion
Through this critical review of the literature, the core elements of IT have been identified and its present and potential applications in MMR have been explored. IT as a collaborative strategy has received little attention within MMR yet may be useful in fostering pragmatic, generative, and synergistic research. IT has potential multilevel benefits for investigators conducting MMR. Expertise may be leveraged, lone-investigator deficits compensated for, and data and inference quality enhanced. IT can be used to explore paradigmatic, epistemological, and methodological tensions influencing study design, analysis, interpretation, and collaboration more generally. Complexities arising from degrees of convergence and divergence may be deconstructed. By extending the use of IT beyond single data strands to generate and explore integrated inferences, a multiplicity of perspectives may be captured and holistic approaches to research and meaning making generated.
Many of the collaborative processes captured throughout this discussion, such as processes of negotiated meaning making and plurality of investigator perspectives (Johnstone, 2007), the flexible partnerships reminiscent of Messick’s (1995) conception of construct validation and later, Dellinger and Leech’s (2007) unified validation framework in MMR, illuminate communication, respect, and the integrative processes of meaning making as foundational undercurrents of their processes. These undercurrents reflect valuable core characteristics of MMR—characteristics that may be fostered through a triangulation of perspectives generated through the deliberate and mindful use of multiple investigators through IT procedures. As IT’s potentials reflect many contemporary discussions in MMR, investigators are urged to consider use of this strategy and continue dialogue related to its development.
Footnotes
Acknowledgements
The author would like to thank the Canadian Child Health Clinician Scientist Program [CIHR], the Women and Children’s Health Research Institute, and the Faculty of Nursing, University of Alberta, for their support of her PhD research; Dr. William Hanson for his consultation; Dr. Cheryl Poth for her guidance and thoughtful comments on an earlier version of this article; and the anonymous reviewers and editorial team whose insights helped strengthen this article.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.