Elaboration,Generalization,Triangulation,and Interpretation

Article Categorization

To understand the prevalence of research designs, I began by conducting a comprehensive literature search, coding the 1,539 peer-reviewed research articles that appeared in the past six years (2009-2014) ¹ in four top empirical journals from the organizational literature (Academy of Management Journal, Administrative Science Quarterly, Organization Science, and Journal of Applied Psychology). These journals were selected given that they each regularly feature in the top five among empirical organizational journals based on total citations or impact factor or as summarized in ratings by organizational scholars (with specific rankings varying from year to year and based on whether the list is focused on psychology, management, or business publications). ²

A team of three coders conducted the categorization (the author and two research fellows), utilizing the definitions provided previously as the guide. We agreed on most categorizations, and in those few instances that we did not, discrepancies were resolved through discussion among the three coders. Most of these discrepancies between coders rested on two issues. First, as noted earlier, there were occasionally differences in understanding the distinction between “mixed method” and “multiple source.” Papers that included multiple sources of data were not considered mixed methods if these sources were all one type of data (e.g., all quantitative or all qualitative). For example, the coding team discussed several recent papers in Journal of Applied Psychology that included multiple studies and sources of data in a single paper. Often, all of the data in a given paper were quantitative (variables represented by numeric scores obtained in surveys or in experiments). According to our conceptualization, even though they included multiple sources of data, publications such as these were not considered mixed method unless some of the data were qualitative. This highlights a misconception often not well understood—simply including multiple sources of data does not necessarily constitute mixed methods.

The second reason for discrepancies was that in several instances, qualitative methods were used simply for preliminary or pilot efforts, for the purpose of developing a survey instrument, and then it was quantitative data alone that were utilized to develop or test theory. Taking heed of the intent in the call for papers for this special issue, that the purpose is to move our understanding of mixed methods “beyond simply having a main method and adding a bit of another method without integration,” discrepancies of this type were discussed among the research team, and it was only if it could be agreed that both qualitative and quantitative data had been utilized in theory development or testing that the paper was considered mixed method. An example of this occurred in the categorization of Autio, Dahlander, and Fredericksen (2013). One coder originally surmised that the authors had used interviews only for survey development (which resulted in quantitative data used to test hypotheses); however, a more careful reading of the paper, along with discussion among the coders, indicated that the interviews had also been drawn on for interpretation and theory development, hence this piece was considered mixed method. In most instances where categorization was ambiguous, we erred on the side of considering the piece mixed method. So, like others before us, rather than dismissing an article if one of the two methods was “weaker,” we considered it mixed method if theory development or testing resulted from its use (Molina-Azorin, 2012). For example, if media reports were content analyzed with a summary of occurrences, we considered this qualitative data that had been quantified, and if other sources of qualitative data were included, we categorized this as Type 4 mixed method/multiple sources.

Thus, source of data (specifically single or multiple) was revealed as important during our coding process for three reasons: (a) Number of sources was sometimes confused with number of methods, (b) use of multiple sources was often a technique used to extend a single method study, and (c) number of sources was often a technique that helped to establish an emphasis on one method over another in a mixed method study. Interestingly, rather than discrete categories, the review demonstrates that the use of multiple sources is a means by which emphasis on one method or the other can be achieved without negating the complementary strengths of the other method. We also noted that in some instances, the same authors were involved in multiple articles, all of which pertained to a common project. Each article may have used a single method, but across the articles, the project may have been mixed method. For example, one article may have reported on the qualitative portion of the project, while a second article may have presented a series of experiments. Although I focus here on the article as the unit of analysis, this is another intriguing alternative that I come back to in the discussion.

Results

As shown in Table 2, a meager 4.48% of the articles published in the four outlets (i.e., 69 articles out of the total of 1,539 that were published across the four journals between 2009 and 2014) are of the Type 3 mixed method/single source or Type 4 mixed method/multiple source. Type 1 single method/single source and Type 2 single method/multiple sources are by far the most prevalent during the period in question (together constituting over 95.52% of the articles published). Of the four journals reviewed for this article, the outlet most likely to publish mixed methods designs was Administrative Science Quarterly, with 13% of articles published during this period categorized as either Type 3 or Type 4. The outlet least likely to publish mixed method designs was Journal of Applied Psychology, with 1% of articles published categorized as either Type 3 or Type 4. Compared to prior reviews, this prevalence is lower than that reported previously yet coincides with prior reports, when articles are only interpreted as mixed method if both methods contributed to theory development or testing.

OPEN IN VIEWER

Table 2.

Comparison of Articles Published in Four Journals With Mixed Methods Single Source and Mixed Methods Multiple Sources 2009-2014.

Journal	Total Articles 2009-2014	Mixed Methods, n	Mixed Methods, %	Mixed/Single, n	Mixed/Single, %	Mixed/Multiple, n	Mixed/Multiple, %
Academy of Management Journal	379	17	4	7	2	10	3
Administrative Science Quarterly	112	15	13	1	1	14	13
Journal of Applied Psychology	550	6	1	4	1	2	0.36
Organization Science	498	31	6	10	2	21	4
Total	1,539	69	4.48	22	1.43	47	3.05

Approaches and Sources of Value

The aim of the second set of analyses was twofold: to understand (a) the approaches utilized in recent mixed method research and (b) what can be done to maximize the value of such research while minimizing the liabilities of it. To accomplish these aims, we conducted a thematic analysis of the 69 mixed methods articles published between 2009 and 2014, followed by an in-depth case study comparison of 4 papers (note that the 69 articles are denoted with an asterisk in the reference list).

Case 1: Qualitative Analysis of Experiences and Quantitative Modeling of Outcomes

Stephen Barley and his colleagues (2011) examined how people use communication devices to construct their availability to others and manage their work lives. Specifically, they provided a theoretically nuanced understanding of the relationship between email use and stress by showing that “one cannot understand how media induce stress without considering how properties of technologies become entangled with social norms, interpretations, and the flow of daily work” (p. 888). Most prior research had argued that email creates extra work in part because its asynchrony allows people to send and receive work-related messages at any time. Yet, as Barley et al. point out, researchers had seldom investigated the social forces that shape people’s experience of technologies.

Among a sample of 79 employees in a computer company, Barley et al. (2011) conducted 40 semi-structured interviews. They analyzed the transcribed interview data in multiple phases, following an inductive process that involved reading and coding the text numerous times (Lofland & Lofland, 1984). The initial phase involved reading through all transcripts to develop a general sense of the data and creation of preliminary codes. During subsequent phases, they developed second- and third-level codes that captured the various ways informants used and talked about email and other technologies. According to the authors, “These early analyses were invaluable in helping eliminate several lines of inquiry and focus our subsequent qualitative analysis on explicating patterns and relationships that appeared to be robust” (p. 896). During a third phase, the authors coded the entire set of interviews using the previously derived codes, then met to compare codings, discuss discrepancies, and refine understanding of when a code should be applied.

Barley et al. (2011) collected quantitative data from two sources. First, respondents recorded in a log book all work-related communication activities that occurred during the course of two workdays as well as that which occurred in the morning before they came to work, in the evening after they left work, and on one weekend day. Second, respondents completed a brief survey measuring which communication devices they used, stress (i.e., the emotional exhaustion subscale of Maslach and Jackson’s [1981] burnout inventory), and coping (Families and Work Institute, 1992). Variables capturing the outcomes “time worked,” “overload,” and “coping” were constructed from these two quantitative sources. Quantitative analysis techniques included hierarchical ordinary least squares regressions to assess whether being a manager, the number of communication events employing various media, and the time spent using various media influenced the length of respondents’ workdays, overload, or coping.

The final phase of their analysis was critical in combining the qualitative and quantitative data:

We identified patterns and relationships in the interview data and triangulated those against the quantitative data from the survey and logs…. All analytic memos included … exceptions to identified patterns, which led to additional analyses that helped us understand the reasons for the exceptions. We triangulated the patterns and tentative theories that emerged from these analyses with the quantitative data, which enabled us to explore further, affirm, and sometimes refute our understandings. Similarly, when our quantitative analysis pointed to relationships between variables, we turned to our interview data to deepen our understanding of these relationships. (Barley et al., 2011, p. 896)

This example illustrates that elaboration was enabled by the mixed method approach in that the quantitative data allowed the authors to uncover sources of stress that were not revealed in the qualitative data. Further, the quantitative data collection involved a wider number of respondents across a greater part of the organization, indicating that the findings were generalizable, not just unique to the small subset of the organization included in the qualitative study. Triangulation was obtained in that numerous patterns in the qualitative data were confirmed in the quantitative data, lending additional confidence that the interview process did not “manufacture” the results. Finally, the example illustrates increased capacity for interpretation in that through the additional data, it was understood that these added sources of stress were not in the foreground of informants’ accounts yet served a symbolic role.

At the same time, reflections from the lead author indicate a variety of challenges. First, he noted that scholars may have the best laid plans for a mixed method study, only to find that planned data sources are not available or that the organization serving as the field site will agree to one method but not another. In this instance, obtaining the communication logs was a first step, but then subsequently gaining permission for interviews and a survey was an even bigger “ask.” With regard to the analysis, Barley commented that mixed method studies can be very time consuming (he estimated that the analyses unfolded over three years on this paper) for many reasons, but chief among them is that sometimes results across methods are inconsistent and the researchers must then decide how to proceed when newly derived patterns don’t make sense. For example, Barley explained that in this study, the quantitative data didn’t support what people were saying in interviews (as much as he thought it would): “There were other things that were troublesome to these people (that turned up on the survey), other than email, but email was all they talked about, and so we had to figure out why.” This meant exploring alternative framing to arrive at a new focus. He also described having to learn new analysis techniques in order to aggregate data to the level necessary for full understanding, given the anomalies. Finally, Barley commented on the difficulty in publishing mixed method research:

The first version had hypotheses, and we got trashed for having them. So we got rid of them. We rewrote the paper, but then we got all kinds of pushback about the combination of the logs and the qualitative data from the interviews. And they didn’t like the regressions where we had the stress scale involved alongside the other types of data. I remember that I was mad enough to kill! But we did overcome it.

Case 2: Quantitative Modeling of Relationships Combined With Qualitative Analysis to Uncover Mechanisms

Gardner (2012) was interested in how performance pressure, defined as the importance of a team delivering superior outcomes, affects team knowledge use. Prior research had indicated that on the one hand, high pressure projects can cripple teams, resulting in members’ psychological withdrawal from the task (Pearsall, Ellis, & Stein, 2009); on the other, it also suggested that high pressure situations can provide contexts for teams to create and use their knowledge (Cannon & Edmondson, 2001). Gardner’s study involved collecting quantitative data pertaining to 72 audit and consulting teams from a global Big Four accounting firm, including a client survey to measure team performance, a survey regarding performance pressure completed by senior partners, and a survey with team members regarding their own and other members’ effort as well as existence, recognition, and use of general and domain-specific expertise.

Results of hierarchical ordinary least squares regression showed that as performance pressure increased, team members overly relied on general expertise while discounting domain-specific expertise, leading to suboptimal performance. Interestingly, contrary to her prediction, quantitative results also showed that use of general professional expertise did not enhance team performance. In true iterative spirit, Gardner (2012) explored why that might be the case with follow-on interviews with clients, who reported that when they hire a Big Four accounting firm, they expect project teams to have sufficient general professional expertise. But for a client to be very highly satisfied, the team must customize its work; and the more critical the project, the more this is important. This customization requires domain-specific expertise. Yet, when the teams needed domain-specific knowledge the most, they tended to use it the least!

In the third phase of the research, she sought to uncover specific group processes involved in this paradox. According to the author, “I designed my follow-up study as a combination of team observation and interviews to explore the behavioral mechanisms that underpin knowledge-suppression effects in teams, rather than to confirm or disconfirm hypotheses” (Gardner, 2012, p. 19). Specifically, she conducted case studies of six project teams across two firms, following each team through its entire project (3 to 10 weeks), observing at least one team meeting per week. This allowed her to understand how performance pressure varies at different phases of a team’s project, capturing more fine-grained data than the surveys allowed. To determine the level of performance pressure in each meeting, she extracted relevant passages from meeting notes and assigned a score on a 7-point scale. Finally, she coded the source material to indicate the two types of expertise, categorizing team members with the highest level of each type of expertise.

Analyses also consisted of mapping knowledge use behaviors of team members under different levels of performance pressure, overlaying role assignments onto team members, and evaluating how behaviors affect expertise contribution and use. She uncovered four team processes that seemed especially strongly associated with team knowledge use behaviors under performance pressure: (a) heightened drive for consensus, (b) increased propensity to focus on common knowledge, (c) switch in priorities from learning to project completion, and (d) emphasizing conformity to members’ position in the team’s hierarchy.

In uncovering these findings, the mixed method design allowed elaboration on a more comprehensive conceptual model to guide future research on teams’ knowledge use and effectiveness, highlighting the multidimensional nature of task-relevant knowledge. Generalizability was increased in that although the survey was only drawn from a single firm, quantitative findings were mirrored with the qualitative data, which was drawn from a considerably different professional service firm. Triangulation was evident in that the qualitative findings replicated the pattern found in the quantitative data, that teams rely on general professional experts more than on domain-specific experts under higher performance pressure, but it also helps us understand how this process unfolds, hence elaborating on what was found with the quantitative data. Finally, depth of interpretation was achieved in that the qualitative data showed that team members of all role types initiated, accepted, and reinforced the full range of knowledge use behaviors; there was no clear pattern suggesting that the general professional expert or the domain-specific expert was more or less responsible for the way the team used knowledge. Such insight could not have been gleaned from the survey alone.

Gardner commented on a number of challenges in this project, including needing access to increasingly more sensitive data as time passed in the project. She described a process of continually providing feedback at each phase as a means of relationship building. Perhaps the most salient challenge was the difficulty in “marrying up” the different types of data about each construct. For example, midway through the project, she had to find a credible way to categorize each meeting’s level of performance pressure based on the observational data, and she ultimately used some ancillary data that she hadn’t initially been sure were valuable in order to do so (i.e., notes about each meeting context based on discussions and observations before the team’s official meeting began). This highlights the resourcefulness that is often required in conducting mixed method research.

Case 3: Social Network Analysis Combined With Comprehensive Interviews

Reviewing the literature on social exchange and leader member relationships, Toegel et al. (2013) found little to explain why and how mangers engage in the process of helping subordinates with their negative emotions. To address these issues, Toegel et al.’s study was guided by four research questions: (a) Do managers actually incorporate the emotion-helping function in their behaviors? (b) What motivates managers to engage in the potentially unpleasant task of intervening in the management of others’ negative emotions? (3) Is emotion helping seen as in-role or extra-role behavior by those providing the care and those to whom it is provided? and (d) Do different types of emotion helping prompt different outcomes from discrepant interpretations on the part of managers and subordinates?

They answered these questions in a sample of 67 employees of a recruiting agency that specialized in providing managerial staff to retail outlets. First, they collected social network data concerning who went to whom for emotion help with negative emotions. They calculated homophily, that is the extent to which people nominated same-category others as those who helped manage and alleviate their emotional distress. In a second phase, they conducted 30 semi-structured interviews asking participants to tell about a situation in which they experienced stress, anxiety, tension, or emotional pain. Analysis progressed from extracting instances concerning incongruent expectations and reciprocity, which they characterized as “informant sensemaking,” to then comparing data from different interviewees to develop higher order analytic dimensions (Van Maanen, 1988), and finally arriving at more general emergent themes useful for building an emergent theoretical model.

They found that emotion help was provided because managers thought that organizational and economic benefits would accrue from helping others cope with their negative emotions or because they felt it was an important skill that could be used in other areas of life. Most provided the help because they thought they would gain something back from the provision, specifically, recognition of their efforts from subordinates. Perhaps their most interesting insight pertained to the incongruence in expectations. As they state, “employees treat caring as part of the mangers’ role that requires no reciprocation, whereas managers see such help-giving as discretionary extra-role behavior that requires reciprocated commitment” (Toegel et al., 2013, p. 334). Disconfirmation of the managers’ expectations resulted in anxiety, insecurity, and helplessness, whereas for employees, it diminished trust in the supervisors.

The mixed method design utilized by Toegel et al. (2013) enabled elaboration on the social exchange literature, illuminating a curious incongruity, as stated by the authors, “that helping behavior designed to ameliorate negative emotions may itself generate negative emotions on the part of managers waiting in vain for employees to replay their kindness” (p. 350). The mixed methods were instrumental for increasing generalizability because they enabled derivation of more general themes that are likely applicable even outside this particular set of network actors, beyond the service sector. The interviews also enabled triangulation, which would not have been as compelling had interviewees not commented directly on the importance of the level at which the help had been provided. Finally, their pairing of the social network data with interviews also enhanced interpretation of the patterns uncovered in the network analysis, which may have been elusive without greater understanding behind the motivations for provision of help uncovered in the interviews.

Toegel reflected on the challenges of this research. She mentioned that unlike single method studies, in which participants are approached only once (or in the case of archival data, may not be approached at all), employees in this study were approached multiple times by the researchers. One employee commented, with a bit of irony, “Is there a third phase of your research?!” Hence, there is the danger of respondent fatigue, which is not as prevalent in other research designs. To combat this, the research team were clever in offering feedback on network centrality to employees following the network data collection, which increased the motivation of people to participate and strengthened the relationship with respondents to enable more candid conversations during the interviews. Toegel also commented on difficulties during the review process, similar to those noted previously by Barley. At least one reviewer was not enthusiastic about the inclusion of the network analysis, but Toegel and her team were adamant that it enabled a deeper interpretation and were insistent with regard to retaining it in the paper. The dramatic benefits of pairing these different methods shows that this persistence paid off.

Case 4: Content Analysis Combined With Quantitative Survey Data

An insightful paper by Bezrukova et al. (2012) provides yet a different example of the means by which qualitative and quantitative data and analysis techniques can be combined. The result was richer and also more discriminating analysis and important contributions to the diversity literature. The authors examined the effect on performance of a specific type of group split: informational faultlines (members’ alignments based on skill, knowledge, and expertise). A key contribution of this piece stems from the ingenious qualitative approach to capturing a condition that may alter the direct effect of faultlines—specifically, the group and departmental cultural environments that provide contextual influence on the groups. As the authors state, in doing so, they were able

to integrate the literature on faultlines and organization-based culture by theorizing about the specific content of cultures (results-focused), recognizing the multilevel nature of organizational cultures, and showing how cultural alignment across organizational levels may operate as a moderating variable that influences performance in groups with faultlines. (p. 85)

Uncovering this finding would not have been possible without their mixed methods approach. To capture group-level results-focused culture, company guides were content-analyzed to understand the extent to which results-focused values were expressed (e.g., see Abrahamson & Hambrick, 1997; Kabanoff, 1997). The number of times a results-focused value (e.g., “competitive thinker,” “results driven”) had been mentioned as being representative of the group was counted. Next, a measure of department-level culture was generated by content-analyzing 5,844 pages of archived open-ended text responses contained in employee surveys. Key terms representing a results-focused culture were systematically derived by multiple raters, and excerpts containing these words were extracted (several lines of text occurring before, during, and after a search term). Following the procedure of Gibson and Zellmer-Bruhn (2001), the raters scored each employee response included in the analysis by counting the number of search terms that occurred in the response. To control for differences in the number of words across responses (the greater the length, the more likely that a given term would appear), they further divided the number of occurrences by the total number of words in the response. “In-context verification” was performed to ensure that the words were used as suggested by the results-focused culture definition.

Quantitative data captured faultlines using the algorithm developed by Thatcher, Jehn, and Zanutto (2003) and used by others (e.g., Bezrukova, Spell, & Perry, 2010), based on multivariate statistical clustering analysis (e.g., Jobson, 1992; Sharma, 1996). Group performance was measured using group stock options and group bonuses. Three-level hierarchical linear modeling (HLM3; Bryk & Raudenbush, 1992; Raudenbush, Bryk, Cheong, & Congdon, 2000) was conducted given the hierarchical structure of the data, with observations at one level of analysis (individuals) nested within a second level of analysis (groups) and within a third level of analysis (departments).

Utilizing the qualitative approach to understanding the cultures enabled elaboration on the boundary conditions for faultline operations. The piece is also a good example of triangulation and generalization assets of mixed method research. As the authors point out, their approach is not plagued with percept-percept bias. The practical implications from the research were also well grounded in the richer understanding of context that resulted from their research design. They suggest strong leadership may result in a cultural alignment throughout various organizational levels that can help to leverage demographic-based faultlines that are prevalent in the current workforce. This can be accomplished either in formal ways (e.g., letters or memos) or through socialization tactics—where group members experience common learning with other groups in their department. Finally, by taking care to represent cultures qualitatively at two different levels (group and department) using two different sources of qualitative data, interpretation was enabled—the authors derived the notion of alignment across the two levels as key for faultlines. This interpretation may not have arisen without the qualitative/multi-source data (e.g., had they only obtained information about culture from one source).

Reflections from the lead author, though, did reveal some challenges. Like Barley and his colleagues, Bezrukova and her team found that often companies provide

random data and you have to make the best out of what you get. In our case, we were lucky and were provided with multiple data sets that could be merged since they had subsets of the same employees represented in each data set. Mixed methods were a solution to this.

Enhancing Elaboration in Mature Domains

In designing research, scholars often seek to select a method that will enhance scientific and practical contributions while at the same time minimizing threats to rigor and balancing time, effort, and resources. Prior reviews have emphasized the choice among quantitative, qualitative, or mixed methods, indicating that mixed methods are particularly appropriate where the goal is to increase validity of new measures or to generate greater understanding of the mechanisms underlying quantitative results in at least partially new territory (Edmondson & McManus, 2007). Less insight has been offered regarding the use of mixed methods in mature research domains. Extending prior reviews, comprehensive analysis of the various approaches in the 69 mixed method articles revealed that even fairly well documented and mature domains of organizational research can benefit from a mixed method approach that is finely tuned to elaborate on shortcomings in prior construct or model development. One of the primary reasons the elaborate deep-level theoretical contributions are possible is that the specific mixed method approaches selected fit the research questions being asked. It wasn’t just a matter of recognizing that mixed methods would be stronger than a single method; rather, the specific mixed method approach was critical in order to obtain the necessary theoretical contributions for a given set of research aims.

For example, Bezrukova et al. (2012) addressed a research domain (faultlines) that has received a tremendous amount of attention in the past 10 years using both qualitative and quantitative methods. Specifically, faultline operationalization has reached a very high level of quantitative sophistication. As a result, if Bezrukova et al. had utilized an inductive initial approach, their insights may have yielded little additional theoretical development beyond the current state of the literature. Focusing on the boundary conditions for faultline effects, the approach they developed, in which text data were qualitatively analyzed and then quantified for use alongside the sophisticated quantitative faultline operationalization, allowed for the set of theoretical extensions they sought. Their research is notable because very little is known about how cultural environments may shape the relationships between group faultlines and outcomes. Presumably, one of the stumbling blocks to exploring this issue in the past is that cultural environment is difficult to capture empirically and even more challenging given that multiple cultures must be captured if variance across group faultlines embedded within them is to be examined (which is important for linking these two constructs). While it might be less onerous to document a single cultural environment richly and deeply, to do so across 138 groups (as did these authors) is nearly an insurmountable task. Hence, the content analysis approach to mixed methods design was an excellent solution. The possibility of utilizing this approach in such a way extends prior reviews (e.g., Greene et al., 1989) that indicated it may be helpful to use naturalistic qualitative methods to assess context first and then use positivist quantitative methods for other purposes later in the study. Bezrukova et al. applied a different type of qualitative analysis (content analysis) simultaneously with their quantitative approach to understand context, and they did so in a relatively mature research domain (i.e., team faultlines), extending it in new and exciting ways.

As another example, Barley et al. (2011) initiated their project because they were skeptical that the role of communication technology can be revealed simply by noting patterns of use. Instead, their research questions pertained to the social norms and interpretations surrounding use. Given this question, it made sense that qualitative data were needed to induce the potential symbolic aspects of use. As the authors themselves note, “Combining both types of data is valuable because it not only allows one to confirm common findings across methods (Jick, 1979), but just as importantly, one can identify dynamics obscured by one data source or another (Bernard, Killworth, Kronenfeld, & Sailor, 1985).” Analyzed alongside the interviews, the quantitative data documented patterns of use and provided a direct measure of stress that helped the authors to point to sources of stress that were not emphasized in the informant’s interview accounts because (according to the authors) “they were not figural to their work culture.” It also allowed the authors to uncover the seemingly contradictory evidence among respondents that although email was “all they talked about,” there were other things on the survey what were troublesome to these people, thus highlighting the symbolic nature of email. Prior reviews have indicated that mixed methods can be used with the explicit intent of uncovering paradox and contradiction (referred to as the purpose of “initiation” by Greene et al., 1989) yet have also noted this is rare. In fact, Barley did not indicate this was an explicit purpose, rather that it emerged as the analysis unfolded.

Toegel commented that being very clear about what the analysis across methods is designed to achieve can be critical—is the emphasis on elaboration, interpretation, or triangulation? In her case, it was on elaboration of findings that were difficult to understand purely on the basis of the social network data, and while this guided her interview protocol and qualitative analysis techniques, it also implied that the qualitative data collection and analysis should occur following the network analysis. This departs from some recommendations that have appeared in the literature, which indicate that if “complementarity” is the goal (i.e., using the results from one method to elaborate, enhance, or illustrate the results from the other), then the mixed methods should be used simultaneously (Green et al., 1989, p. 266). Such an approach would not have worked for Toegel et al., whose insights were made possible specifically by the sequence of the design. Generally, there has been recognition in the social network literature of the importance of understanding the psychology that undergirds ties (for a recent perspective on this, see Casciaro et al., 2015). But gaining this understanding likely requires mixed method approaches, and as Toegel et al. have demonstrated, in particular supplementing traditional network analysis with inductive techniques can be fruitful but necessitate a very different skill set and frame of mind. The key lesson here is that even in mature research domains, elaboration on phenomena that could be hidden by a single method is made possible using mixed methods, and this is contingent on selecting the approach to mixed methods that enables this, given the particular research aim.

Utilize Iterative Processes to Expand Access and Improve Generalizability

Moving into data collection, the gaining of access to firms, teams, respondents, or data is often the bane of researchers’ existence, and many would soon have it over and done with as quickly as possible. Many research design handbooks indicate that data collection should be strategic and pre-formulated and conducted according to plan (Tashakkori & Teddlie, 1998, 2003, 2010; Teddlie & Tashakkori, 2009); although several do acknowledge that emergent designs might be viable, these are not as well elaborated (Creswell & Plano Clark, 2011). The analysis here revealed, however, that often important knowledge comes from unexpected sources during the process of conducting mixed method research and iterative, negotiation, and relationship-based techniques are essential for improving access and generalizability and, more generally, ensuring rigor and impact.

The authors interviewed all indicated that it is important for researchers to remain open to insights and data sources, even if they are not part of the original design. This is part and parcel of a proximal approach to theory building (Gibson et al., 2012), hence the reason mixed method designs may be particularly well suited for such endeavors. For example, Gardner advised that researchers suspend assumptions while in the field about what data might be crucial, remaining open to collecting and utilizing data, even from unanticipated sources, given it might prove valuable later in the process. Drawing on the metaphor of a powerful vacuum cleaner, she claims her motto in research since the project described here has been, “Hoover up all the data I can get my hands on while it’s available.”

The research teams I have participated in have gleaned similar insights. For example, in one project, we planned to collect data using interviews and longitudinal surveys. As part of our relationship building phase with the site, we were invited to attend a face-to-face meeting. Doing so initially seemed outside the scope of our research purview. However, upon accepting the invitation, we realized that attending such meetings constituted a valuable ethnography that could reveal phenomenon not likely to be illuminated in the interviews or surveys, hence we requested and received permission to attend all such meetings during our research process, even traveling overseas in order to do so. In the end, the most valuable comparisons were made during the ethnography, allowing us to extend our findings to include non-domestic operations and intercultural issues within an expanded scope of theory building as well as to deliver important feedback to the firm’s headquarters about regional challenges faced by study participants. It should be noted that remaining open to data sources is not the same as the “paramedic” use of mixed methods described by Greene et al. (1989). In their analysis of articles published in educational science, they noted that “qualitative data often appeared the emergency room of report writing as a life-saving device to resuscitate what was either a failed program or a failed evaluation” (p. 269); yet they also argued that mixed method studies had not yet tested the limits of their potential and suggested a more integrated approach. Authors in our examination appeared to have heeded this advice in that the openness to data and resulting mix of methods wasn’t intended to “fix” problematic results but rather in order to expand explanations and create a deeper understanding.

Several of the lead authors mentioned that being able to navigate between the worlds of academia and practice was instrumental, as was being willing to provide feedback and consultation to respondents during the project. Gardner described a process of starting with a small request (e.g., interviews with some relatively junior professionals) and then using what she learned from that to inform the senior client contact such that he saw the value of the study and then advocated for her to get access to several executives. She then used insights from those interviews to persuade them to conduct team surveys in their divisions. As Gardner commented, “I never asked them for access to new data sources until I had a fresh insight I could share with them from my ongoing research. Ultimately, they granted me access to their clients, which is extremely delicate.”

Barley suggested that gaining such access often involves negotiating sample size for subsequent phases of the study. If analyses are to be triangulated across methods, this may require matching participants from phase to phase, hence the entire study may end up with a smaller sample across methods. Yet, such a compromise is likely worthwhile. As the Barley et al. (2011) piece illustrates, being willing to accept the smaller sample overall allowed for quantitative and qualitative data for each respondent to be compared, hence increasing the trustworthiness of the data while still maintaining breadth across the organization.

Coordinating and synchronizing multiple data sets mean that logistics and project administration become critically important. Bezrukova recommends careful protocols for labeling and storing data sets. She also notes that given mixed methods often mean the merging of data sets, if one anticipates using a mixed method approach, it is helpful to know some programming language (e.g., SQL) or to invite someone to join the research team who does. Other tactics such as keeping an updated data dictionary with information on data format are also helpful. Bezrukova explained that these techniques are particularly necessary when conducting mixed method research to be sure that results are reproducible. Likewise, Barley suggested attention to these details before embarking on the research in order to be efficient during the analysis process.

Equally important is building a team with a variety of methodological expertise and revisiting with discussions among the team the emphasis placed on and insights provided by the different methods at numerous junctures. The importance of planning research team composition and methodological skill breadth was emphasized by Hurmerinta-Peltomaki and Nummela (2006), but Bezrukova also emphasized the importance of an iterative approach in which findings from one phase inform the data collection and the addition of research team members for subsequent phases, ensuring greater interpretability across the project. This approach was corroborated by Barley, who suggests using specific, concrete findings from one method to inform explorations that can be continued with a different method. In his project, instances recorded in the communication logs informed specific questions the researchers asked of those respondents during interviews, hence they deviated from a structured interview protocol to help solve the puzzles that arose. The prescription here is that impact on both knowledge generation and practice can be increased if negotiating access and designing data collection strategies are ongoing processes, informed by emergent insights, with generalizability in mind.

Obtain Triangulation Through Shifts in Sequence and Unit of Analysis

As they reflected on the analysis phase of research, many researchers lamented the difficulty in developing consistency across methods, yet obtaining triangulation is often an explicit goal of mixed methods research! Triangulation is less precarious in the approaches that involve content analysis and quantification of text data and in the case of comparisons. These approaches share aspects of what Creswell (2003) referred to as “concurrent designs” and those categorized by Molina-Azorin (2012) as “simultaneous,” which involve collecting both forms of data at the same time and then integrating the information in the interpretation of the overall results. Yet determining the order and priority when methods are sequenced may be more challenging. The interviews and examination of articles here revealed a number of strategies to achieve triangulation by expanding the analytical options, including decoupling the sequence of analysis from the sequence in which data was collected, shifting the unit of analysis, and revisiting the literature to help inform analysis.

First, researchers may need to reconsider the sequencing of their analysis as it unfolds. Although one may design a project to begin with collection of one data type followed by a second data type, it may be that the data collected first are most useful for theory development if they come into play later in the analysis. This process is different than the “emergent mixed method design” described by Creswell and Plano Clark (2011), which occurs when a second approach (quantitative or qualitative) is added after the study is underway because one method is found to be inadequate. Rather, interviewees indicated that a mixed method study may be designed with a particular sequence in mind at the outset, and the order of data collection may have a particular intent, but the usefulness of the data might change during the analysis process, necessitating a resequencing. In fact, beginning analysis with the data that were collected first (in time) just because that was the sequence of data collection may make triangulation more difficult.

For example, it is interesting to note that although Barley et al. (2011) collected quantitative data first, their analysis began with the qualitative data (hence our coding of the article as Qual-Quant). They first identified patterns and relationships in the interview data and then triangulated those against the quantitative data from the survey and logs, noting the areas of discrepancy as potentially ripe for theory development. This suggests that researchers should not be shy about reordering the sequence of analysis based on an emerging sense of the findings even if the analysis sequence then differs from the data collection sequence. So decoupling the sequence of data collection from the analysis and then shifting back and forth between types of data analysis is a technique for achieving triangulation. Although they collected surveys and interviews sequentially, Barley et al. (2011) reported “teleporting back and forth among the qualitative data, the quantitative data, and emerging categories and relationships” (p. 893). Hence, although they described “phases of the analysis,” they note that the analytic processes that characterized each phase were rarely confined to that phase.

Gardner indicated that analysis is even more challenging in mixed method, multilevel research as findings obtained at one level or with one method may not coincide with those at other levels achieved with different methods. Addressing this issue may require shifting the focal unit of analysis, which in her case became the team meeting. As a result of very similar issues, I am currently working on a mixed method project that has required shifting the unit of analysis from the team to the transitions that each team makes in their activities over time. Given each team typically makes three to four such identifiable shifts each year, analyzing the transitions is more appropriate for the dynamic focus that has emerged in the piece, and it increases the number of observations. But it also requires a shift in the conceptualization and theory development because the emergent theory becomes one of transitions as opposed to being centered on the team. Finally, it requires multilevel modeling techniques, which have different parameters and constraints. So, a very practical lesson from these design challenges is that value can actually be maximized if researchers remain open to such shifts even if they are as dramatic as changing the unit of analysis.

Coinciding with openness in analytical approach, researchers will need to be willing to revisit the literature as they are conducting analysis. As Gardner describes, initially she generated a set of descriptive codes from prior research to categorize segments of text according to certain classes of phenomena. Next, she assigned interpretive codes to data units, reflecting a deeper understanding of the situation dynamics and to offer additional insight in to the initial codes; this meant “modifying the list of descriptive codes to capture additional actions embedded in the sequences of action” (Gardner, 2013, p. 21). She then analyzed the coded data to uncover behavioral patterns across larger passages, which depicted how initial codes fit together. As she put it, “I sought meaning in the patterns as it related to my research question…. I iterated between the data and existing small groups research to understand how data units fit together into meaningful processes” (p. 23). This process of moving between literature, data, and emerging theory may be foreign for those accustomed to a more deductive approach, and if the research team consists of members having different comfort levels with it, negotiation within the research team will be necessary. In summary, important prescriptions for achieving triangulation include being willing to shift unit and order of analysis as well as revisiting the literature during analysis.

Including Benefit of Mixed Methods to Fortify Interpretation for Publication

Finally, several researchers commented on the publication process and that incorporating strategic framing about the value of both qualitative and quantitative methods within the submission itself and during the review process can be critical for winning over “purists” who discount one method or the other. The key is to emphasize how the combination of methods lends itself to deeper, novel, or more useful interpretations. Although handbooks on conducting mixed methods research sometimes provide guidance on the structure of a writing a mixed method article, they rarely address this aspect of the publication process in much detail. For example, in a very helpful section on writing and evaluating mixed methods research, Creswell and Plano Clark (2011) suggest including reasons that mixed method research is appropriate to address the research problem, but the analysis here extends this to suggest the strategic framing of findings to illustrate the depth of interpretation made possible by mixed methods, both within the article itself but also during the submission and revision process.

For example, Gardner (2012) was able to refine the construct of performance pressure through her iterative, mixed method approach, and she emphasized this aspect of the findings in order to convince both reviewers and readers of the value of such designs. As she stated,

Combining the results [from the qualitative study] with my survey findings allowed me to develop a more comprehensive conceptual model of performance pressure’s effects on team processes and outcomes … my findings highlight an irony of team life: when teams are under pressure to perform at their best—and even when they are motivated to do so—they engage in behaviors that actually constrain their performance. (p. 32)

Interviewees also commented on the need to persevere when one has a strong case for the value of the mixed method approach. Gardner described how she initially failed to convince one reviewer on the merit of the qualitative study but that he came around by the second revision: “If I hadn’t been so personally passionate about the idea, I may have been tempted to listen to his advice.” Seeking encouragement from others who have made the journey to publication successfully using a mixed method approach can help to address this challenge. For those who aim to pursue mixed method research, Barley advises the following:

If your analyses are empirically viable, you have been thorough and careful, and you get a review that is driven by a reviewer’s religion (methodological or theoretical), you need to stand up for your work. Sometimes you will get rejected, but often you will get a receptive audience.

Often, the balance between depth and breadth across methods and levels is precarious. It may be impossible to conduct as deep an analysis as might have been optimal for each type of data in one paper. To address this, it is helpful to consider what the primary data sources are, provide depth around analysis of those data, and then utilize the secondary data sources in a less elaborate manner. Another strategy alluded to earlier is that authors might consider publishing each method separately in different papers. The project is mixed method, but any given article from the project becomes single method. Yet, this prohibits the aim of increasing elaboration or interpretation within a single article and relies on readers’ willingness to read multiple articles, which may not be available simultaneously (e.g., in the same publication year) or in the same outlet; hence, the reason I focus here on mixed method articles.

In my own publishing efforts using mixed methods, an important key to success has been to focus on what is revealed by one method that would have been obscured without the other. Quantitative scores often do not reflect the “shades of grey” that comprise the actual experience of work by respondents, and qualitative evidence can provide these interpretations. Explaining these nuances requires a very comprehensive understanding of the current state of the literature so that one can have confidence in the novelty of the findings to strengthen the case for mixed methods. Scholars have offered suggestions for a variety of means of integrating quantitative and qualitative data, including using data analysis software or using meta-inferences (Johnson et al., 2007; Teddlie & Tashakkori, 2009). Likewise, our interviewees indicated the criticality of integration for publication success.

In conclusion, mixed methods offer great potential for increasing rigor and impact yet remain rare in the organizational literature. Increasing the representation of mixed methods research is a matter of proactively addressing the challenges and practical liabilities of such designs, even before embarking on the research, but equally importantly, remaining open to a dynamic and emergent approach during the data collection, analysis, and publication process. Doing so enhances the value of mixed methods for organizational research.