Applying Mixed Methods Research to Conduct Human Resource Development Inquiry: An Update

Abstract

Scholars of human resource development (HRD) and related fields should stay abreast of mixed methods research developments to advance their scholarship. In support this idea, two companion works published in Human Resource Development Review (Hitchcock & Newman, 2013; Newman & Hitchcock, 2011) were offered to help HRD scholars embrace the concept of mixed methods research. It is now approaching 10 years since the second of these two articles was published and, since then, there have been important mixed methods research updates that can facilitate HRD inquiry. This article therefore contributes to Human Resource Development Review’s Instructor’s Corner by reviewing: (a) updates in paradigmatic thinking that support the use of mixed methods, (b) new approaches to integration, and (c) legitimation approaches that offer a validity framework for understanding mixed methods design quality. These three descriptions include discussion around how they might be applied in HRD research.

Keywords

credibility techniques dialectical pluralism human resource development integration legitimation mixed methods research

Applying Mixed Methods Research to Conduct Human Resources Development Inquiry: An Update

Mixed methods inquiry should be of interest to scholars of human resource development (HRD) because of the synergy that could be created by integrating qualitative and quantitative research (Johnson & Onwuegbuzie, 2004), which can in turn yield a better understanding of phenomena. Furthermore, discussion and thinking around use of mixed methods continues to expand. Indeed, sources that describe the merits of mixed methods research are numerous and varied, as they appear as books, handbooks, textbooks, book chapters, and articles (e.g., Bazeley, 2018; Creamer, 2018; Creswell & Plano Clark, 2018; Fetters & Molina-Azorin, 2017; Greene, 2007; Greene et al., 1989; Hesse-Biber & Johnson, 2015; Hitchcock & Onwuegbuzie, 2022; Johnson & Christensen, 2020; Johnson & Onwuegbuzie, 2004; Klingner & Boardman, 2011; Maxwell, 2004; Mertens, 2007; Onwuegbuzie & Johnson, 2021; Plano Clark & Ivankova, 2016; Ridenour & Newman, 2008; Tashakkori & Teddlie, 1998, 2010; Teddlie & Tashakkori, 2009).¹ Furthermore, there are now two specialty journals, launched in 2007, focusing on mixed methods: the Journal of Mixed Methods Research and the International Journal of Multiple Research Approaches. There are also relatively new special interest groups in established research organizations (e.g., the Mixed Methods Special Interest Group housed within the American Education Research Association and the Mixed Methods in Evaluation Technical Interest Group housed within the American Evaluation Association). There is even an association dedicated to the study of mixed methods (i.e., the Mixed Methods International Research Association) that has been operating since 2005. With respect to HRD research, HRDR published two companion works (Hitchcock & Newman, 2013; Newman & Hitchcock, 2011) to help human resources scholars embrace the concept of mixed methods research. Similar journals have published works on mixed methods guidance for this field (e.g., Newman et al., 2014; Onwuegbuzie & Corrigan, 2014). If the past is a prologue to the future, we can expect an ever-burgeoning scholarship base for the study, advancement of, and advocacy for mixed methods research within HRD and beyond.

The purpose of this contribution to Human Resource Development Review’s (HRDR) Instructors’ Corner is to offer readers an updated report on three ongoing advancements in mixed methods research. The three areas of advancement covered here are: (a) updates in paradigmatic thinking that support the use of mixed methods, (b) new approaches to integrating qualitative and quantitative research, and (c) legitimation approaches that offer a validity framework for understanding mixed methods design quality. It is my hope that promoting awareness of these advancements within the HRD community will serve to support ongoing use of mixed methods research in the field.

Having covered this article’s purpose, I offer one delimitation and three caveats. The delimitation is that because this is should be a HRDR Instructor’s Corner piece, this article does not offer empirical findings; it instead summarizes ideas from other sources to explain how the three advancements might be applied in HRD research. The three caveats are: (a) the ways in which the mixed methods field broadly addresses the three advancements covered in this article are not uniform (i.e., the ideas offered here are not universally agreed upon), (b) related thinking is subject to change, and (c) readers should assume that, had some other mixed methods person wrote this piece, other areas of advancement might be identified/highlighted. However, I expect readers will see the three areas I have chosen to review are broad and foundational, and the citations used to support my points show related discussion in the literature is recent, which is important because this work is meant to be an updated discussion of mixed methods research.

The Advantages of Mixed Methods Inquiry: An Example

Before proceeding with a discussion of the three areas of advancement, readers might appreciate a primer on why they might bother with mixed methods in the first place. The fundamental reason for conducting mixed methods research can be seen in a definition for this form of inquiry; Johnson et al. (2007),² define MMR as:

the type of research in which a researcher or team of researchers combines elements of qualitative and quantitative research approaches (e.g., use of qualitative and quantitative viewpoints, data collection, analysis, inference techniques) for the broad purposes of breadth and depth of understanding and corroboration. (p. 123)

An important idea to glean from this definition is the notion that qualitative inquiry and quantitative inquiry are both used to promote breadth and depth when researching phenomena. This leads to the overall justification for mixed methods, which is that the use of different approaches to inquiry entail a form of weakness minimization, such that the strengths of one method (or approach) compensates for the weaknesses of another method (Nastasi et al., 2021; Onwuegbuzie & Johnson, 2006; O’Cathain, 2010), yielding a form of methodological synergy (Fetters & Freshwater, 2015; Onwuegbuzie & Hitchcock, 2019). In other words, the whole of mixed methods inquiry can be greater than the sum of its methodological parts.

Consider for example an effort to understand employee turnover and related hiring challenges in the leisure and hospitality sector as it struggles to deal with the COVID-19 pandemic and society’s fitful recovery. One might use a survey to understand job factors that promote turnover and inhibit hiring in the sector, but also to understand protective elements that ameliorate stresses and make work palatable. But some trepidation for such a survey project is warranted because the context is complicated and volatile. Consider that peak layoffs in this sector within the United States occurred in March 2020 (5.2 million; Congressional Research Service, 2022) and that leisure and hospitality work was especially vulnerable to the nature of pandemic mitigation efforts (business closures, staff being asked to enforce mask mandates among businesses that remained open, increased sanitation tasks, etc.); furthermore, related work remains complicated because employee and customer fears of infection must still be managed.

Adding yet more complexity is the U.S. federal government offered unusually supportive unemployment benefits during 2020, creating a new but temporary disincentive to work. However, as of this writing, nearly 25% of national job gains in the economy fall within this sector and there remains a limited number of workers relative to available openings (Soto, 2022). In sum, this represents a research scenario that is influenced by relationships between wages, high inflation, job stresses, supply chain challenges, and health concerns due to changing virus variants. The relationships between these factors are likely non-linear and likely interact with employee characteristics such as worker age, gender, education level, other employment opportunities, and their overall intersectionality; that is, people work within dynamic employment contexts that shape their identities (American Psychological Association, 2017). Given these complexities, it seems unlikely that surveys designed to understand hiring in the gambling industry (should any surveys even be available) would adequately capture the phenomena of interest and a researcher would be therefore tasked with developing a complex research design solution (Nastasi et al., 2010; Poth, 2022).

For sake of working with a specific HRD example, suppose that a research task at hand is determining the incentives that will entice experienced workers to return to and remain in a job within the gambling industry.³ Given the context at hand, it seems advisable to think carefully about survey item writing, item coverage, item ordering, and survey completion incentives (see Dillman et al., 2014 for an overview of survey writing matters). Hence, unless a researcher is already well-embedded in the gambling industry it would seem to be a serious challenge for even experienced survey methodologists to understand the nuances that influence effective survey instrument design and administration. Fortunately, a mixed methods approach to support survey design (Onwuegbuzie et al., 2010) can be used. Given the example at hand, the Exploratory Sequential Design (Creswell & Plano Clark, 2018) should be useful. See Figure 1.

Figure 1.

The basic exploratory sequential design applied to the gambling industry example.

This design can demonstrate the aforementioned ideas around qualitative-quantitative synergy and weakness minimization. With respect to weakness minimization, consider general strengths of qualitative inquiry. Qualitative work tends to focus on naturalistic, interpretive discovery and co-construction of knowledge between the researchers and participants (Lincoln & Guba, 1985; Maxwell, 2004; Nastasi et al., 2021; Schwandt, 2000). That is, related methods allow information to emerge from relatively open-ended and exploratory data collection processes (e.g., interviews, focus groups, observations, document analyses) that can then be analyzed and interpreted. Having an initial qualitative stage allows investigators to explore, make conjectures, test their understanding, and refine their initial expectations and beliefs. Furthermore, although initial qualitative work within the overall design is, by definition, exploratory in nature, it can also be incisive, systematic, and rigorous by deploying design advice such as by using established sample size considerations (e.g., Guest et al., 2006). In addition, several well-established credibility (a concept similar to or arguably the same as validity, or some researchers prefer the word trustworthiness) techniques designed to check assumptions, assertions, and findings can be deployed (e.g., Brantlinger et al., 2005; Lincoln & Guba, 1985; Nastasi et al., 2021; Newman & Hitchcock, 2011; Patton, 2015). See Table 1 for a broad description of different credibility techniques.

Table 1.

Qualitative credibility techniques and application to HRD research.

Technique	Description	Example HRD Application
Audit trails	Keeping track of records, dates, and other study information to help substantiate claims	Maintaining well-documented case files to allow for later data checking to inform assertions
Collaborative work	Involving multiple HRD researchers and staff to inform data interpretation, inter-coder reliability, etc., and obtain multiple stakeholder perspectives	Relying on multiple staff (e.g., managers, trainers, sales staff administrative assistants) to promote understanding of human resource programs
Disconfirming evidence	Search for discrepant/contrary information after establishing initial themes and conjectures	Testing and refining initial findings by searching for disparate information when conceptualizing employee programs
External auditors	Working with people who are otherwise removed from the study to check findings and help ensure they are grounded in data	Working with employees and staff who are more removed from a program to offer outside assessments of program needs and effectiveness
Member checks	Asking study participants to review the accuracy of initial interview findings, raw notes, and/or observation data	Sharing raw data and/or sharing initial assertions with managers, employees, customers, and other stakeholders to help assess understanding of human resource programs
Peer debriefing	Seeking critical feedback from colleagues to support accuracy of interpretations and reporting	Seeking feedback from specialists with expertise in a particular approach to research, topic, program and/or industry
Prolonged field engagement	Conducting enough observations and interviews to ensure adequate understanding of context	Spending adequate time and effort to deeply understand the HRD context; including staff with longstanding familiarity with a company and/or industry to inform inquiry
Researcher reflexivity	Self-disclosure of personal assumptions and biases to promote self-reflection and report transparency	Taking stock of prior beliefs around what employee programs work, when, and for whom
Thick, detailed description	Providing enough information for users of HRD reports to promote full understanding of conclusions	Provision of full, detailed HRD reports that help end-users of reports to determine the degree to which an approach might work in a new context
Triangulation	Examining convergence (and divergence) of evidence across different data sources, stakeholders, methods, and theoretical perspectives to inform broader HRD findings	Using employee records, manager reports, staff interviews, on-the-job observations, and assessments to develop and later to validate explanations for employee behavior

Note. See Lincoln and Guba (1985) for more detailed discussion. Brantlinger et al. (2005).

and Nastasi et al. (2021) provide examples of how these techniques might be used in.

in education research.

Applying these techniques

Member checks can promote credibility because they entail checking interview findings or even raw data with interviewees. Likewise, one can triangulate information across different resources (e.g., documents and interviews) to corroborate evidence and understand disparate findings drawn from component methodologies. Researchers might purposefully search for and openly address any disconfirming evidence during analyses. Furthermore, researchers might use any number of data analysis techniques (e.g., first and second cycle coding methods like exploratory coding, holistic coding, axial coding, and pattern coding) that are well established in the literature (see Saldaña, 2016). Hopefully these techniques provide a sense of how in-depth qualitative inquiry can be when studying any given topic.

Getting back to the example at hand, suppose a researcher initially found, through qualitative inquiry, five key themes that summarize worker reluctance to re-enter the gambling industry: (a) respect for the nature of the work (or lack thereof) (b) lack of employment stability, (c) competing employment opportunities outside of the industry, (d) health concerns, and (e) a sense that the work can be enjoyable given the gaming aspect to it and because of general employee camaraderie. Simply put, the work can be fun. Suppose further that this last theme was an unexpected one, demonstrating why researchers engage in exploratory work.

Member checks, triangulation and disconfirming evidence in the Exploratory Sequential Design

Credibility techniques could be deployed to check the trustworthiness of these themes by asking interviewees to comment on them (see Member Checks in Table 1). In addition, triangulation of interviews with employment records might or might not support these themes, and researchers might also purposefully search for counterevidence to test their veracity (Disconfirming Evidence). These techniques can interact. Suppose a researcher initially had the two themes lack of employment stability and competing employment opportunities outside of the industry in one larger theme (employee turnover). But during member checking the researcher detected some nuances that the larger theme did not capture with respect to turnover. This caused the researcher to re-check some employment records and it was confirmed that employees were leaving on their own accord and for two general reasons: (a) they reported they took another job in a different industry or (b) they left even if they did not have another job lined up for reasons like cancelled shifts. Adopting a disconfirming evidence lens, the researcher split the larger theme into lack of employment stability and competing employment opportunities outside of the industry to follow Saldaña’s (2016) advice to generate themes that are mutually exclusive (i.e., determining data fit in one category but not another) and jointly exhaustive (i.e., all data are accounted for during thematic analyses). If a researcher generated these themes following sample size guidance, then the Exploratory Sequential Design will have had a good beginning because the themes would have emerged from open-ended inquiry (e.g., the effort led the unexpected finding that some workers in the gambling industry find the work to be fun), lending credence to the idea the researcher did not miss a theme because of a lack of familiarity with context. Furthermore, the themes should be nuanced enough to support later design stages.

Despite the strengths described in this first step of the Exploratory Sequential Design, there are limitations to qualitative work. Without considerable resources, it is rare to see qualitative studies be conducted with large sample sizes because it is expensive to interview large numbers of people. Furthermore, industry leaders might want more evidence, and different forms of evidence, to inform their practices. Researchers might be interested in understanding the degree to which the five qualitative themes hold over a broader group of respondents because they wish to update HRD theory to account for pandemic related events in industry rehiring. Suppose the researcher wants to develop statistically generalizable information that informs policies and strategies for employers in the sector. Generating statistically generalizable findings would be best accomplished by gathering a random sample from some population, or more precisely a list of former employees who have not yet returned to their formal roles (i.e., a frame from which to draw a random sample) to complete a survey. An important mixing stage here is for the researcher to develop the survey with items grounded in the prior qualitive work. This would mean creating a series of survey questions based on the five themes identified in the exploratory qualitative phase. At this design stage, the researcher should have considerable confidence in item coverage but also insights on how to best word items so that the survey is well targeted to the population of interest, former employees. That is, the survey would have strong content validity. Furthermore, the prior qualitative steps would likely have yielded important contextual information that could inform how to locate sampled members of the population and the survey incentives needed to generate a strong response rate.

It might even be the case that the qualitative information might inform sampling procedures. For example, rather than simple random sampling it might turn out to be advisable to use stratified random sampling if it turns out that respondent characteristics like age, level of experience in the gambling industry, and degree of technical skill needed for a job (e.g., capacity to deal cards in high stakes poker games, serving as a pit boss, security technology, accounting) compel the surveyor to oversample some types of former employees. At the survey analysis stage, this contextual information might inform statistical weighting schemes (e.g., Valliant & Dever, 2018). If so, one can draw a direct line from prior qualitative inquiry in the design to statistical decisions like sample weighting and generating confidence intervals when developing statistically generalizable estimates that might inform industry-wide rehiring practices.

In summary, this mixed methods design sets a researcher up for developing a well-targeted survey instrument and sampling approach, demonstrating an interweaving of qualitative and quantitative methods that should minimize weaknesses of component design parts and generate powerful synergies in inquiry. This primer⁴ hopefully demonstrates a basic rationale for using mixed methods.

Three Mixed Methods Advancements

In this section, I offer an overview of some relatively recent works in (a) paradigmatic thinking, (b) integration, and (c) validity in the hopes that these topics will further support HRD mixed methods research.

Dialectical Pluralism

Johnson (2012, 2017) and Greene (2007) have offered some writings that I found to be useful with respect to addressing paradigm debates (see also Morgan, 2022 for a more recent review of paradigms). Researchers use paradigms to guide their view of what questions are important and the methods used to address their questions. There are of course arguments in most fields over what phenomena are important to study, and how to go about studying them. This gets into the topic of the so-called paradigm wars although we need not be so dramatic (Onwuegbuzie, 2012), given these are debates, and even civil discussions. One outgrowth of broader debates is there are several paradigms available to the mixed methods researcher, and one can pick one (or perhaps use one that best articulates the researcher’s pre-existing tendences) to help articulate a worldview. For example, did you see value in random sampling in the primer example? If so, perhaps you are a postpositivist (i.e., you adhere to postpositivism).

Paradigmatic self-awareness is important in the sense that not knowing one’s paradigm reflects a lack of self-knowledge and opens researchers to criticism that their work is unpolished and unsophisticated. This can be worrisome for those of us who wish to advance mixed methods work in arenas like identifying new ways to integrate quantitative and qualitative work, question the degree to which quantitative and qualitative research differ, and promote credibility and validity techniques. To further add to reason to worry, some colleagues describe quantitative and qualitative methods as being incompatible (Tashakkori & Teddlie, 2010 provide a useful discussion of several issues). There must be an answer to this charge. If quantitative inquiry and qualitative inquiry are incompatible, how can one legitimately be a mixed methods researcher? Much less, how can one be a mixed methods researcher who seeks new approaches for integrating? Just writing these sentences makes me want to look over my shoulder in fear of the paradigm police. One way to ward off arrest, however, is to declare one’s paradigm and standard options for the mixed methods researcher are the pragmatic, critical realism, or transformative paradigms. It is however the case that other paradigms such as postpositivism and constructivism are compatible with mixed methods (Morgan, 2022).

Some work in this arena has been updated over the last decade. For those of us who are uncomfortable labels and groupings that might seem overly simplistic (e.g., Hitchcock is a classical pragmatist is phrase I never embraced), the Dialectical Pluralism approach⁵ to paradigmatic thinking might be refreshing because it emphasizes dialogue to understand differences (Johnson, 2017). That is, dialectical refers to dialogue and pluralism is of course a reference to many, or at least more than one. Dialectical Pluralism is hence described as a metaparadigm. A key takeaway point from this metaparadigm is it focuses less on paradigm opposition and researcher exclusion and more on working through disparate beliefs and taking advantage of different viewpoints. It also helps support a mixed methods way of thinking (Greene, 2007, p. 20). For example, recall the Exploratory Sequential Design. I have written about it before and used it in some of my own work (e.g., Hitchcock et al., 2005), but I recently found, through dialogue, a greater appreciation for connections between qualitative inquiry and selecting sampling strata, and how strata and survey nonresponse can help one interpret weights and confidence intervals. The methodological issues underneath these connections are complex; I personally lack the background needed to understand it all and so I consult sampling statisticians. This creates dialogue between different types of researchers who might not normally interact.

Fortunately, Dialectical Pluralism provides a basis for advancing thought and creativity when integrating because its use requires asking questions, listening, and creating mutual understanding and solutions. Whereas I learn about weighting and strata, colleagues learn about context, content, and theory for a project as we build solutions together. I am not pursuing survey research around rehiring in the gambling industry but if I were I would rely on Dialectical Pluralism to help me as I dialogue with sampling statisticians.

For another example of the usefulness of Dialectical Pluralism, consider randomized controlled trials (RCT) and quasi-experiments. There are mixed methods ways to understand one of the main outcome metrics from these designs: standardized effect sizes (Hitchcock & Newman, 2013; Newman & Hitchcock, 2011; Cohen’s (1988)). Assume that an RCT mounted to study a bonus incentive program found a Hedges’ g effect of 0.33 on employees’ self-reported job satisfaction. That is, employees who experienced the program reported greater job satisfaction relative to comparison-group employees who did not participate in the bonus program. Interpretating this quantitative estimate using only quantitative research would limit the researcher to: (a) understanding the magnitude of the effect (it is a about a third of a standard deviation), (b) comparing its size to other studies (Cohen, 1988) thresholds place this effect in the small-to-medium range), (c) converting the standardized effect to its unstandardized variant (in this case, the size of the average difference in job satisfaction scores between the two study conditions) to facilitate interpretation, and (d) whether the effect size is statistically significant. Expanding the use of modeling techniques would allow the researcher to investigate if the effect is moderated by context and sample member characteristics and if there are any treatment mediators. One might also add cost estimates to understand the cost-benefit of the program (e.g., does the bonus program improve worker productivity such that it saves money in the long run?).

Now consider these ideas of mediating and moderating variables, and costs, but assume a researcher can mix in qualitative techniques to further understand bonus program impacts. Through exploratory interviews researchers might ask research participants about elements of the bonus program (or its implementation) that mediate the effect (e.g., cash bonuses by themselves do not yield the job satisfaction effect; manager praise and social recognition among peers are also needed). This would mean qualitative inquiry was used to help identify a mediating analysis, yielding integration of quantitative and qualitative methods in experimental analyses. Similarly, interviews might clue researchers into identifying moderating variables they might not otherwise have considered. In terms of cost analyses, a bonus program has the obvious cost of the financial bonus itself, but interviews with managers and/or document analyses and observations might reveal other costs such as employee time spent in recognition ceremonies and bonus program administration, and materials cost to advertise the program and recognize employees (e.g., cards, posters). Getting back to Dialectical Pluralism, I personally find value in this metaparadigm because it can yield mixing opportunities borne out of dialogue between researchers that in turn promote synergistic inquiry.

This can be especially true when dealing with very different research lenses. For example, whereas an experimental design lens might compel a focus on whether a bonus program led to improved job satisfaction and for whom, a critical research stance might compel focus on whether the bonus program reifies unequal power structures, making some employees to perceive the program to be demeaning. I mentioned intersectionality at the outset of this article. Very briefly, it is the idea that people work within dynamic employment contexts that shape their identities. I engage in researcher reflexivity (see Table 1), and I put effort into remembering some facts about myself when conducting social science research. For example, I am a straight, cisgender male who is highly educated and afforded enough power to not simply operate within Western social science culture but also to influence it. I am also compensated well enough for my work that a bonus program is not something that holds serious financial repercussions for my personal life. Hence, it occurs to me that while I value a critical lens, I purposefully seek help when trying on these glasses to help me see more than what I might otherwise without the benefit of Dialectical Pluralism.

This can create a kind of kaleidoscope (see Perumal et al., 2022) that can help me, and other researchers to see different truths, and to consider different facets of truth. One might then conclude and reconcile seemingly contradictory ideas. Perhaps the aforementioned 0.33 Hedges’ g is a defensible finding, leading managers to conclude the bonus program is worthy if wider adoption. At the same time, it might also be true that some employees see the bonus program as a means to reward only straight, cisgender males at the expense of other staff. This would be a powerful eye-opener for the program administrators, and this concern could be ameliorated once they become aware of it. With these examples in mind, I hope HRD researchers will find some of this more recent work on the Dialectical Pluralism metaparadigm to be informative, and even freeing, as they think about and pursue their inquiry.

Advances in Integration

The integration (or mixing) of quantitative and qualitive work is at its essence a sine qua non of mixed methods research; without integration at least at a conceptual level then one is pursuing a multiple methods design (and not a mixed methods design). As it turns out, there are many strategies for integration on the basis that there are many mixed methods design variants, and the mixed methods field continues to advance in this regard. In fact, Fetters and Freshwater (2015) offered an editorial in the Journal of Mixed Methods Research referencing what they called the “integration challenge” (p. 115), arguing that there can be important advances in inquiry by advancing approached to qualitative and quantitative integration. They state:

We describe the integration challenge qualitatively as the imperative to produce a whole through integration that is greater than the sum of the individual qualitative and quantitative parts….Now, with more experience under the field’s belt, we hope to get all mixed methods researchers to consider the mixed methods challenge. Quantitatively, we express this as 1 + 1 = 3. That is, qualitative + quantitative = more than the individual components. We believe this framework should push all mixed methodologists to think about how integration has and can push research methods to higher levels that would not have been achieved by simply adding together the results of separate qualitative and quantitative studies conducted without full attention to integration. (Fetters & Freshwater, 2015, pp. 115–116)

Hitchcock & Onwuegbuzie (2022) attempted to rise to this challenge and define integration as “the optimal mixing, combining, blending, amalgamating, incorporating, joining, linking, merging, consolidating, or unifying of research approaches, methodologies, philosophies, methods, techniques, concepts, language, modes, disciplines, fields, and/or teams within a single study” (p.3). The use of the word optimal does not specify some pre-existing threshold for reaching integration; instead, one should think of an integration continuum. This is like the notion of validity; one should not think of validity as a dichotomous concept (i.e., this claim is valid or not) but instead consider the degree of evidence available to support argumentation around validity claims (e.g., Kane, 2013). Similarly, there are degrees of integration, and the usefulness of claims/assertions based on integration should be informed by the research questions and goals at hand.

Having established that there are degrees of integration, and the usefulness of integration should be informed by the research questions at hand, how might an advancement in integration be achieved, particularly in a study that might interest an HRD researcher? One approach where there have been some advancements is the so-called crossover analysis; this entails using analysis techniques from one research tradition (e.g., qualitative) to analyze data associated with another tradition (quantitative) and vice versa (Combs & Onwuegbuzie, 2010; Greene, 2008; Hitchcock & Onwuegbuzie, 2020; Onwuegbuzie & Combs, 2010; Onwuegbuzie et al., 2011; Onwuegbuzie & Teddlie, 2003). For example, ethnographic work is generally classified as a qualitative form of inquiry although it has been argued that it can be seen as mixed methods research (e.g., Hitchcock & Onwuegbuzie, 2022).

Applying a crossover ethnographic analysis to an HRD topic, suppose a researcher wanted to use an ethnographic framework to understand emerging cultural differences in industry-specific, post-pandemic business conduct among newly minted executives. Questions that might be addressed in a crossover ethnography could be: (a) What is the etiquette for face-to-face meetings (e.g., do we shake hands now)? (b) Are face-to-face meetings valued? (c) What message is conveyed to customers and business partners when executives travel? (d) What behaviors are expected for on-line meetings? (e) What does culturally appropriate messaging look like within a given customer bases? (f) Are we back to pre-pandemic behavior when we interact with partners and customers in a given culture? (g) When considering all these questions, what type of new training is now necessary for company employees and (h) how can we evaluate the effectiveness of this training?

In an ethnographic crossover mixed analysis, a researcher could begin to qualitize quantitative data and/or findings (Onwuegbuzie & Leech, 2019); that is, generate qualitative elaborations drawn from what was initially quantitative information. One might ask subgroups of survey respondents more details about why they responded to a survey and to elaborate on their meaning. For example, survey respondents might respond to an instrument querying about their business interaction behaviors (e.g., how people greet each other given COVID-19 concerns, expectations around wearing facemasks, and etiquette during dinners). If analyses determined that respondents within a particular subgroup yielded homogenous responses demonstrating some belief (e.g., executives who travel to meet me are subordinates) then follow-up interviews, observations, and document analyses would provide a way to qualitize this finding to learn more about it and perhaps gain further insights around how to train executives for future sales meetings.

Conversely, one could quantitize qualitative data (e.g., assigning numbers to themes) so that they can be included in statistical modeling (Sandelowski et al., 2009). Suppose thematic analyses of interview data revealed a series of themes that summarized what different members in a cultural setting shared about their views of business behavior. Themes could be assigned numbers (i.e., be quantitized) and then further examined via any number of statistical techniques capable of handling categorical (i.e., nominal, ordinal) data. Such modeling of initially qualitative data need not replace traditional qualitative analyses but could instead be used as a supplement to see if new findings might be gleaned from the crossover work. This form of integration of technique would yield a qualitative-dominant crossover analysis within an ethnography that potentially yield strong insights into business expectations and employee professional development that could be led by human resource leaders.

There are potentially dozens of new ways to conceptualize and pursue highly integrated work (see Hitchcock & Onwuegbuzie, 2022), and readers might be pleasantly surprised that some approaches are not complex if researchers remain cognizant that data is another word for information, and information has both quantitative and qualitative elements. Doing so can remind researchers there is value in ideas like: (a) remembering to consider how quantitative findings might be perceived by stakeholders (recall the prior section on Dialectical Pluralism) and/or (b) reviewing numerical information in qualitative work, such as how many interviewees contributed to some theme. For purposes of this section, two take home points are that (a) new crossover analyses, like ethnographic crossover analyses, have been conceptualized and (b) there have been advancements in ideas like quantitizing/qualitizing data in pursuit of new analyses.

A Mixed Methods Validity Framework

I now turn to a third broad and ongoing development in mixed methods research: how thinkers in the field are conceptualizing a form of validity, or legitimation. Nastasi et al. (2021) describe how different approaches to inquiry (e.g., experimental designs psychometrics, grounded theory) bring frameworks for assessing the credibility, validity, and/or trustworthiness of findings. In survey work for example there are sources of survey error (measurement, coverage, sampling, and nonresponse error; Dillman et al., 2014). The degree to which these sources of error are minimized informs the validity of survey findings. Similar frameworks can be found in experimental design (Shadish et al., 2002), specific forms of qualitative inquiry (Lincoln & Guba, 1985), and so on (Nastasi et al., 2021).

A variant of a mixed methods validity framework is legitimation, an idea presented by Onwuegbuzie and Johnson (2006) for purposes of offering evaluative criteria for mixed methods research at different stages of inquiry. Legitimation supports ongoing assessment of research procedures and quality and helps researchers to make inferences such as meta-inferences. A meta-inference is an inference wherein qualitative and quantitative findings are integrated into either two distinct sets of findings or an overall whole (O’Cathain, 2010; Schoonenboom, 2022; Tashakkori & Teddlie, 1998).

Onwuegbuzie and Johnson (2006) initially identified nine legitimation types; this list was modified, updated, and expanded by Johnson and Christensen (2017, 2020). These legitimation types are presented in Table 2, below. Before reviewing the table, it is important to keep in mind that not all legitimation types will apply to a given mixed methods research project. If for example one did not quantitize qualitative data or vice versa, then conversion legitimation would not be applicable. It is also important to keep in mind that the legitimation types are not meant to be mutually exclusive. For example, there can be overlap between inside-outside legitimation and sociopolitical/political legitimation. Finally, each type should not be treated as a checklist with some threshold wherein a researcher can claim to have legitimated mixed methods research. Rather, each legitimation type provides a concept that allows for reflection, self-critique of one’s work, and communication of study strengths and weaknesses to consider when describing the overall quality of findings.

Table 2.

Legitimation Types as Quality Indicators.

Type	Description
Commensurability approximation	The degree to which meta-inferences are based on quantitative and qualitative worldviews and informed by researchers’ purposefully seeking research synergy derived from applying two or more methods. Example: Researchers might consider how experimental evidence and qualitative evidence combine to allow for overall inferences about an employee bonus program
Conversion	The degree to which quantitative data can be qualitized and qualitative data can be quantitized to support meta-inferences. Example: Quantifying written records taken from employee evaluations to support statistical modeling, or interviewing staff to learn more about sales records initially presented in numerical form
Inside-outside	The degree to which study participants’ views are accounted for, relative to the views of a researcher/observer. Example: Attempting to understand the views of employees, managers, and executives, relative to the researcher’s own views and beliefs
Integration	The extent to which researchers appropriately integrate quantitative and qualitative data into coherent findings. Example: Assuring that all findings from a thematic analysis are accounted for in a subsequent survey
Multiple validities	For example, the degree to which different statistical assumptions underlying component quantitative research processes are considered, the degree to which trustworthiness can be established in qualitive research components, and the degree to which legitimation is supported. Example: Ensuring statistical assumptions are addressed during experimental analyses and ensuring qualitative quality indicators (e.g., member checks, triangulation, audit trails) are used to support interview findings
Paradigmatic/philosophical	The degree to which researchers’ worldviews/beliefs are accounted for during the inquiry process. Example: Researchers who adopt a critical theory lens identify inequitable distribution of employee bonuses
Sample integration	Considering how well sampling designs allow for making meta-inferences and otherwise support the purpose of the mixed methods research at hand. Example: Ensuring that study participants who participate in interviews about an employee program are like participants who join an experiment and then merging component findings into a coherent whole
Sequential	Considering if there might be an order effect that undermines a meta-inference or other mixed methods finding. Example: Always showing positive employee performance results [a quantitative phase] and then interviewing employees about the reasons for their performance; this might create an order effect
Sociopolitical/Political	Evidence of tension that can arise when some members of a team value quantitative evidence over qualitative evidence or vice versa; or, when consumers of research privilege different forms of evidence and inadvertently (or purposefully) suppress other forms of evidence. Example: Corporate executives might push adoption of new employee program based on quantitative efficacy evidence even if employees report reservations about practice change
Weakness minimization	The degree to which a given quantitative research approach compensates for weaknesses in a qualitative approach used in a mixed methods study, and vice versa. Example: a Survey study asking employees about sales techniques can be used to make population inferences based on a randomly drawn sample, but a survey might not effectively capture all detail; interviews can in contrast allow researchers to explore the matter in-depth (keeping in mind it is difficult to conduct interviews with large sample sizes)

Note. See the article: Onwuegbuzie, A. J., & Johnson, R. B. (2006). The validity issue in mixed research. Research in the Schools, 13(1), 48–63 for a fuller discussion of legitimation.

Applying these legitimation types provide one framework for assessing mixed methods research quality and validity. HRD researchers might consider these types of legitimation when designing a study, during the inquiry process, and when completing a project. Doing so would be like experimental design, where different considerations around threats to validity are considered: (a) during study conceptualization (e.g., sample size needs, how randomization will be carried out, ensuring the treatment contrast against a control condition will yield policy and/or business-relevant information), (b) during the study (e.g., preventing sample loss, ensuring the program being studied is implemented correctly) and (c) at the end of the study (e.g., when making causal inferences, report writing and dissemination). These legitimation types, or lenses, could be applied to, for example, the exploratory sequential survey project described earlier in the mixed methods primer (see again Figure 1).

I return to that fictitious example where a researcher sought to understand employee rehiring, turnover, and retention in the gambling industry during the COVID-19 pandemic recovery period using an exploratory sequential design. Commensurability approximation would be addressed in this design on the basis that Phase 1 qualitative inquiry should yield thematic information that can be used to inform survey item writing. That is, survey items can be grounded in qualitative themes, creating potential methodological synergy between the initial qualitative phase and subsequent quantitively-driven survey analyses, as established earlier in this article.

Conversion legitimation could be achieved in this design by, for example, writing an item like: what would it take to convince you to start dealing cards for “X” company again? Assume that the item is followed by a text box to solicit open-ended ideas (the respondent is expected to write a few sentences). Resulting text data could then be numerically coded into response categories, effectively quantitizing (converting) open-ended exploratory text data. Similarly, the project could identify subgroups of responses who yield compelling and/or unusual information and follow-up with tailored interview questions (e.g., can you tell us more about why you answered this item set in this way?), thereby qualitizing information. Inside-outside legitimation could be pursued should a researcher gather information from employees, prospective rehires, and hiring managers, while juxtaposing these viewpoints against the researcher’s own perspectives and views of HRD theory (e.g., a researcher determines that survey respondents answered questions about work enjoyment because of “X” reason [s]; however, would respondents concur with this interpretation?). Integration could be addressed at the analysis stage by comparing thematic results with quantitative survey findings, such as by comparing previously developed qualitative themes against survey factor analysis results to investigate how themes and factors compare; see Hitchcock et al., 2005 for an example).

Multiple validities could be addressed by, say, deploying credibility techniques in Table 1 to promote the trustworthiness of the exploratory qualitative portion of the design. Using standard psychometric analyses (e.g., reliability analyses, construct validity; see Kane, 2013), along with a review of sources of survey error, would assess the validity of the quantitative survey findings. Paradigmatic/philosophical legitimation might be addressed if a HRD researcher adopts a classical pragmatic position or perhaps works through the tenants of Dialectical Pluralism to articulate how qualitative and quantitative inquiry yield complementary, divergent, and synergistic findings. Sample integration legitimation might be pursued by considering and reporting clearly about the degree of similarity between study participants in the opening qualitative stage and survey respondents. Sequential legitimation could be considered by reflecting on whether the fact that qualitative inquiry preceded survey work yielded some bias in item writing or when interpreting overall findings. Sociopolitical legitimation can remind researchers to ponder if some evidence is being privileged at the expense of other evidence because a researcher or stakeholder places excessive value on some component finding. Finally, weakness minimization legitimization invites query around whether mixed methods was worthwhile, and if methodological synergy was indeed achieved. In sum, readers will hopefully agree that consideration of these legitimation types offers an overall useful tool for advancing mixed methods research quality and validity of inferences.

Discussion and Conclusion

The purpose for this presentation was to promote the use of mixed methods research in HRD scholarship or at the very least report to HRDR readers details around three advancements in mixed methods work. This Instructor’s Corner article therefore offered a primer on the merits of general mixed methods research, followed by: (a) updates in paradigmatic thinking that support the use of mixed methods, (b) new approaches to integration, and (c) legitimation approaches that offer a validity framework for understanding mixed methods design quality. As a caveat to keep in mind, my voice is but one in a chorus of mixed methods scholars and I remain aware that other writers who might have offered this report to HRDR might have taken a different direction. However, I wished to offer a discussion of contemporary mixed methods ideas and I think this it is evident this was achieved given the dates of supportive citations used throughout this article. I also intended to discuss mixed methods topics that are important and far reaching; I hope readers would agree that a review of research paradigms, methodological approaches, and validity (legitimation) are indeed important matters. At the very least, it is my hope that this contribution to HRDRs Instructor’s Corner provides a list of mixed methods references that might be consulted by colleagues who wish to pursue empirical mixed methods inquiry.

From a methodological point of view, HRD researchers can deploy mixed methods to address goals that can be hard to address within a single study, such as exploring new phenomena while developing findings that may be generalized to new settings and contexts (as is the case for the exploratory sequential survey design). Or they might avail themselves to case study methods to explore the circumstances of outliers identified after some initial quantitative analysis (e.g., learning more about employees who might be expected to leave an organization but stayed on and prospered). For another example, HRD researchers might rely on thinking around Dialectical Pluralism to justify steps like quantitizing thematic codes drawn from qualitative work to use statistical modeling to help them test different propositions. And HRD researchers might use any number of legitimation types (or lenses) to help them test their findings and further think through their implications.

In this article I offered conceptual examples meant to depict that we are facing complex research situations in the social sciences, but these seem almost mundane compared to what HRD scholars need to be working on. Moving beyond that which is complex, HRD researchers will be dealing with wicked problems. These are research problems that “involve multiple interacting systems, are replete with social and institutional uncertainties, and for which only imperfect knowledge and about their nature and solutions exist” (Mertens, 2015, p. 3). Public and private sectors are interconnected with all aspects of society because all societies need to consider its various economies, and HRD provides an interface between effective business practices, ethics, and citizenry as we all grapple with climate change, employee marginalization, degradation in political discourse, and resource competition that at times culminate in wars. HRD research is therefore not the province of armchair scholarship; it will continue to be difficult and messy, but also quite meaningful when it gives at least partial answers on an increasingly populated planet. I hope mixed methods research is seen as a useful tool as social scientists in general, HRD researchers in particular, and stakeholders work together to think through wicked problems and generate solutions.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

John H. Hitchcock

Notes

Author Biography

John H. Hitchcock is a Principal Research Scientist at Westat, where he conducts grant-funded and contract-based research for government agencies and philanthropies. He writes about mixed methods research and other types of research designs.

References

American Psychological Association (2017). Multicultural Guidelines: An ecological approach to context, identity, and intersectionality. http://www.apa.org/about/policy/multicultural-guidelines.pdf

Bazeley

(2018). Integrating analyses in mixed methods research. Sage.

Brantlinger

Jimenez

Klingner

(2005). Qualitative studies in special education. Exceptional Children, 71(2), 195–207. https://doi.org/10.1177/001440290507100205

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum.

Combs

J. P.

Onwuegbuzie

A. J.

(2010). Describing and illustrating data analysis in mixed research. International. Journal of Education, 2, 1–23. http://www.macrothink.org/journal/index.php/ije/article/viewFile/526/392

Congressional Research Service (2022). Job openings and labor turnover before and during the COVID-19 pandemic. https://crsreports.congress.gov/product/pdf/R/R47047/2

Creamer

E. G.

(2018). An introduction to fully integrated mixed methods research. Sage.

Creswell

J. W.

Plano Clark

V. L.

(2018). Designing and conducting mixed methods esearch. Sage.

Dillman

D. A.

Smyth

J. D.

Christian

L. M.

(2014). Internet, phone, mail and mixed-mode surveys: The tailored design method (4th ed.). John Wiley & Sons.

10.

Fetters

M. D.

Freshwater

(2015). The 1 + 1 = 3 integration challenge. Journal of Mixed Methods Research, 9, 115–117. https://doi.org/10.1177/1558689815581222

11.

Fetters

M. D.

Molina-Azorin

J. F.

(2017). The Journal of Mixed Methods Research starts a new decade: The mixed methods research integration trilogy and its dimensions. Journal of Mixed Methods Research, 11(3), 291–307. https://doi.org/10.1177/1558689817714066

12.

Greene

J. C.

(2007). Mixed methods in social inquiry. San Francisco, CA: Jossey-Bass.

13.

Greene

J. C.

(2008). Is mixed methods social inquiry a distinctive methodology? Journal of Mixed Methods Research, 2(1), 7–22. doi: https://doi.org/10.1177/1558689807309969

14.

Greene

J. C.

Caracelli

V. J.

Graham

W. F.

(1989). Toward a conceptual framework for Mixed method evaluation designs. Educational Evaluation and Policy Analysis, 11(3), 255–274. https://doi.org/10.3102/01623737011003255

15.

Guest

Bunce

Johnson

(2006). How many interviews are enough?: An experiment with data saturation and variability. Field Methods, 18(1), 59–82. https://doi.org/10.1177/1525822X05279903

16.

Hesse-Biber

Johnson

R.B.

(2015). The Oxford handbook of multimethod and mixed methods research inquiry. Oxford University Press.

17.

Hitchcock

J. H.

Nastasi

B. K.

Dai

Newman

Jayasena

Bernstein-Moore

Sarkar

Varjas

(2005). Illustrating a mixed-method approach for validating culturally specific constructs. Journal of School Psychology, 43, 259–278. https://doi:10.1016/j.jsp.2005.04.007

18.

Hitchcock

J. H.

Newman

(2013). Applying an interactive quantitative-qualitative framework: How identifying common intent can enhance inquiry. Human Resource Development Review, 12(1), 36–52. https://doi.org/10.1177/1534484312462127

19.

Hitchcock

J. H.

Onwuegbuzie

A. J.

(2022). The Routledge Handbook for advancing integration in mixed methods research. Routledge.

20.

Johnson

R. B.

(2012). Dialectical pluralism and mixed research. American Behavioral Scientist, 56, 751–754. https://doi.org/10.1177/0002764212442494

21.

Johnson

R. B.

(2017). Dialectical pluralism: A meta-paradigm whose time has come. Journal of Mixed Methods Research, 11, 156–173. https://doi.org/10.1177/1558689815607692

22.

Johnson

R. B.

Christensen

L. B.

(2017). Educational research: Quantitative, qualitative, and mixed approaches (6th ed.). Sage.

23.

Johnson

R. B.

Christensen

L. B.

(2020). Educational research: Quantitative, qualitative, and mixed approaches (7th ed.). Sage.

24.

Johnson

R. B.

Onwuegbuzie

A. J.

(2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14–26. https://doi.org/10.3102/0013189X033007014

25.

Johnson

R. B.

Onwuegbuzie

A. J.

Turner

L. A.

(2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1(2), 112–133. https://doi.org/10.1177/1558689806298224

26.

Kane

M.T.

(2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. https://doi.org/10.1111/jedm.12000

27.

Klingner

J. K.

Boardman

A. G.

(2011). Addressing the “Research Gap” in special education through mixed methods. Learning Disability Quarterly, 34(3), 208–218. https://doi.org/10.1177/0731948711417559

28.

Lincoln

Y. S.

Guba

E. G.

(1985). Naturalistic inquiry. Sage.

29.

Maxwell

J. A.

(2004). Causal explanation, qualitative research, and scientific inquiry in education (pp. 3–11). https://dx-doi-org.web.bisu.edu.cn/10.3102/0013189x033002003

30.

Mertens

D. M.

(2007). Transformative paradigm: Mixed methods and social justice. Journal of Mixed Methods Research, 1(3), 212–225. https://doi.org/10.1177/1558689807302811

31.

Mertens

D. M.

(2015). Mixed methods and wicked problems. Journal of Mixed Methods Research, 9, 1–6. https://doi.org/10.1177/1558689814562944.i

32.

Morgan

D.L.

(2022). Paradigms in mixed methods research. In Hitchcock

J. H.

Onwuegbuzie

A. J.

(Eds), The Routledge Handbook for advancing integration in mixed methods research (pp. 97–112). Routledge.

33.

Nastasi

B. K.

Hitchcock

J. H.

Brown

L. M.

(2010). An inclusive framework for conceptualizing mixed methods design typologies: Moving toward fully integrated synergistic research models. In Tashakkori

Teddlie

(Eds), Sage handbook of mixed methods in social and behavioral research (2nd ed., pp. 305–338). Sage.

34.

Nastasi

B.K.

Hitchcock

J.H

Gutierrez

Oshrin

(2021). Promoting quality of mixed methods consultation research: A synergistic participatory framework. Advanced on-line publication. https://doi.org/10.1080/10474412.2021.1887743

35.

Newman

Hitchcock

J. H.

(2011). Underlying agreements between quantitative and qualitative research: The short and tall of It all. Human Resource Development Review, 10(4), 381–398. https://doi.org/10.1177/1534484311413867

36.

Newman

Hitchcock

J. H.

Newman

(2014). The use of meta-analysis and nomological networks to develop HRD Theory. Advances in Developing Human Resources, 17(1), 117–134. https://doi:10.1177/1523422314559810

37.

O’Cathain

(2010). Assessing the quality of mixed methods research: Toward a comprehensive framework. In Tashakkori

Teddlie

(Eds), Handbook of mixed methods in social and behavioral research (2nd ed., pp. 531–555). Sage.

38.

Onwuegbuzie

A. J.

(2012). Introduction: Putting the mixed back into quantitative and qualitative research in educational research and beyond: Moving towards the radical middle. International Journal of Multiple Research Approaches, 6, 192–219. https://doi.org/10.5172/mra.2012.6.3.192

39.

Onwuegbuzie

A. J.

Bustamante

R. M.

Nelson

J. A.

(2010). Mixed research as a tool for developing quantitative instruments. Journal of Mixed Methods Research, 4(1), 56–78. https://doi.org/10.1177/1558689809355805

40.

Onwuegbuzie

A. J.

Combs

J. P.

(2010). Emergent data analysis techniques in mixed methods research: A synthesis. In Tashakkori

Teddlie

(Eds), Sage Handbook of mixed methods in social and behavioral research (2nd ed., pp. 397-430). Sage.

41.

Onwuegbuzie

A. J.

Corrigan

J. A.

(2014). Improving the quality of mixed research reports in the field of human resource development and beyond: A call for rigor as an ethical practice. Human Resource Development Quarterly, 25, 273–299. https://doi.org/10.1002/hrdq.21197

42.

Onwuegbuzie

A. J.

Hitchcock

J. H.

(2019). Toward a fully integrated approach to mixed methods research via the 1 + 1 = 1 integration approach: Mixed Research 2.0. International Journal of Multiple Research Approaches, 11(1), 7–28. https://doi.org/10.29034/ijmra.v11n1editorial1

43.

Onwuegbuzie

A. J.

Johnson

R. B.

(2006). The validity issue in mixed research. Research in the Schools, 13(1), 48–63.

44.

Onwuegbuzie

A.J.

Johnson

R.B.

(2021). The Routledge Reviewer’s Guide to mixed methods analysis. Routledge.

45.

Onwuegbuzie

A. J.

Leech

N. L.

(2019). On qualitizing. International Journal of Multiple Research Approaches, 11(2), 98–131. http://doi:10.29034/ijmra.v11n2editorial2

46.

Onwuegbuzie

A. J.

Leech

N. L.

Collins

K. M. T.

(2011). Toward a new era for conducting mixed analyses: The role of quantitative dominant and qualitative dominant crossover mixed analyses. In Williams

Vogt

W. P.

(Eds), The Sage handbook of innovation in social research methods (pp. 353–384). Sage.

47.

Onwuegbuzie

A. J.

Teddlie

(2003). A framework for analyzing data in mixed methods research. In Tashakkori

Teddlie

(Eds), Handbook of mixed methods in social and behavioral research (pp. 351–383). Sage.

48.

Patton

M.Q.

(2015). Qualitative research and evaluation methods (3rd ed.). Sage.

49.

Poth

(2022). Adaptive practices for a complexity-sensitive approach to mixed integration. In Hitchcock

J. H.

Onwuegbuzie

A. J.

(Eds), The Routledge Handbook for advancing integration in mixed methods research (pp. 174–195). Routledge.

50.

Ridenour

C. S.

Newman

(2008). Mixed methods research: Exploring the interactive continuum. Carbondale, IL: University Press.

51.

Saldaña

(2016). The coding manual for qualitative researchers (3rd ed.). Sage.

52.

Sandelowski

Voils

C. I.

Knafl

(2009). On quantitizing. Journal of Mixed Methods Research, 3, 208–222. https://doi.org/10.1177/1558689809334210

53.

Schoonenboom

(2022). Developing the meta-inference in mixed methods research through successive integration claims. In Hitchcock

J. H.

Onwuegbuzie

A. J.

(Eds), The Routledge Handbook for advancing integration in mixed methods research (pp. 55–70). Routledge.

54.

Schwandt

T. A.

(2000). Three epistemological stances for qualitative inquiry: Interpretivism, hermeneutics, and social constructionism. In Denzin

N. K.

Lincoln

Y. S.

(Eds), Handbook of qualitative research (2nd ed., pp. 189–215). Sage.

55.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi- experimental designs for generalized causal inference. Houghton Mifflin.

56.

Tashakkori

Teddlie

(1998). Mixed methodology: Combining qualitative and quantitative approaches (applied social research methods series. Sage.

57.

Tashakkori

Teddlie

(2010). Handbook of mixed methods in social & behavioral research. Sage.

58.

Teddlie

Tashakkori

(2009). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. Sage.

59.

Valliant

Dever

J.A.

(2018). Survey weights: A step-by-step guide to calculation. Stata Press. Available: https://www.stata-press.com/books/sw-preview.pdf