Abstract
A defining characteristic of mixed methods research, integration receives considerable attention in the field, yet evidence suggests that the process of integration may be challenging in practice. Crossover analyses, in which methods typically used with one data type are applied to a different data type, can deepen integration. Using an example from the field of education, this article demonstrates the iterative application of multiple correspondence analysis as a crossover analysis through a four-stage integration process. Contributions to the field of mixed methods include (a) the utility and potential of multiple correspondence analysis to surface new dimensions of integrated findings, (b) considerations for rigor in the application of crossover methods, and (c) considerations for making sense of diffractive or dynamic findings.
A fundamental theory of change within the field of mixed methods research is that the combination of quantitative and qualitative approaches yields a more complete understanding of a phenomenon than either approach alone (Fetters & Freshwater, 2015). Operating within this assumption, mixed methods approaches have gained increasing attention in fields that seek to address complex social challenges, including the field of education (Louie, 2016). However, to fully capitalize on the potential of mixed methods, researchers must move beyond simply collecting and analyzing quantitative and qualitative data to integrating the two approaches (Bazeley, 2018; Fetters et al., 2013; Fetters & Freshwater, 2015).
When planning a mixed methods investigation, researchers should consider how integration occurs across multiple levels of the study, including the study design, methods, and interpretation (Fetters et al., 2013; Moseholm & Fetters, 2017). Decisions regarding integration at one level can have implications for integration at other levels. For example, in sequential designs, where a study proceeds in phases (e.g., quantitative phase followed by a qualitative phase), integration often occurs when the results of the first phase inform the development of the second phase (e.g., responses from a survey inform interview questions). However, integration within convergent designs, when data are collected concurrently, may present challenges, as integration typically occurs after separate analyses of the quantitative and qualitative data are complete (Moseholm & Fetters, 2017). Although integration receives considerable attention in the field of mixed methods, evidence suggests that the process of integration may be challenging in practice and has implications for rigor (Bryman, 2007; Harrison et al., 2020). Among investigations that use both quantitative and qualitative approaches, analyses are commonly conducted separately, without explicitly investigating relationships across findings (Bryman, 2006; O’Cathain et al., 2007). Such findings suggest a need for examples of how integration approaches can be applied at the analysis and interpretation levels and how conceptual frameworks can be used to facilitate the integration process.
Multiple Correspondence Analysis as an Approach to Support Integration
Fetters et al. (2013) identified three approaches to support integration at the interpretation level: narrative approaches, joint displays, and data transformation. Both narrative approaches and joint displays allow for the integration of quantitative and qualitative findings that resulted from the application of methods typically associated with each strand. Data transformation, however, occurs when qualitative findings are assigned numerical values or quantitative data are assigned qualitative labels (Bazeley, 2018; Onwuegbuzie & Hitchcock, 2015). Although data transformation can be its own end to integration, researchers can also make use of transformed data sets and apply crossover analyses. Crossover analyses require the use of transformed data, as data analyses typically associated with one strand of data are applied to the other data strand (Onwuegbuzie & Hitchcock, 2015). Although many analytical approaches can be applied as a crossover analysis (Bazeley, 2018), this article focuses specifically on the quantitative approach of correspondence analysis as applied to transformed qualitative data.
Correspondence analysis can be applied as a qualitative-dominant crossover analysis in which relationships among transformed qualitative data can be mapped into dimensional space using a quantitative approach (Bazeley, 2018; Onwuegbuzie & Hitchcock, 2015). Correspondence analysis has been defined as “a multivariate graphical technique designed to explore the relationships among categorical variables” (Sourial et al., 2010, p. 638). In other words, correspondence analysis allows researchers to visualize relationships between categorical variables by identifying the variables’ relationship with underlying dimensions (Greenacre, 2016). For example, researchers may transform participants’ description of an experience into categories of positive, negative, and neutral; correspondence analysis can be applied to the transformed data to identify patterns in participants’ experiences based on their demographic characteristics (e.g., age bracket, education level). Correspondence analysis is conceptually analogous to the more-familiar approach of factor analysis (i.e., correspondence analysis is to categorical data as factor analysis is to continuous data; Greenacre, 2016). Multiple correspondence analysis (MCA) is an extension of correspondence analysis that allows for the comparison of three or more categorical variables (Abdi & Valentin, 2007). Both correspondence analysis and MCA may be particularly useful as a crossover analysis for qualitative findings, because the approach does not require large data sets nor mutually exclusive observations (Bazeley, 2018).
We suggest that there is untapped potential for using MCA to support integration in two ways. First, there is potential for mixed methods researchers to expand on their application of correspondence analysis in order to support the integration of both qualitative and quantitative data sources. At the current moment, correspondence analysis has typically been used in two ways in the mixed methods literature. One current application involves the use of correspondence analysis as a quantitative approach during the quantitative strand of a sequential mixed methods study that does not make use of transformed data (e.g., Maltseva, 2016; Silva et al., 2009; see Figure 1). The second application within the mixed methods literature involves the use of correspondence analysis as a crossover analysis with transformed qualitative data (e.g., Onwuegbuzie et al., 2014). In such examples, correspondence analysis was used to analyze patterns in themes emerging from qualitative analyses only. However, assuming that data are collected for the same sample of participants in both qualitative and quantitative strands, participants’ quantitative responses could be combined with transformed qualitative data into a merged data set for analysis.
Overlap and intersection of correspondence analysis with mixed methods approaches.
Second, correspondence analysis and MCA have the potential to expand integration by supporting a crossover-tracks approach in a convergent design. A typical conceptualization of a convergent design is that qualitative and quantitative strands are conducted independently, and findings are combined after separate analyses are complete (Creswell & Plano Clark, 2018), an approach termed a parallel-tracks analysis (Datta, 2001; Teddlie & Tashakkori, 2009). In contrast, a crossover-tracks approach intertwines qualitative and quantitative strands, with each strand interacting with and informing the other strand throughout the study. For example, Hatta et al. (2020) conducted a convergent mixed methods investigation of clinical dialogues about cancer treatment in which researchers switched between inductive and deductive lenses across multiple phases of analysis and interpretation. The use of correspondence analysis offers similar opportunities for iterative integration and movement across qualitative and quantitative frames; as an exploratory method, correspondence analysis can be appropriately applied to conduct multiple analyses (Bazeley, 2018). Thus, the use of MCA across several stages of analysis could contribute to understanding integration as a process rather than a single moment occurring when findings are combined.
Conceptual Frameworks for Integration
To support researchers in articulating their process for merging data, Moseholm and Fetters (2017) developed a typology of integration approaches within the context of convergent mixed methods designs. The authors identified three dimensions of integration during a mixed methods analysis: relational, methodological, and directional. The relational dimension of integration refers to the extent to which data are “in conversation” with each other. The methodological dimension of integration refers to understanding which strand “drives” the analysis. Studies may be qualitatively driven, quantitatively driven, or equivalently driven. These analytic decisions are related to researchers’ epistemological frames (Johnson et al., 2007). Finally, the directional dimension refers to the movement of the integration process between the two strands of data (Moseholm & Fetters, 2017). In a unidirectional analysis, the complete analysis of one strand of data informs the integration of the other data strand. A bidirectional analysis, on the other hand, involves movement between the two strands throughout the integration process. The integration of the two strands may occur simultaneously, or may iteratively move from one strand, to the other strand, and back to the original strand.
Moseholm and Fetters (2017) characterized researchers’ integration decisions within the methodological and directional dimensions as falling into five conceptual frameworks 1 of integration for studies with a convergent design: explanatory unidirectional, exploratory unidirectional, simultaneous bidirectional, explanatory bidirectional, and exploratory bidirectional. In an explanatory unidirectional framework, the quantitative strand frames the integration of qualitative data to help clarify quantitative results. An exploratory unidirectional framework has the opposite emphasis; findings emerging through qualitatively framed analyses are deepened by quantitative results, to enhance understanding of the phenomenon.
Bidirectional approaches can occur simultaneously or iteratively. A simultaneous bidirectional framework is characterized by an interaction between the two strands as qualitative and quantitative strands are merged. However, bidirectional analyses can also be iterative. In an explanatory bidirectional framework, an initial quantitatively framed analysis is followed by a qualitatively framed analysis, and then the analyses are merged for interpretation. Similarly, an exploratory bidirectional framework involves movement from a qualitatively framed analysis to a quantitatively framed analysis, followed by a final interpretation. In their review of convergent mixed methods integration approaches, Moseholm and Fetters (2017) noted that examples of studies using explanatory or exploratory bidirectional frameworks were difficult to locate, indicating a need for additional illustrations of these approaches.
Methodological Objectives
Using an example from the field of education, the objectives of this article are to (a) demonstrate the application of MCA as a crossover analysis in order to surface new dimensions of integrated findings and (b) illustrate data integration at the analysis and interpretation levels, using an exploratory bidirectional framework through the iterative application of MCA to address a gap in the research literature (Moseholm & Fetters, 2017). We begin by describing the context and purpose for the empirical study, followed by an explanation of the methods used for participant sampling, data collection, and analyses procedures. We then detail our integration process, which progressed through four stages: (a) implement a convergent mixed methods design, (b) advance understanding through initial crossover analysis, (c) return to qualitative analysis, and (d) advance understanding through revised crossover analysis. These stages align with the characteristics of rigorous mixed methods research (Harrison et al., 2020). Finally, we discuss the integration process within the context of the research and theoretical literature, noting implications for the field of mixed methods research, challenges and opportunities of MCA, study limitations, and considerations for interpretation.
Empirical Study: Purpose and Context
A study investigating factors affecting the college transition of young adults with disabilities provides context for illustrating the methodological objectives. The purpose of the empirical study was to develop a more comprehensive understanding of the high school factors facilitating a smooth transition to college among students with disabilities (Kutscher & Tuckwiller, 2020). Youth with disabilities are a marginalized group who are at risk for experiencing a challenging transition to adult life. In the United States, young adults with disabilities are less likely to attend college than their peers without disabilities (Newman et al., 2011) and adults with disabilities are more likely to be unemployed (U.S. Department of Labor, 2020). However, completion of a college program can provide individuals with a pathway to employment and financial independence (Carnevale et al., 2016). Federal policy tasks schools with preparing students with disabilities for postsecondary education, in addition to employment and independence (Individuals with Disabilities Education Improvement Act, 2004). In alignment with this focus, researchers have devoted significant effort to identifying high school–level predictors that are associated with postschool success for this group of students (Mazzotti et al, 2015; Test et al., 2009).
One predictor that has received considerable attention is that of self-determination. Self-determination refers to a person’s ability to make decisions and take action in order to pursue personally meaningful goals and achieve a high quality of life (Wehmeyer, 2005). Based on correlational evidence, self-determination has been identified as a “promising” predictor of college participation (Test et al., 2009). Additionally, themes emerging from qualitative studies suggest that individuals with disabilities attribute their college success in part to self-determined behaviors, such as self-awareness and self-advocacy (Anctil et al., 2008; Vaccaro et al., 2015).
Although self-determination is widely accepted as contributing to college students with disabilities’ adjustment to college (McConnell et al., 2012), some correlational studies have failed to find a relationship between self-determination and postsecondary outcomes (Haber et al., 2016). Additionally, despite ongoing attention to improving the postsecondary outcomes of students with disabilities since the 1980s (Halpern, 1994), the postsecondary outcomes of students with disabilities continue to lag behind those of their peers (Newman et al., 2011) and our understanding of the factors supporting enrollment and persistence in college remains incomplete (Haber et al., 2016; Kutscher & Tuckwiller, 2019). Studies examining the experiences of students with disabilities at both K-12 and postsecondary levels disproportionately rely on quantitative approaches, followed by qualitative approaches, with few studies making use of mixed methods (Faggella-Luby et al., 2014; Onwuegbuzie & Corrigan, 2018). However, given the complexity of the challenges presented in special education research, mixed methods may be particularly well suited to understanding phenomena and informing potential solutions (Klingner & Boardman, 2011).
Method
This study used a convergent, mixed methods design with a qualitative-dominant crossover analysis (Creswell & Plano Clark, 2018; Onwuegbuzie & Hitchcock, 2015). For the purpose of this article, we will briefly summarize the initial steps in data collection and analysis and then focus our attention on four stages that occurred during the integration process. This study uses a philosophical stance of dialectical pluralism, a meta-paradigm that encompasses the tensions resulting from the use of multiple paradigms in mixed methods research (Johnson, 2017). These tensions present in our efforts to work across perspectives, epistemologies, methods, and values that may or may not be commensurable. To this end, we considered how the application of constructivist perspectives during qualitatively driven stages and postpositivist assumptions that underlie quantitatively driven stages could introduce complexity and dissonance when integrating across strands.
Participants
Thirteen individuals with disabilities participated in both qualitative and quantitative strands of the study, allowing for sample integration (Onwuegbuzie & Johnson, 2006). Participants were enrolled in at least their third year of college or had completed a college program (i.e., postsecondary certificate, 2-year degree, or 4-year degree) and self-identified as receiving disability-related services in high school, college, or both. Five participants initially attended a community or technical college, four enrolled in a 4-year program at a university, three attended private liberal arts colleges, and one enrolled in a 2-year, for-profit program. Most participants attended programs located on the east coast (10), with two participants located on the west coast and one participant in the south. Participants ranged in age from 20 to 38 years (62% were between the ages of 20-21 years), and most were White (69%), with two participants identifying as Black or African American, one as Asian, and one as multiple races or ethnicities. Information on participants’ socioeconomic status was not collected. Self-reported disabilities in high school included attention-deficit/hyperactivity disorder (46%), learning disabilities (46%), physical impairments (23%), autism (15%), and emotional disorders (8%); note that participants could report more than one disability. Participants were recruited through notices sent to the offices of disability resources at local postsecondary institutions and snowballing.
Qualitative Data Collection and Analysis
Interviews
Qualitative data were collected through semistructured interviews that asked participants to reflect on the K-12 and college experiences they felt influenced their persistence in college. 2 Most interviews lasted approximately 1 hour and occurred at a location of the participant’s choosing. Four interviews were conducted via video call due to geographic distance or participant preference. All interviews were recorded and transcribed, and transcripts were emailed to participants for member checking.
Analysis
Analysis began immediately on completion of the first interview and proceeded through several cycles. The final interview occurred after much of the qualitative analysis was complete and confirmed saturation had been reached. Coding was inductive and involved a combination of descriptive, in vivo, and emotion coding (Saldaña, 2013). The initial cycle of coding supported the development of a code book; once code definitions were stable, all interviews were uploaded into a qualitative text analysis software program (QDA Miner 5) and consistently coded using the codebook definitions. Participants’ experiences were then analyzed in a series of data matrices (Miles et al., 2014), leading to the emergence of salient themes.
Quantitative Data Collection and Analysis
AIR Self-Determination Scale
Quantitative data were collected using an online survey platform (Qualtrics). The survey included four sections that asked about participants’ background/demographics, high school experiences, levels of self-determination, and identification with a disability community. For the purpose of this illustration, only the self-determination scale will be discussed. Participants’ levels of self-determination were measured using 18 items from the 24-item AIR Self-determination Scale (AIR; Wolman et al., 1994). 3 The AIR asks respondents to rate how often they engage in self-determined behavior (e.g., “I know what I need, what I like, and what I am good at”) on a scale of one to five (5 = always). The AIR has shown acceptable reliability in previous studies with youth with disabilities (αs above .87; Wolman et al., 1994); in the current study, the AIR had a Cronbach’s α of .94.
Data Collection and Analysis
All participants completed the survey within 1 week following the interview, and the data were analyzed once data collection was complete. Descriptive statistics (i.e., mean, standard deviation, median, range, correlations) and Cronbach’s α were calculated using MS Excel 2016 and XLSTAT 2018.
Multiple Correspondence Analysis
Following completion of qualitative and quantitative analyses, the crossover approach of MCA was applied to qualitative and quantitative results using XLSTAT 2018. MCA is a multivariate technique that explores underlying patterns of response in categorical variables. Greenacre (2016) provides a thorough description of the conceptual and mathematical grounding for this approach. Briefly, the observed frequency of categorical variables (referred to as “profile points”) are plotted relative to their expected frequency as reflected in a contingency table. The distance between the observed and the expected point is a weighted Euclidean distance; because the distance is weighted by the expected profile position, this is referred to as the chi-square distance. Variability in the MCA map is reflected in the measure of inertia, which is calculated by dividing the chi-square statistic by the sample size. However, it is important to note that inertia “is a measure of how much variance there is in the [contingency] table and does not depend on the sample size” (Greenacre, 2016, p. 28); high inertia indicates high variability among responses, while low inertia (i.e., values close to zero) indicates limited variability. The first principal axis accounts for the greatest proportion of the total inertia or variance, conceptually similar to eigenvalues in factor analysis. The dimension accounting for the greatest proportion of the remaining inertia is the second principal axis, and so on. The analysis in this article uses “adjusted” MCA, which was developed by Greenacre (2016) to better account for inertia explained by the solution.
The correlation of a given profile point with an underlying dimension can be determined by examining the profile point’s angle cosine in relation to the axis, conceptually analogous to factor loadings in factor analysis. Angle cosines above 0.70 are considered to be highly correlated with the dimension (Greenacre, 2016). Additionally, researchers can determine how well a two-dimensional map represents the location of a given profile point by summing the squared angle cosines for the profile point in the two dimensions. Profile points with summed squared angle cosines of less than 0.50 are not well represented by the given two-dimensional solution, and their position in the map should be interpreted with caution. In MCA, researchers can also examine the map within the context of supplemental variables; such variables are hypothesized to affect relationships among profile points (e.g., control, confounding variables) and facilitate researchers’ understanding of the profile points’ location on the map (Greenacre, 2016).
Correspondence analysis should be used with nonnegative, categorical data (Lam, 2016) and does not make assumptions about the distribution of the data (Doey & Kurta, 2011). Additionally, correspondence analysis can be appropriately applied with small sample sizes (Bazeley, 2018), and has been used in prior mixed methods research with sample sizes as small as eight participants (e.g., Onwuegbuzie et al., 2014). MCA follows these assumptions of correspondence analysis (e.g., Garson, 2012), but involves the analysis of a set of categorical variables, each with two or more levels (see Abdi & Valentin, 2007 for a simple example of MCA).
Integration Process
In this study, integration occurred through four stages (see Figure 2). During Stage 1, the analysis proceeded as typical for a basic, convergent mixed methods design. Stage 2 involved conducting and interpreting the MCA map, revealing surprising findings. This prompted a return to the qualitative data in Stage 3, during which codes and themes were revisited. Finally, during Stage 4, a revised MCA map was generated, and findings were interpreted in the context of the entire integration process.
Study design and integration process.
Stage 1: Implement a Convergent Mixed Methods Design
The first stage of the integration process involved analyzing and merging qualitative and quantitative data strands. Qualitative and quantitative data were analyzed separately, then combined for analysis in a joint display (Fetters et al., 2013). This stage of the integration process was framed from a qualitative perspective; priority was given to the voices of participants, and it was expected that findings would emerge from the data.
Qualitative Analysis
Qualitative analysis revealed eight high school experiences that participants felt supported their college success. Three facilitators were related to resources: strategic resources (targeted instruction or accommodations designed to address a participant’s learning profile), private resources (private tutoring, evaluations, specialized schools, health insurance, or health care), and transition resources (college exploration activities or bridge program between high school and college). Three facilitators were related to participants’ environmental contexts: encouraging teachers (supportive teachers who believed in the participant’s ability to succeed), challenging high school (rigorous instruction available to the general student body), and parents supportive of college (parents who expected or encouraged pursuit of college). Finally, two facilitators were related to participants’ personal experiences: extracurricular interests (passions that developed participants’ confidence and understanding of their goals) and realizations (moments when participants realized that they need to change their approach or that they were capable of more than what they previously believed).
In addition to these eight emergent themes, participants also discussed their experiences and adjustment as they began their college programs. Based on these descriptions, participants could be classified as experiencing a smooth transition or a challenging transition to college. The seven participants who described their transition as smooth shared that they found their initial college experiences to be aligned with their prior expectations; in some cases, participants described the adjustment to college as being easier than expected. The remaining six participants described how social or academic challenges presented obstacles as they began their college studies.
Quantitative Analysis
Participants’ responses to the AIR indicated that they had strong levels of self-determination (M = 3.83, SD = 0.68). The finding that participants reported high levels of self-determination was expected, given that self-determination is considered a critical characteristic for postsecondary success, and all participants were persisting in or had completed a college program. Nevertheless, the scores reflected some variability in responses.
Integrated Analysis
Finally, qualitative and quantitative data strands were merged using a joint display that compared the responses of participants who reported a smooth transition with the responses of those who did not (see Table 1). The number and percentage of participants endorsing each facilitator, as well as the mean and range score on the AIR, were calculated for both groups of participants. However, the joint display revealed few differences in experiences; a similar percentage of participants endorsed all themes, with the largest differences occurring in the parent support (86% vs. 50%, favoring participants with a smooth transition) and extracurricular themes (50% vs. 14%, favoring those with a challenging transition). The mean and range of self-determination scores was also similar for participants reporting a smooth transition (M = 3.89, range = 2.89-4.67) and those reporting a challenging transition (M = 3.76, range = 2.94-4.72). An independent samples t test showed no statistically significant difference between the means for these groups, t(11) = 0.33, p = .75.
Joint Display of Interview Themes and Mean Self-Determination Scores by Transition Experience.
Note. Percentage indicates the percentage of participants in the transition profile endorsing the facilitator. Self-determination was measured on a scale of 1 to 5 (5 = high levels of self-determination). HS = high school.
Stage 2: Advance Understanding Through Initial Crossover Analysis
Stage 2 of the integration process was quantitatively framed and involved conducting MCA to further understanding of the relationships among the identified themes and variables. Participants’ endorsement of each qualitative theme was transformed into a categorical variable for analysis. If a participant discussed a theme as important to college success, it was coded as “yes”; if the theme was not identified as important, it was coded as “no.” The analysis also included two supplemental variables: level of self-determination and ease of transition experience. Specifically, participants’ level of self-determination was dichotomized to distinguish between participants with “high” self-determination (scores of four or above on a 5-point scale) and those with moderate to low levels (scores below four). Finally, participants’ description of their transition to college as smooth or challenging was transformed to be included as a supplemental variable.
The total inertia for the MCA map was 0.13, indicating limited variability among participants’ endorsement of themes. The first principal axis represented 68.07% of the total inertia, and the second principal axis represented for 8.07% of the inertia, yielding a two-dimensional map that accounted for 76.14% of the total inertia. All profile points were well represented in this solution (i.e., summed squared angled cosines above 0.50), with the exception of self-determination, which had a summed squared angle cosine of 0.31 (see Table 2).
Correlations (Angle Cosines) of High School Facilitators and Supplementary Variables for Dimensions 1 and 2—Original Multiple Correspondence Analysis Map.
Note.Kutscher and Tuckwiller (2020, p. 109).
summed squared angle cosines for dimensions 1 and 2 were below 0.50, indicating the variable is not well-represented in the multiple correspondence analysis map.
Dimensions of MCA maps must be interpreted independently (Greenacre, 2016). Of the variables that were well-represented in the map, all were mostly strongly associated with the primary principal axis (Dimension 1), with the exception of parent support, which was associated with the second principal axis (Dimension 2). In interpreting the MCA map, it is important to note that—when variables are binary—it is an artifact in MCA that the positive and negative points will be located 180° from each other, with a line running through the map’s origin. Therefore, it is of interest on which side of the origin variables fall, relative to other variables (Sourial et al., 2010). Examination of the original MCA map (see Figure 3) reveals that profile points representing the endorsement of themes (i.e., “yes”) were clustered to the right of the origin along Dimension 1, while those representing a lack of endorsement (i.e., “no”) fell to the left. This suggests that participants who endorsed one facilitator were likely to endorse other facilitators, while those who did not discuss a facilitator were less likely to discuss other facilitators.
Original multiple correspondence analysis map of high school facilitators and supplementary variables.
As noted previously, the supplemental variable of self-determination was not well-represented in this solution, and its location should not be interpreted. However, the supplemental variable regarding participants’ transition experience was well represented (summed squared angle cosine of 0.80) and was associated with Dimension 1 (angle cosine of 0.71). Examination of the MCA map revealed that the location of these profile points showed the opposite pattern when compared with participants’ endorsement of facilitators; the profile point “smooth transition-yes” was located to the left of the origin, and “smooth transition-no” was located to the right. This indicates that participants describing their transition as smooth were less likely to endorse high school facilitators, while those who described their transition as challenging were more likely to discuss supportive experience in high school. This finding was unexpected, given existing literature indicating that supportive experiences in high school promote positive outcomes among young adults with disabilities.
Stage 3: Return to Qualitative Analysis
During Stage 3 of the integration process, we returned to the qualitative data to better understand the unexpected results generated in the original MCA. One potential explanation was that participants accessed high school facilitators due to greater disability-related needs, which complicated their college transition. However, reexamination of the qualitative and demographic data did not support this explanation. Rather, it seemed that the definitions and boundaries of two identified themes could be influencing results. Specifically, some participants described the importance of two high school facilitators—challenging high school and parents supporting college—in qualitatively different ways when compared with other participants endorsing those themes.
The challenging high school theme was originally defined as “rigorous instruction available to the general student body.” Of the seven participants endorsing the importance of this theme, four participants described their overall high school experience as academically rigorous. However, three participants described access to challenging coursework as important to their college preparation because it was an anomaly in their high school experiences. These participants described generally encountering low expectations in high school and felt that their participation in one or two advanced courses provided them with a critical glimpse of the academic challenges that awaited them in college.
Similarly, the parents supporting college theme was originally defined as “parents who expected or encouraged pursuit of college.” However, eight of the nine participants endorsing this theme described college as a parent expectation. Many of these participants never questioned if they would attend college after high school but instead were following a path set out for them by their parents. In contrast, one participant described her parents as “surprised” when she told them that she wanted to attend college after high school. However, once she had established this goal, her parents provided unconditional support; this participant saw her parents’ encouragement as essential to her college success.
In light of these qualitative differences in participants’ experiences, we redefined these two themes. The challenging high school theme was redefined as “overall high school experience was academically rigorous.” The parents supporting college was redefined as “parents expected pursuit of college.” We then transformed the data regarding participants’ endorsement of themes into categorical variables using the revised definitions (see Table 3).
Participant Endorsement of Qualitative Themes.
Note. Yes = participant endorsed theme; No = participant did not endorse theme; Yes/No = in the original analysis, participant was identified as endorsing theme, and in the revised analysis, participant was considered to not endorse the theme; HS = high school.
Stage 4: Advance Understanding Through Revised Crossover Analysis
During Stage 4 of the integration process, we conducted MCA with the data transformed using revised theme definitions. Similar to the original MCA map, the revised MCA map showed limited variability among profile points, with a total inertia of 0.11. The first dimension represented 74.6% of the inertia, and the second dimension represented 4.0%, with 78.6% of the total inertia accounted for in this solution. All variables were well represented (i.e., summed squared angle cosines above 0.50), with the exception of self-determination (see Table 4).
Correlations (Angle Cosines) of High School Facilitators and Supplementary Variables for Dimensions 1 and 2—Revised Multiple Correspondence Analysis Map.
Note. asummed squared angle cosines for Dimensions 1 and 2 were below 0.50, indicating that the variable is not well-represented in the multiple correspondence analysis map.
In comparing the revised MCA map with the original, some interesting observations can be made (see Figure 4). First, endorsement of themes (“yes”) was again clustered on one side of the origin (this time to the left of the origin, but this does not have meaning for interpretation), while lack of endorsement (“no”) was clustered on the other side of the origin. The change in definition to the challenging high school theme did not appear to affect its relationship with underlying dimensions and themes. This theme was still more likely to be endorsed when other themes were endorsed and less likely to be discussed as important when other facilitators were not discussed.
Revised multiple correspondence analysis map of high school facilitators and supplementary variables.
However, the profile point representing parent expectations—which was more strongly associated with the second dimension in the original MCA—showed similar association with the first and second dimensions in the revised map. This theme showed the opposite pattern on Dimension 1, with relation to the other profile points. Based on the “parent-yes” profile point location in the revised MCA map, participants who discussed the importance of parent expectations were less likely to have discussed the importance of other facilitators. Additionally, the supplementary variable of “smooth transition”—which was associated with the first dimension in the original MCA—showed slightly stronger association with the second dimension in the revision. Examination of the revised MCA map revealed that both the “smooth transition” and “parent expectations” variables are located in the same quadrant of the map, suggesting that participants who discussed the importance of parent expectations were also more likely to describe their transition to college as smooth.
Discussion
The purpose of this study was to demonstrate how the crossover analysis approach of MCA can be used to support an iterative process of integration. These findings offer three contributions to the field of mixed methods research. First, the findings of this study demonstrate the utility and potential of MCA as an integration approach that can reveal unexpected conversations among qualitative and quantitative data strands. Second, the study alerted us to how our decisions to maintain rigor within each strand had implications for how to proceed within and at subsequent stages of integration and analysis. Finally, these study findings highlight the contribution of iterative integration approaches in offering insights that may be new and/or unclear, raising additional questions that warrant further investigation.
Contributions to the Field of Mixed Methods Research
Utility and Potential of Multiple Correspondence Analysis
Iterative integration through data collection and analysis has been referred to as a conversation between qualitative and quantitative data strands, in which “the data talk to each other” (Moseholm & Fetters, 2017, p. 4). This conversation will be different in each study. In some cases, it is an agreeable discussion of varying points of view. In other studies, the conversation may be a disagreement or may involve one data strand giving suggestions to the other data strand. Although researchers may anticipate or hypothesize what those conversations will sound like before the study begins, it is impossible to know until data collection and analysis are underway. Importantly, conversations are dynamic and ongoing, rather than static and unidirectional, allowing for the construction of an understanding that is mutually influencing. These conversations allow for the emergence, development, testing, and refinement of hypotheses and inductive reasoning that guide subsequent analytic steps and interpretations (e.g., Hatta et al., 2020).
Researchers use integration approaches—including narrative, joint displays, data transformation, and crossover analyses—to detect, understand, and interpret these conversations. While all of these approaches are useful and effective tools for integration, each approach may offer different strengths to the analysis and integration process. This study illustrates how the use of a crossover approach, MCA, surfaced findings that were not apparent when applying other approaches to integration. Although the joint display during Stage 1 did not suggest meaningful differences in participants’ experiences, the MCA map revealed a surprising association between the discussion of facilitators and the experience of challenging transitions. Crossover approaches and data transformation, while less frequently reported in the mixed methods literature (Fakis et al., 2014), offer support to researchers seeking to detect nuanced conversations between data strands. In this study, iterations of analysis provided evidence that varying influences are at play and can inform additional investigations. The Stage 2 MCA map revealed the surprising finding that smooth transitions were associated with not discussing facilitators. The Stage 4 MCA map, on the other hand, showed a relationship between smooth transitions and parent expectations, a finding that aligns with previous research (Doren et al., 2012). However, without the ability to compare findings across all stages of this study, the nuance of parent expectations versus parent support or encouragement would not have been discovered or observed. The iterative approach taken in the current study provided an opportunity to identify nuances in theoretical and empirical understandings of experiences that prepare students with disabilities for college, with the dynamic findings offering insights into the complex interplay of participants’ experiences and their adjustment to college. Although these findings occurred within a special education context, the methodological approaches used in the current study may be applicable in other areas of inquiry where researchers are seeking to explore or develop new understandings of relationships among existing and emerging research constructs of interest.
Rigor and Limitations in Crossover Analysis
Ensuring rigor in mixed methods research requires attention to characteristics of rigorous qualitative and quantitative research, as well as considerations specific to mixed methods studies (Onwuegbuzie & Johnson, 2006). Rigor and the related construct of quality are concerned with bias and determining if the research approaches and findings can be trusted (i.e., trustworthiness; for further discussion of these constructs see Harrison et al., 2020). Qualitative and quantitative researchers typically take different approaches to understanding the role of bias in their studies. Qualitative researchers are often interested in understanding the trustworthiness of their findings; they reflect on and acknowledge the lens and experiences they bring to the study and seek to understand how those preconceptions are shaping their analysis and interpretations (Lincoln, 2002; Miles et al., 2014). Quantitative researchers, on the other hand, are interested in determining the validity of their results and therefore seek to limit or control for bias in their study design (Wiersma & Jurs, 2009). However, iterative integration that takes place across qualitative and quantitative frames may complicate actions taken to maintain the rigor of each methodological approach (Moseholm & Fetters, 2017).
The current study reveals how decisions regarding trustworthiness at a qualitatively framed stage can have implications for validity in subsequent steps in the iterative process, potentially limiting analyses within a quantitatively framed stage. During Stage 1, we developed a code book through several iterations of analysis and then applied our “final” codes consistently across all participant transcripts. Use of a code book supported both trustworthiness in Stage 1 and the validity of MCA results at Stage 2. The return to our code definitions in Stage 3 was also appropriate for a qualitatively framed analysis, as we considered how the boundaries of our codes might be affecting our results. Although such iterations of code development and refining are typical in qualitative inquiry (reflecting the interpretive and grounded nature of understanding phenomenon), changes to the definition of a variable can raise questions about the validity and reliability of quantitative instruments. Therefore, applying quantitative standards of practice to address bias (e.g., reliable and valid instruments) to our code book and definitions could raise serious questions about the validity of results in the quantitatively framed results in Stages 2 and 4.
Similarly, we found that characteristics typically considered to support the trustworthiness of qualitative findings could complicate quantitative analyses at later stages. When developing themes and codes, qualitative researchers identify experiences that are commonly shared across participants, resulting in limited variability across participants. However, many quantitative analyses, including MCA, require variability in participant responses. In this case, the limited variability in responses—perhaps viewed as a strength from a qualitatively framed perspective—became a limitation in the quantitatively framed stages. This was evident in the relatively low measures of inertia for both the original and revised MCA maps. In the illustrated study, an effort was made to support rigorous research approaches and be aware of limitations at each stage of analysis. Moseholm and Fetter (2017) suggest that the approach to addressing bias should follow the underlying predominant methodological approach. However, in our study, rigorous application of research methods at one stage introduced limitations at later stages. It became evident that decision making about rigor extended beyond the boundaries of a single stage of research and required additional efforts to understand the implications of these decisions (e.g., reviewing exemplar studies, methodological consultation, etc.). Researchers engaged in crossover analyses may need additional guidance on how to address considerations for rigor across analysis stages.
Diffractive and Dynamic Findings
Finally, the unexpected study findings illustrate the inherent complexities of social science research and challenge us to question our underlying assumptions about the role of research and what can be known. A fundamental theory of change within the field of mixed methods research is that the combination of quantitative and qualitative approaches yields a more complete understanding of a phenomenon than either approach alone (Fetters & Freshwater, 2015). However, Uprichard and Dawney (2019) challenged the assumption that it is possible to gain a complete understanding of complex social phenomena and emphasized that researchers must also attend to data diffraction—or the way in which our research methods can disrupt or splinter the phenomenon under study. They discuss how the methods researchers use create boundaries and result in different “cuts” of phenomena, which affect our analysis and interpretation: We tend to assume that one method depicts one part or aspect of the object of study and if another method presents a different part or aspect, then the methods have together shown different parts or aspects of the same thing. But . . . there is no sure way of knowing whether empirical data are reflecting one or more objects; these are interpretations that are made after the fact. (p. 22)
The meta-paradigm of dialectical pluralism also recognizes the possibility that it may not be possible to integrate a mixed methods investigation into a comprehensive whole. As Johnson (2017) writes, “It is important to remember that this type of dialecticism might not lead to integration or full merging or a happy resolution of long-standing dualism” (p. 160).
The surprising findings in the current study illustrate the complexity of the college transition process and suggest a need for deeper questioning of our own attempts to generate a comprehensive understanding of this experience. Specifically, the quantitative variable of self-determination is an established predictor of postsecondary success among students with disabilities (Test et al., 2009). However, in this study, self-determination did not show a strong association with any of the MCA map dimensions. One possible explanation for this is that the small sample size, although allowed for with MCA, may have limited variability and contributed to indeterminable associations. However, it is also possible that the measure of self-determination in this study is capturing an entirely different dimension of participants’ experience that cannot be satisfactorily integrated into the crossover findings. This does not mean that the variable of self-determination is unrelated to participants’ experiences of success in college, but instead, it highlights that further investigation, perhaps using a different analytic approach or research design, is needed to better understand how this research-supported construct is related to college persistence.
Limitations
Although mixed method research can provide new insights into phenomena, it is also subject to the traditional limitations of qualitative and quantitative research methodology. First, the small sample size warrants attention from both a qualitative and a quantitative perspective. Using the qualitative guideline of data saturation (i.e., new interviews do not result in major changes to codes or themes; Guest et al., 2006), analyses suggested that saturation was reached with 13 interviews. Based on an empirical investigation of sample size for interviews, Guest et al. (2006) suggested that saturation can be reached with a sample size of six to 12 participant interviews, although they noted that a larger sample size may be needed if making comparisons across groups. In the current study, the two groups (i.e., smooth and not smooth transition) emerged from the participant sample, but a larger sample could have served to strengthen qualitative findings. Likewise, quantitative analyses were limited by the small sample size and lack of variability in participant responses. Variability was also reduced when we coded participants’ scores on the AIR as “high” or “low to moderate” to allow for the inclusion of self-determination as a variable in the MCA. Although self-determination was not well-represented in either MCA map in the current study, this does not mean that quantitatively collected variables should not be merged with qualitative data sets. Rather, the lack of relationship in the current study could be a function of participants’ high levels of self-determination in this sample. Alternatively, as described previously, the inability to uncover a relationship between the theory-driven variable of self-determination and the emergent variables raised by the participants could represent an occurrence of data diffraction. Additional studies could seek to better understand the complexity of combining quantitative data with transformed qualitative data when conducting a crossover analysis.
Although we worked to include data collection and analysis approaches that would support the rigor and trustworthiness of our qualitative data and findings (e.g., member checks, memos, code book), qualitative findings were further limited in several ways. First, the phrasing of the interview questions could have influenced the experiences participants recalled as most salient in high school. Additionally, one must be cautious when interpreting transformed qualitative results; the lack of endorsement of a theme does not necessarily mean that the facilitator was not important, just that it was not salient enough to emerge in the context of the interview. In addition to the limitations of the qualitative and quantitative strands, it is also important to consider the limitations of this mixed methods design (Onwuegbuzie & Johnson, 2006). In this convergent investigation, participants responded to the survey following the interviews. This was a deliberate design decision in that it was anticipated that the interviews would prime participants’ recollection of their high school experiences (one component of the survey). It was also hoped that the interview would establish rapport and support honest engagement with the survey, but it is also possible that this sequence introduced social desirability or other bias.
Considerations for Interpreting Dynamic Findings
Given an underlying assumption that mixed methods support a more complete understanding than qualitative or quantitative methods alone, mixed methods researchers may struggle to make sense of the dynamic findings that surface during integration and subsequent analytic processes. We offer the following reflective questions as a guide to support researchers in recentering their interpretations, using dimensions of the integration trilogy (Fetters & Molina-Azorin, 2017) as a frame:
Philosophical or theoretical dimension. What happens when the researcher interprets dynamic findings through the researcher’s philosophical or theoretical lens?
Researcher dimension. How do dynamic findings resonate or conflict with the researcher’s prior experience or understanding of the phenomenon?
Literature review dimension. Where do dynamic findings fit within existing literature?
Study purpose, aims, and research questions dimension. How do dynamic findings relate back to and evolve the research questions as initially articulated?
Interpretation dimension. How do dynamic findings “talk” to each other (e.g., confirmation, complementarity, expansion, or discordance)?
Research rigor and integrity. Do approaches used to develop dynamic findings support research integrity (e.g., validity, reliability, trustworthiness)?
While we have focused our attention on making sense of dynamic findings after they have come to light through analysis, future discussions may consider how to design mixed methods studies in such a way that allows space for dynamic findings to surface.
Conclusion
Although infrequently applied in mixed methods research, MCA can be a useful approach to promote the integration and analysis of transformed data. Importantly, the iterative application of MCA can lead to research findings that may not emerge through other integration approaches, such as joint displays or narrative integration. However, researchers implementing MCA or other crossover analyses should attend to how decisions made in one stage of analysis may have implications for or limit analyses that occur in later stages. Researchers should also remain open to the possibility of unresolved findings (diffraction), as well as findings that may counter prevailing ideas in the research literature or hypotheses. When thoughtfully applied, MCA offers mixed methods researchers the opportunity to identify new dimensions and connections among qualitative and quantitative findings, advancing new knowledge and insight into novel and long-studied social science phenomena.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
