Using a Joint Display for Building Integration in a Sequential Study: Informing Data Collection for a Participatory Second Phase

Abstract

In the explanatory sequential mixed methods design, methods-level integration occurs between quantitative and qualitative phases. In typical use of this design, statistical findings alone may inform the qualitative phase. However, in studies that apply a participatory approach in the second phase, this can be a missed opportunity to incorporate participant perspectives. This Research Note presents the integration of quantitative and qualitative survey results from phase one of an explanatory sequential study using a joint display, which is then used to inform the development of qualitative data collection prompts. This strategy contributes to the field of mixed methods research by detailing the process for undertaking building integration to inform an explanatory sequential study’s second participatory qualitative phase.

Keywords

explanatory sequential building integration joint display participatory mixed methods integration

Conceptual and empirical literature increasingly outlines and emphasizes the importance of integration in mixed methods research (MMR) (e.g., Guetterman et al., 2020). However, mixed methods scholars have found that quality descriptions (e.g., intent, procedures, and interpretation) of integration are often lacking or absent from articles (Creswell and Plano Clark, 2018), especially in non-core mixed methods journals (Morgan, 2022). Researchers continue to call for clear descriptions and examples of integration practices during mixed methods studies (McCrudden and McTigue, 2019; Zhou and Wu, 2022). Integration can occur at multiple levels: design, interpretation, and methods (Creswell and Plano Clark, 2018; Fetters et al., 2013). Methods-level integration practices include connecting, merging, and building. Building integration is the focus of this Research Note and takes place when results from phase one inform the data collection of phase two (Fetters et al., 2013). Most literature on building integration in sequential designs explicates how qualitative results inform the quantitative phase in exploratory sequential studies (e.g., Crede and Borrego, 2013; Howell Smith et al., 2020).

Relative to exploratory sequential studies, fewer explanatory sequential studies detail if or how the second phase (qualitative) builds from the first (quantitative). Even when other elements of a quality mixed methods study are reported, such as the intent of a design, details about how building integration is accomplished are not often described (e.g., Hyun et al., 2022; Thornberg et al., 2022). This gap in the explanatory sequential literature is widened when considering added complexity from applying a participatory qualitative phase (i.e., participants driving research processes), since the explanatory design is typically found to be quantitatively driven (Morgan and Hoffman, 2021). This is not to say that participatory studies cannot be quantitative, but that using findings from a quantitative measure selected by a research team to inform a subsequent participatory qualitative phase may not build to data collection prompts that optimally capture key ideas relevant to the community of interest. Relevance to focal populations is a key rationale among researchers who combine MMR and participatory approaches (DeJonckheere et al., 2019).

In addition to concerns about relevance, lack of description of building processes limits the assessment of study findings, as decisions about integrating between phases can limit MMR study validity (Ivankova, 2014). Decisions related to building include selecting the key quantitative results to explain in the qualitative phase, such as unexpected findings (Ivankova, 2014), significant or nonsignificant results, outliers, or group differences (Creswell and Plano Clark, 2018).

McCrudden and McTigue (2019) noted practical examples of integration during explanatory sequential studies are limited but needed to support the use of integration techniques. This need for practical integration examples extends to complex mixed methods designs where multiple research approaches intersect (Creswell and Plano Clark, 2018). Joint displays—visual presentations of quantitative and qualitative research components (e.g., results, data, and methods)—are examples of tools that can be used to support mixed methods integration and develop research insights (Guetterman et al., 2021). The current methodological discussion first describes the example explanatory sequential study used to discuss building and then outlines the process used for building integration by using a joint display to inform the development of qualitative prompts for a second participatory phase.

The Example Study

This methodological discussion stems from an explanatory sequential study about adolescent sexual violence prevention (ASVP) program sustainability (Jackson-Gordon, 2022). The study was approved by the authors’ affiliated Institutional Review Board. The primary purpose of the study was to develop recommendations for sustaining ASVP programs that are important and relevant for prevention practitioners doing this work. Figure 1 depicts the study’s procedural diagram.

Figure 1.

Procedural Diagram of Example Study About Sustainability of Programs (Jackson-Gordon, 2022). Note. See Vaughn and Lohmueller (2014) for more information on the GLA approach. qual = qualitative; ASVP = adolescent sexual violence prevention.

Methods-level study decisions were informed by the plan to apply a participatory approach, or research that involves those with insider knowledge for the purpose of action or change (Vaughn and Jacquez, 2020). We chose group-level assessment (GLA), a participatory qualitative method (Vaughn and Lohmueller, 2014), for study phase two because our goal was to develop recommendations for programs that would be useful to practitioners. We note that we do not describe our study as fully participatory, where participants are engaged from study inception to conclusion. To better prepare for the participatory phase of the study, we adjusted study plans to use open-ended survey responses from the first phase as a proxy for participant input and developed a joint display to inform the creation of phase two data collection prompts. This decision making is detailed in the Process Summary and Takeaways section of this manuscript.

The first phase of the study was a survey completed by 97 ASVP program professionals that included closed- and open-ended questions related to ASVP program sustainment. The survey data sources relevant to the current discussion include the following: (1) a measure of program sustainment as an outcome (Palinkas et al., 2020), (2) measures of determinants of program sustainment (e.g., funding, community networks, Palinkas et al., 2020), (3) a closed-ended question about the determinants that the respondents would like to learn more about, and (4) two open-ended questions (e.g., “What are the greatest barriers your program has experienced as it works to sustain over time?”) yielding qualitative data. The second phase applied a GLA with 19 prevention practitioners aimed at expanding upon survey findings and creating recommendations (the intent of this explanatory design). Interested readers can find further details about the study elsewhere (Jackson-Gordon, 2022).

As noted previously, explanatory sequential studies are usually quantitatively focused. However, the open-ended survey responses gathered in the first phase provided rich contextual information that described program-related experiences of the respondents. Because of the planned participatory approach in phase two, we decided to utilize the quantitative as well as qualitative survey results to integrate between the survey and qualitative phase. We created a joint display for qualitative phase prompt development to accomplish this building integration using both forms of survey results. We describe our display as a variant of the interview questions joint display type (Guetterman et al., 2021). This display served as an integration tool to ensure that the most important concepts for sustaining ASVP programs were addressed during the qualitative phase by informing the qualitative data collection prompts. Reflections from the first author on the study process and conversations with the second author informed this discussion.

Methodological Processes for Integration

Our methodological reflections are organized to provide insights on the process for developing the integrative strategy and then a description of the resulting joint display.

Process Summary and Takeaways

The initial building integration plan was to use quantitative survey findings from the first phase to develop prompts for the qualitative second phase. The qualitative survey responses were intended to provide context when reporting quantitative results. The plan changed for three reasons: survey open-ended responses had rich content (i.e., field-specific insights about survey topics); there was little variability in quantitative results across survey constructs (e.g., similar means), posing challenges for identifying what should be expanded upon in the qualitative phase; and we revisited phase two participatory principles. We realized that by excluding the open-ended survey responses during prompt development, the qualitative phase protocol may not capture what is most relevant for the prevention practitioners.

For example, although funding was found to be significantly correlated with sustainment and had the lowest-rated mean compared to other sustainment constructs, practitioners provided more open-ended responses related to the community networks determinant when compared to funding. Further, most community network-related qualitative survey responses revolved around schools, whereas the survey subscale did not address schools. As such, the phase two discussion prompts highlighted community networks and used the term “schools” to increase relevance to participants. In summary, because the survey tool about sustainability was general and not designed for ASVP programs alone, the added qualitative results aligned general perspectives on sustainability with field-specific sustainability. A joint display helped us identify these findings.

To develop the display, we made decisions about how to combine result types, organize the display, decide which results to include, and use the display to facilitate building to the qualitative phase. These decisions resulted in the following process:

• Identify quantitative and qualitative results about each major construct in the survey. Open-ended responses typically fell into predefined constructs from Palinkas et al. (2020). As such, we deductively analyzed open-ended responses based on these constructs, consequently aligning with the categories of quantitative results for each construct (e.g., funding).

• Organize the joint display by major constructs (rows) and type of results and interpretations (columns).

• Include results that demonstrate the relative importance of each construct to the overall goal of program sustainment (e.g., correlation between constructs and overall sustainment score), as identified quantitatively and qualitatively through survey responses.

• Create a column and describe the integrated takeaways derived from reviewing the quantitative and qualitative results. We weighed the relative importance of each construct for program sustainability with priority given to qualitative results (quantitative results were similar). Had quantitative results been more variable, there likely would have been more equal priority in this process.

A strong foundational study purpose, research questions, and guiding participatory goals directly guided and justified integration decisions between phases. For example, the planned use of a participatory method helped to justify the inclusion of qualitative survey responses for building integration. It should be noted that in a fully participatory study, the participants would have assisted in the actual creation of the joint display; this was not feasible for the current study due to advice about ensuring a short time frame between the survey and GLA to avoid disengagement. Participants were unknown to one another prior to the GLA and located across US time zones, posing difficulties for timely scheduling. Convening to develop the joint display may be more feasible for pre-established groups or when incentives are available.

Joint Display and Prompt Development

The final joint display included (1) subscale means from the sustainment instrument (Palinkas et al., 2020); (2) correlations between subscale means and the sustainment outcome measure; (3) numbers of respondents who indicated interest in learning more about each construct; (4) open-ended comment responses, analyzed deductively into the subscale categories; (5) summaries comparing results for constructs; (6) interpretations; and (7) prompts developed for the qualitative phase resulting from interpretations. See Table 1 for an excerpt of the display for two of the major constructs (i.e., funding and community networks). Information provided in the display columns represented key quantitative findings alongside qualitative results intended to capture constructs most relevant to the work of the prevention practitioners who responded (open-ended and learning interest questions). Because of the focus on the survey constructs as determinants of sustainment, the display was organized as such.

Table 1.

Excerpt From Joint Display for Developing Prompts for Two Sustainability Constructs.

Construct	Subscale Mean (SD)	Subscale Correlation With Sustainment	# (%) Interested in Learning More About Subscale Construct	Example Quotes for Each Construct (Verbatim Survey Responses)	Summaries of Results (How This Construct Compares to the Other 7 Constructs for Each Result Type in the Survey)	Interpretation	Prompts Developed Through Integrated Results
Funding and financial support	3.31 (1.10)	.25*	35 (36%)	(B) Funding- the program is entirely grant funded, and we are encouraged every year to “do more with less.” (S) The [ORG] has strong connections to other community organizations and non-profits. This helps with financial sustainability and the continued engagement of community members in the implementation of and participation in [PROGRAM] programming.	(1) Subscale mean rank 8 (lowest score, needs most work) (2) Weakest correlation to sustainment score (1 of 8) (3) 4th concept of interest (in middle) (4) Qualitatively mentioned briefly but frequently as a barrier (5) When mentioned as strength, in connection with other constructs	Funding is a barrier to sustainability. However, the weaker correlation suggests that other factors could play a greater role in relation to sustaining programs and is perhaps connected to community relationships.	(1) To encourage program sustainment and funding, I wish funders knew... (2) _______ has been the best way we have navigated funding issues to optimize sustainability. (3) A strong community network would help with funding by…
Coalitions, partnerships, and networks	3.48 (0.87)	.36**	39 (40%)	(B) We do not have a physical office in the neighborhood we implement programming, and thus, little community members from that area know who we are (S) The biggest factor in sustainability with our teen dating violence prevention program is our connection and development of relationships with the schools, teachers, and districts.	(1) Subscale mean rank 7 (second to lowest score) (2) 3^rd strongest correlation (3) 3^rd highest level of interest (4) Emerged most frequently qualitatively (5) Qual responses provided more context than others	Practitioners have a lot to say about community networks as a strength or barrier. It is also one of the lowest scoring constructs, but it is one of the highest correlations to sustainment.	(1) The relationship between our program and our schools/community is ____________ because ______________. (2) Leaders in our schools and/or communities have gotten involved with our program because… (3) Something we could do to improve our community partnerships, coalitions, and networks is... (4) The theme song for a movie about our program’s relationship with the school/community would be

Note. Subscale and items from Palinkas et al. (2020). (B) refers to barrier; (S) refers to strength. *p < .05. **p < .01. Qual = qualitative.

Each column of the display served a different purpose and was considered when developing qualitative prompts. See Table 2 for intentions of each column category. When determining the conceptual emphasis for the qualitative phase—represented by number of prompts for each construct—each construct was rated by the first author on importance for sustainability based on the results available in each column. Numeric averages of the rated importance for each construct (1–8) were evaluated in the context of the content of qualitative responses to determine which constructs should drive the GLA qualitative phase to increase relevance to participants.

Table 2.

Intentions of Each Column Category of the Joint Display.

General Practice	Joint Display Column Category	Purpose of Inclusion
Identify key quantitative findings from a validated scale to obtain objective importance indices	Construct	• Ideas by which the results are categorized
	Subscale mean and standard deviation	• Relative strength/challenge when comparing subscale constructs
	Subscale correlations with sustainment outcome	• Importance of subscale for program sustainment
Identify subjective quantitative and qualitative indices that reveal relevance to practitioners in addition to nuanced ways of understanding validated constructs	#(%) of respondents interested in learning more about the subscale construct	• Gauging level of interest in each construct for sample
	Example quotes for each construct	• Additional context for nuanced understanding of each construct • Gauging which constructs are most salient to stakeholders
Integrate findings to develop holistic perspectives on key study topics and facilitate triangulation of findings	Summaries of results	• Compilation of construct findings relative to other constructs
	Interpretation	• Bigger picture—need to learn more based on relative importance/relevance based on mixed-methods findings
Develop prompts for phase two that align with the priorities identified from the previous steps	Prompts developed through integrated results	• Demonstrate alignment with key sustainment constructs • # of prompts for each construct based on interpretations • Content of prompts based on all results

Note. GLA = group-level assessment (Vaughn and Lohmueller, 2014).

Prompts (24) were developed based on guidance from Vaughn and Lohmueller (2014), including recommendations for intentional redundancy, open-ended/structured, and fun/serious variations among prompts. Prompt responses develop a database of information available to participants to review and analyze during the GLA (Vaughn and Lohmueller, 2014). Most prompts were designed to address specific constructs, but some were intended to be broad to identify constructs that may not have been reflected in the survey. Some construct-specific prompts were developed to explain or gain additional information about repetitive ideas from the survey responses. Language for prompts was pulled directly from qualitative survey responses to increase relevance to the participants, as participants drive data production and analysis in this method.

Discussion

As mixed methods and participatory research approaches continue to grow in popularity, additional examples of meaningful integration—especially at the methods level—are needed in the scholarly community. The current example of intentional building by using closed- and open-ended survey responses to develop subsequent qualitative data collection prompts is well-suited for studies incorporating participatory frameworks; the prompts placed greater value on participant perspectives in moving the study forward when compared to statistics alone.

Contribution to Mixed Methods Research

This work contributes to understanding about the explanatory sequential design and use of joint displays for accomplishing building integration. Regardless of the philosophical or theoretical underpinnings of the study approach, using open-ended comments from quantitative phase surveys in an explanatory sequential design warrants further consideration and discussion among scholars. For example, this strategy may have potential to serve as an additional safeguard for validity of findings both after the first phase as well as when drawing final conclusions in a study. Integration between phases in the explanatory design is an important step to support overall study validity (Ivankova, 2014). Including qualitative survey content may reveal discordant findings, which could then be compared in the joint display and directly addressed through prompts for the next phase, further supporting validity. Accordingly, the strategies used here could transfer to the development of interview or focus group protocols following a survey. In light of a more fully participatory approach, researchers should also consider participant involvement in the development of the display or review of prompts.

The integration through the joint display and prompt development led to engaged conversations among the study participants during the GLA, which was valuable given the participant-driven data production and analysis method. In the future, researchers could conduct a methodological study to compare results for groups where survey-driven prompts did or did not integrate qualitative survey results. In summary, this discussion contributes an example of building integration in explanatory sequential designs with a participatory second phase, helping to fill the gap in academic literature focused on this specific mixed methods concept.

Footnotes

Acknowledgments

We thank study participants for their part in this study, as well as the funding provided by the University of Cincinnati Graduate Student Governance Association Interdisciplinary Research Fellowship. This work was previously shared at the 2022 American Educational Research Association (AERA) conference. Special thanks to Drs Foote, Sanders, Yang, and Toraman Turk for their review of this manuscript.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Rachel Jackson-Gordon

Vicki L. Plano Clark

References

Crede

Borrego

(2013). From ethnography to items: A mixed methods approach to developing a survey to examine graduate engineering student retention. Journal of Mixed Methods Research, 7(1), 62–80. https://doi.org/10.1177/1558689812451792

Creswell

J. W.

Plano Clark

V. L.

(2018). Designing and conducting mixed methods research (3rd ed.). Sage.

DeJonckheere

Lindquist-Grantz

Toraman

Haddad

Vaughn

L. M.

(2019). Intersection of mixed methods and community-based participatory research: A methodological review. Journal of Mixed Methods Research, 13(4), 481–502. https://doi.org/10.1177/1558689818778469

Fetters

M. D.

Curry

L. A.

Creswell

J. W.

(2013). Achieving integration in mixed methods designs-principles and practices. Health Services Research, 48(6), 2134–2156. https://doi.org/10.1111/1475-6773.12117

Guetterman

T. C.

Fàbregues

Sakakibara

(2021). Visuals in joint displays to represent integration in mixed methods research: A methodological review. Methods in Psychology, 5(1), 100080. https://doi.org/10.1016/j.metip.2021.100080

Guetterman

T. C.

Molina-Azorin

J. F.

Fetters

M. D.

(2020). Virtual special issue on “integration in mixed methods research. Journal of Mixed Methods Research, 14(4), 430–435. https://doi.org/10.1177/1558689820956401

Howell Smith

M. C.

Babchuk

W. A.

Stevens

Garrett

A. L.

Wang

S. C.

Guetterman

T. C.

(2020). Modeling the use of mixed methods–grounded theory: Developing scales for a new measurement model. Journal of Mixed Methods Research, 14(2), 184–206. https://doi.org/10.1177/1558689819872599

Hyun

McWayne

C. M.

Guetterman

T. C.

(2022). Examination of Chinese immigrant parents’ beliefs about children’s social development through play: A mixed methods explanatory sequential study. Developmental Psychology, 58(2), 325–338. https://doi.org/10.1037/dev0001292

Ivankova

N. V.

(2014). Implementing quality criteria in designing and conducting a sequential QUAN → QUAL mixed methods study of student engagement with learning applied research methods online. Journal of Mixed Methods Research, 8(1), 25–51. https://doi.org/10.1177/1558689813487945

10.

Jackson-Gordon

(2022). Adolescent sexual violence prevention program Implementation and sustainability: A mixed methods dissertation (order No. 29282293) [doctoral dissertation, University of Cincinnati]. ProQuest Dissertations and Theses Global.

11.

McCrudden

M. T.

McTigue

E. M.

(2019). Implementing integration in an explanatory sequential mixed methods study of belief bias about climate change with high school students. Journal of Mixed Methods Research, 13(3), 381–400. https://doi.org/10.1177/1558689818762576

12.

Morgan

D. L.

(2022). Who is on the bandwagon? Core and periphery in mixed methods research. Journal of Mixed Methods Research. Advance online publication. https://doi.org/10.1177/15586898221096319

13.

Morgan

D. L.

Hoffman

(2021). Searching for qualitatively driven mixed methods research: A citation analysis. Quality and Quantity, 55(2), 731–740. https://doi.org/10.1007/s11135-020-01025-2

14.

Palinkas

L. A.

Chou

Spear

S. E.

Mendon

S. J.

Villamar

Brown

C. H.

(2020). Measurement of sustainment of prevention programs and initiatives: The sustainment measurement system scale. Implementation Science, 15(71), 1–15. https://doi.org/10.1186/s13012-020-01030-x

15.

Thornberg

Forsberg

Hammar Chiriac

Bjereld

(2022). Teacher-student relationship quality and student engagement: A sequential explanatory mixed-methods study. Research Papers in Education, 37(6), 840–859. https://doi.org/10.1080/02671522.2020.1864772

16.

Vaughn

L. M.

Jacquez

(2020). Participatory research methods – choice points in the research process. Journal of Participatory Research Methods, 1(1), 13244. https://doi.org/10.35844/001c.13244

17.

Vaughn

L. M.

Lohmueller

(2014). Calling all stakeholders: Group-level assessment (GLA)—a qualitative and participatory method for large groups. Evaluation Review, 38(4), 336–355. https://doi.org/10.1177/0193841X14544903

18.

Zhou

M. L.

(2022). Reported methodological challenges in empirical mixed methods articles: A review on jmmr and ijmra. Journal of Mixed Methods Research, 16(1), 47–63. https://doi.org/10.1177/1558689820980212