Using Group Concept Mapping to Engage a Hard-to-Reach Population in Research: Young Adults With Life-Limiting Conditions

Abstract

Patient engagement strategies are used in community-based participatory research. A successful strategy requires that patients, researchers, and health-care providers collaborate to create meaningful outcomes. Hard-to-reach patient populations such as those living with complex physical or psychosocial conditions, who are geographically dispersed, or who are disadvantaged financially or socially, experience judgment, stigmatization, and marginalization within society and in the research process. Therefore, strategies are needed to better engage hard-to-reach populations in research. One strategy to engage this population is group concept mapping (GCM). This article illustrates how GCM was utilized to engage a hard-to-reach population of young adults (YAs) with life-limiting conditions (LLC), parents of YAs with LLC, and health and health and community experts. Study participants were involved in generating, analyzing, and interpreting data. Five attributes of GCM are outlined, and suggestions are made for how other researchers could use GCM to engage their hard-to-reach patient populations.

Keywords

community-based research focus groups mixed methods PAR—participatory action research whole-systems change

Patient engagement research focuses on priorities that are important to patients with an emphasis on producing information that improves health-care practices. This type of research requires collaboration among patients, researchers, and health-care providers to create meaningful outcomes for patients and build sustainable programs (Government of Canada, 2017). Community-based participatory research (CBPR) is often used to involve patients and their families in the research process (Windsor, 2013); however, traditional CBPR methods such as focus groups, interviews, and survey methods exclude some patient populations in research (Humphrey et al., 2017). For example, patients living with complex physical or psychosocial conditions and/or are geographically dispersed may be unable to participate in these methods and may experience judgment, stigmatization, and marginalization within society and research process (Kauffman et al., 2013). Innovative methods are required to engage this patient population in CBPR.

One strategy to engage patients in the CBPR process is group concept mapping (GCM). GCM is a mixed methods approach that engages participants throughout the research process in structured qualitative and quantitative methods (Kane & Rosas, 2018, Kane & Trochim, 2007; Rosas, 2012). This approach enables researchers to capture, organize, and rate conceptual data from individuals and groups and creates a meaningful conceptualization of the issues being investigated (Kane & Rosas, 2018, Kane & Tochim, 2007, Rosas, 2012).

Some researchers have compared GCM to interviews as a patient engagement strategy. One study found that while participants felt engaged participating in an interview and GCM, the GCM subjects appreciated having a role in the research process (Humphrey et al., 2017). GCM was also found to be useful in the measurement of health-related concepts (Humphrey et al., 2017). In another study, the brainstorming phase of GCM was found to be more efficient when considering both time and financial costs and more comprehensive than individual interviews (Rising et al., 2019a). It was also suggested that when choosing a research method, GCM would be a better choice if researchers want to move beyond content generation to higher level conceptualization (Rising et al., 2019a). These studies support GCM as a promising research method for engaging the voices of hard-to-reach patient populations; however, more research is needed to better understand the benefits of using GCM as a patient engagement strategy.

While GCM has been used to understand the health-care system from the perspective of professionals and how to better serve hard-to-reach populations (Mills, Bergeron, & Pérez, 2015), only a limited number of studies have used GCM to elicit hard-to-reach patient experiences (LaNoue, Mills, Cunningham, & Sharbaugh, 2016; Ogden, Barr, & Greenfield, 2016; Velonis et al., 2018). This article illustrates how we utilized GCM to engage a hard-to-reach population of young adults (YAs) with life-limiting conditions (LLC), parents of YAs with LLC, and health and community providers in research. We aim to demonstrate how this hard-to-reach patient population was involved in generating, analyzing, and interpreting GCM data to develop a strategic plan for a public health approach to palliative care.

YAs With LLC

Our hard to reach population was YAs with LLC who experience significant challenges and barriers to achieve their life goals. The YAs in this study have degenerative neuromuscular conditions, may require 24-hr care attendants, fatigue easily, their independent function may be limited to a few fingers or hand to control their power wheelchairs and computers, are geographically dispersed, and may have limited access to transportation and support staff to attend meetings. Their complex physical disabilities, changing health status and energy levels, which include respiratory fatigue and muscular coordination to speak, and difficulty traveling to meetings are significant barriers for a number of these YAs to participate in research requiring face-to-face and focus group meetings (Cook, Siden, Jack, Thabane, & Browne, 2013; Cook, Jack, Siden, Thabane, & Browne, 2016; Joly, 2015a, 2015b).

GCM Methodology

Unlike traditional CBPR methods, GCM methodology (1) generates group wisdom by creating consensus on key ideas and supports systems thinking by eliciting individual experiences to identify group conceptualization of an issue; (2) equalizes or lessens power differentials among participants; (3) engages patients as both study participants and collaborative decision makers throughout the research process; and (4) includes the availability of specific web-based software to engage participants, provide project management, and generate concept maps (Kane & Rosas, 2018; Rosas, 2012). GCM also draws on techniques from many sources such as group facilitation, organizational development, the human potential movement, and the Delphi method (Kane & Rosas, 2018), thus offering an innovative way to engage hard-to-reach populations.

The GCM process has qualitative and quantitative components that can be conducted online, face-to-face, or in combination (Kane & Rosas, 2018; Kane & Trochim, 2007). The qualitative elements are (1) brainstorming ideas from a prepared focus question, (2) structuring the ideas by sorting them into thematic groups based on their perceived similarity, and (3) analyzing and interpreting the concept maps generated (Kane & Rosas, 2018; Kane & Trochim, 2007). The quantitative elements are (1) rating the qualitative statements generated and (2) creating different visual maps to convey key concepts. Typically, GCM is organized into four structured and sequenced activities: (1) brainstorming, (2) sorting, (3) rating, and (4) data interpretation.

Sampling and Recruitment

Ethical approval was obtained from Athabasca University (21795) and a community health Ethics Research Board (2015-073). Purposeful and snowball sampling (Patton, 2014) was used to recruit all participants: YAs, parents of YAs, and health and community providers. For example, YAs and parents were recruited from those that had attended a children’s hospice and/or connected with the youth coordinator for the regional neuromuscular organization. Parents whose YA had died were recruited to include their perspective about the barriers and challenges their YA experienced and their recommendations for change. Health and community providers were recruited to reflect our philosophical framework of a public health approach to care that supports both health and well-being (Abel & Kellehear, 2016; Kraus de Camargo, 2011; Patient Engagement Resource Centre, 2017).

Determining which YAs and parents to invite to participate was sensitively managed by vetting the names of potential graduates of the children’s hospice (when youth turn 18 years of age, they can no longer continue in child and youth programs) through the counselling program and a neuromuscular nurse clinician who continued to be in contact with them. Invitations were not distributed to any YAs or parents if the health and well-being of the YA was not known. The recruitment strategy took into consideration the need to minimize burden and stress on the YAs, and the information letter outlined participation options to suit varying energy levels and time. For example, the flexibility of GCM methodology allows participants to contribute to all phases, or just one or two phases, without compromising reliability or validity (Kane & Rosas, 2018). Participants can determine their contribution each time the study moved to a new phase, and the GCM web-based software allows easy access with one log-in, password, and demographic information for the entire study.

An e-mail invitation with the letter of information was sent to all potential participants. In addition to this e-mail invitation, the youth coordinator recruited through his social media connections with the YAs. The letter of information outlined the purpose of the study, the different phases, and how they would be conducted (online or face-to-face workshop). Participants were informed that they did not have to commit to all phases but could determine their involvement at the beginning of each phase. They were also informed that brainstorming, sorting, and rating phases would be done using a web-based platform, and data analysis and interpretation would be done at a face-to-face workshop. Sharing information about the requirements for each phase of the study was intentional so that participants understood the varied opportunities for their involvement and could make an informed choice for their overall participation. A link to the online secure project website Concept Systems^® Global MAX™(2016, version 2016.046.12) was provided at the end of the letter. Those interested in participating in the web-based phases of the study clicked on the link, created a user profile, reviewed the information for informed consent, and entered the project site. Participants were asked to identify their participant type (YAs with LLC, parent of YAs with LLC, parent of a YA with LLC who has died, or health-care professional).

From our previous experience with these YAs, we learned that reminders were required and to allow a generous amount of time for response to an invitation and online participation. Two e-mail reminders were sent, and the YAs and parents who had not responded were phoned to ensure that they received the e-mail invitation. The health and community providers were also sent e-mail reminders.

GCM web-based system does not identify individual participants, thus assuring confidentiality for online participants. Confidentiality for those that attended the workshop was not assured; however, the researchers advised participants to share what they were publicly comfortable sharing and only discuss concepts and ideas outside the workshop that did not identify specific people.

Method

Phase 1

Brainstorming

In this phase, participants were asked to answer the following qualitative question by filling in a text box: “What could be done to improve services and opportunities that will provide a palliative approach to care for YAs with life-limiting conditions?” There was no limit to the number of responses participants could offer. While all ideas submitted were anonymous, they could be viewed, but not edited by all online participants. If necessary, participants could ask their care attendant to type in their responses. To accommodate the health status of the YAs, we allowed a generous amount of time for participants to respond and sent two e-mail reminders 2 weeks apart.

After 4 weeks, brainstorming was closed to participants, and the researchers synthesized the responses to remove similar and duplicate ideas to create a unique list of statements for Phase 2. Forty-eight participants generated 84 statements that were synthesized to 64 unique statements. For example, the statements “Establish a young adult hospice program similar to (name of children’s hospice), but for young adults” and “Ensure they have access to adolescent-appropriate palliative care distinct from pediatric or adult” were synthesized as “Establish a young adult palliative care program to ensure access to appropriate care that is distinct from pediatric and adults programs.” From beginning to end, this phase of the study took 6 weeks. In studies involving groups without health and mobility issues, Phase 1 could be completed in 1–2 weeks (Kane & Trochim, 2007).

Phase 2

Sorting and rating ideas

The same pool of participants invited to Phase 1 were invited by e-mail to participate in Phase 2. It was clearly stated that they were being invited, but not required to participate in Phase 2, and they did not need to participate in Phase 1 to participate in Phase 2. Participants from Phase 1 logged directly onto the project website, and participants who did not participate in Phase 1 reviewed and accepted the consent information prior to entering the study.

Phase 2 participants were asked to (1) sort the statements generated in Phase 1 into virtual groups based on their perceived similarity in meaning to each other and to choose a name to describe each group of statements and/or (2) rate the statements by considering the importance of the statement relative to the other statements generated. Participants had the option to complete both tasks or choose just one, there was no time limit to complete either task, and the participants could log on or out of the project website as many times as needed. This part of the study took 4 weeks.

Generating a concept map

A certified Concept Systems facilitator (K.B.) analyzed the sorting and rating data using GCM software (version 2016.046.12) (Concept Systems Incorporated, 2016). Multidimensional scaling and hierarchical cluster analyses were used to depict relationships between statements, create the clusters, and generate the point, cluster, and importance rating cluster concept maps (Kane & Rosas, 2018; Kane & Trochim, 2007). These concept maps represent the group conceptualization of the ideas generated from individual responses.

Point map

The point map is the first map in GCM analysis, and it demonstrates how the participants sorted the 64 statements, named each group they created, and ranked each of the statements relative to the other statements listed. The point map shows how the statements are plotted on an x–y graph (Figure 1). Each dot represents a statement that is identified by a number. The closer the dots are to each other, the more often participants sorted these statements together, and the further away they are from each other, the less often participants sorted them together (Kane & Trochim, 2007).

Figure 1.

The point map demonstrates a visual arrangement of each of the ideas generated by the participants and plotted on an x-y graph. Each point is an idea, and the closer the points are to each other, the more often participant sorted these ideas together.

Cluster map

Next, cluster maps were created to display the 64 points as two-dimensional polygons based on how conceptually similar or dissimilar the statements were to each other (Kane & Trochim, 2007). The distance between the dots, rather than the exact location of the ideas on the map, illustrates the degree of similarity between statements. The software can be used to generate numerous cluster maps, each of which displays a different number of clusters (from 2 to 23 clusters). There is no formula for selecting the “correct” number of clusters for a study. The process to determine the “best fitting” cluster solution requires the researchers to examine a range of possible cluster solutions suggested by the software program, consider the content of the statements within clusters and the average number of categories from the sorting process, and, finally, invite a small number of participants to review the proposed cluster solution (Kane & Trochim, 2007).

For this study, a ledger of how clusters were reduced from a 12- to a 3-cluster solution was constructed. This included the average bridging values (a calculation of whether a statement was sorted with statements that are further away on the map) and examination of the content of statements for each cluster solution (Kane & Trochim, 2007). Then, the average number of categories used by participants was calculated. Both researchers reviewed the results and agreed that a 5-cluster solution was the best fit for this study. Next, the software generated a label for each cluster, which was reviewed by the researchers, and a draft title was assigned to each cluster.

Importance rating cluster map

The third map generated was the importance rating cluster map. In concept mapping, the number of layers in each cluster represents the importance ratings of the statements per cluster. The more layers within the cluster, the more highly the statements in that cluster were rated (Kane & Rosas, 2018; Kane & Trochim, 2007).

Phase 3

Map interpretation

GCM has the flexibility to be conducted online or in-person (Kane & Trochim, 2007). We determined that the third phase would be most effective as a face-to-face workshop so the participants could engage in group dialogue to hear and understand the perspectives of all the participants. Giving voice to some of these YAs among the health and community providers was an important aspect of our patient engagement strategy. To support equal representation and have optimal numbers for small group work and larger group discussions (Patton, 2014), we targeted 15 participants to attend a half-day face-to-face workshop.

Participants were purposively selected based on their unique perspective and experience and included representation of the YAs, parents, and health and community providers. The YAs and parents were selected from those who were within a 1-hr drive or public transport and comfortable speaking in a group. YAs and parents were not required to attend as a dyad. Health and community providers were selected to provide a range of expertise in clinical and advocacy roles. Reimbursement of parking expenses and taxi or public transit fares were offered to all participants. The workshop included both large and small group discussions as a means to provide ample opportunities for people to share their viewpoints.

The purpose of the workshop was to provide study participants an opportunity to review the generated maps, finalize the naming of the clusters, make recommendations for further data analysis, and consider ways to use the information to develop a collaborative and strategic plan (Kane & Rosas, 2018; Kane & Trochim, 2007).

Special attention was given to ensure participation in the workshop was as easy as possible for the representative YAs (Patient Engagement Resource Centre, 2017). For example, the workshop space had to be accessible to public transit routes and large enough to accommodate power wheelchairs; the start time was set to accommodate the extra time the YAs need to get ready and access public transit; and refreshments (e.g., smoothies and soft pastries) were specifically chosen for their ease of consumption in public and to accommodate chewing and swallowing difficulties.

Review of the maps

Participants formed small groups to review the statements that comprised each cluster on the map and suggest a name that best described its theme. Suggested names were shared with the larger group and then participants voted anonymously for the name they believed best suited each cluster. Next, participants confirmed the importance of each of the clusters relative to each other. This phase is an important demonstration of the integration of quantitatively generated data with group qualitative data analysis and patient engagement where participants are both researchers and collaborators.

Finally, we moved toward a conceptual analysis to determine the most important actions to create a public health approach to palliative care for YAs. Considering the placement of the clusters, and theme analysis of the content and rating of statements within clusters, three conceptual groups of the five-cluster solution were identified.

Results

YAs with LLC, parents, and health and community providers were represented in all phases of this study. Table 1 displays how many participants were involved in each phase of the study. Statistical requirements for GCM were met with more than double the number of participants needed for each phase of the study (Kane & Rosas, 2018).

Table 1.

Number and Type of Participants in Each Phase of the Study.

Phases	Young Adults	Parents of Young Adults	Community and Health Providers	Total Number of Participants
Phase 1: Online brainstorming	10	13	25	48
Phase 2: Online sorting	5	2	12	19
Phase 2: Online rating	9	6	8	23
Phase 3: Face-to-face workshop	3	3	8	14

The 64 statements were sorted to represent five clusters. Participants agreed on the following descriptive names: (1) community education and development, (2) opportunities and resources, (3) creating a continuum of care, (4) personal funding, and (5) eligibility criteria and coordinated funding and services (Figure 2). Considering the placement of the clusters, and theme analysis of the content and rating of statements within clusters, participants next identified three conceptual groups (1) programming, (2) creating a continuum of care, and (3) funding. Programming had the highest average importance rating followed by creating a continuum of care and funding, respectively. Further, 30 of the 64 originally generated statements were rated as important by all participants.

Figure 2.

The five-cluster solution with importance ratings and the three conceptual groupings demonstrates the analysis and integration of the group concept mapping quantitatively generated data with the participants’ qualitatively generated data. The more layers within a cluster, the more highly the statements within that cluster were rated by participants.

The calculated stress value for this study was 0.2655. A stress value reflects how often ideas were sorted together and is compared to the average stress value (0.28) calculated from pooled GCM study analyses (Rosas & Kane, 2012). Having a lower stress value than the calculated average verifies that the participants sorted the statements in a similar manner (Rosas & Kane, 2012).

In summary, this study found that the YAs with LLC, parents, and health and community providers identified the need to improve programming, create a continuum of care, and improve funding as the most important strategic directions to develop a public health approach to palliative care based on 30 specific action-oriented recommendations to achieve these goals.

Discussion

This study demonstrates how GCM supported our primary aim to engage YAs with LLC, parents, and health and community providers in CBPR. To our knowledge, this study is the first of its kind to engage YAs with neuromuscular LLC in CBPR as a patient engagement strategy. GCM provided the opportunity for this population to participate in a discussion and offer their recommendations to overcome the barriers they face.

Researchers can consider the attributes of GCM to engage other hard-to-reach patient populations such as those living with other complex physical or psychosocial conditions, who are geographically dispersed, or who are disadvantaged financially, socially, and/or by race. Five important attributes of GCM to support patient engagement with a hard-to-reach population are described below.

First, GCM supported us to reach out, engage, and hear the complex real-world issues these YAs face (Rising et al., 2019b; Velonis et al., 2018) and include a full range of participants with expertise on actions to support a public health approach to palliative care for YAs. Two online phases (1 and 2) allowed participants to contribute to the study when and where it was most convenient for them. This group of YAs have lived with progressive degenerative conditions most of their lives and are equipped with technology-supported computers to enable them to participate in the computer-generated activities. Phase 3 (in-person workshop) supported those that wanted to have their voices heard among other stakeholders and had transportation options to attend the workshop. While not all participants would have the ability to participate online and come to the workshop, it gave voice to a representative group that have never had their collective voices heard before.

The second important attribute of GCM was generating group wisdom and conceptualizing this wisdom into action-oriented recommendations. We used the structured and sequenced GCM method to (1) brainstorm individual ideas anonymously and without dialogue with others, (2) engage participants in the sorting and rating phase independently of each other to make sense of what was shared, and (3) facilitate a group discussion to gain a richer understanding of the recommendations that evolved from the group (Rosas, 2017). Findings based on the collective wisdom of the group have greater credibility with policy makers and other stakeholders who can change the system (Kane & Rosas, 2018).

The third important attribute was equalizing the power dynamics among the YAs, parents, and health and community providers. In GCM, particularly when using the web-based platform versus face-to-face interviews, power dynamics between those who are seen as “experts” (e.g., health and community providers) and those that are viewed as “patients” are diminished (Kane & Rosas, 2018). Everyone can generate ideas without having to defend, dialogue, or develop consensus with other participants. In other CBPR methods, participants can generate their own ideas, but the presence of other people may influence what they share based on hierarchies among users and providers in health care (Israel, Eng, Schulz, & Parker, 2012). Additionally, no one participant holds the decision-making power, including the researchers in the GCM process; all participants have equal participation in the outcome. For example, the statistical procedures, rather than the researchers, analyzed the sorting and rating data, and everyone had a vote on the final conceptual naming and grouping of clusters.

A fourth important attribute demonstrated was the ability of GCM to engage participants as both data generators and as collaborative decision makers throughout the research process. In traditional CPBR studies, participants are seen as data sources and seldom have a voice in the overall interpretation or conceptual interpretations of the data (Israel et al., 2012). GCM provides a way to capture the contextual nuances of participants experiences (Vaugh, Jones, Booth, & Burke, 2017). Participants told us that “we felt like we were finally being heard” and “the concept mapping process was effective for us to be involved and feel empowered.”

Finally, purchasing Concept Systems^® Global MAX™ (Concept Systems Incorporated, 2016) software developed to conduct GCM research studies was beneficial. From a participants’ perspective, it provided a web-based site for contributing to the research process. For example, YAs could choose the date, time, and location of when to participate and be able to log on and off as needed without losing data. The first two online phases of the study did not require attendance at meetings to participate, which offered opportunities for those who would be unable to attend a meeting at a specified time and day, and/or require a wheel chair equipped van and support staff to accompany them to a face-to-face meeting. The flexible access also enabled busy parents and health and community providers to participate at times that were most convenient for them. This is similar to other findings using GCM with hard-to-reach populations (Kane & Rosas, 2018; Mills et al., 2015; Vaughn et al., 2017). The software also allowed for the generation of the concept maps based on the quantitative methods used in GCM. This provided easy-to-use visuals to dialogue with participants.

The cost of purchasing GCM software is reasonable for funded research projects and makes conducting a GCM project easy. However, if purchasing is not possible, GCM is still a feasible research method. For example, online survey platforms such as SurveyMonkey™ could be used to generate ideas from a focal question and rate ideas along with using statistical software programs such as SPSS or SAS to analyze data. This allows researchers to maintain the participatory feature of GCM with participants regardless of whether the web-based platform specifically developed for GCM is used. In addition, researchers can determine what aspect of the GCM method is web-based, face-to-face, or a combination. In our study, we chose to use web-based platform for brainstorming, sorting and rating, and face-to-face for interpretation. Other studies have used a variety of web-based and face-to-face combinations (Bergeron & Lévesque, 2014; Kane & Rosas, 2018; Mills et al., 2015; Rising et al., 2019b).

In summary, GCM provided important methodological options to engage our hard-to-reach patient population in research, and it moved the research process from problem identification and content generation to a higher level conceptualization with an action plan.

Conclusion

GCM offered us flexible strategies to successfully engage our hard-to-reach patient population of YAs with LLC in research. We outlined the sensitive management of the recruitment process, our purposeful decision-making to address potential barriers related to participating, and provided options for study participants to choose their level of involvement in our study. The GCM process led to the identification of three important actions to develop a public health approach to palliative care: improve programming, create a continuum of care, and improve funding along with 30 specific actions to achieve these goals. We outlined five significant attributes of GCM methods for patient engagement in this study. The ability to (1) engage hard-to-reach patient populations; (2) generate group wisdom and conceptualize this wisdom into action-oriented recommendations; (3) equalize the power dynamics among participants including patients, their families, and health and community providers; (4) engage participants as both data generators and collaborative decision makers throughout the research process; and (5) the option to utilize specific GCM software to engage participants, analyze results, and generate visual maps of the outcomes. Finally, we offered suggestions for other researchers using GCM to engage their hard-to-reach patient populations.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded as a CIHR SPOR Patient Engagement Strategy Grant FRN 139755. Ethical approval was obtained from Athabasca University (21795) and a community health Ethics Research Board (2015-073). Informed consent was confirmed in all the three phases.

References

Abel

Kellehear

(2016). Palliative care reimagined: A needed shift. BMJ Supportive & Palliative Care, 6, 21–26. doi:10.1136/bmjspcare-2015-001009

Bergeron

Lévesque

(2014). Designing active communities: A coordinated action framework for planners and public health professionals. Journal of Physical Activity & Health, 11, 1041–1051. doi:10.1123/jpah.2012-0178

Concept Systems Incorporated. (2016). Concept systems global MAX™. Retrieved from www.conceptsystems.com/home

Cook

Siden

Jack

Thabane

Browne

. (2013). Up against the system: A case study of young adult perspectives transitioning from pediatric palliative care. Nursing Research and Practice, 2013, 286751. 10 pages. doi:10.1155/2013/286751

Cook

K. A.

Jack

S. M.

Siden

Thabane

Browne

(2016). Investing in uncertainty: Young adults with life-limiting conditions achieving their developmental goals. Journal of Palliative Medicine, 19, 830–835. doi:10.1089/jpm.2015.0241

Government of Canada. (2017, July 23). Patient engagement—CIHR. Retrieved October 24, 2018, from http://www.cihr-irsc.gc.ca/e/45851.html

Humphrey

Willgoss

Trigg

Meysner

Kane

Dickinson

Kitchen

. (2017). A comparison of three methods to generate a conceptual understanding of a disease based on the patients’ perspective. Journal of Patient-Reported Outcomes, 1–9. doi.10.1186/s41687-007-0013-6

Israel

B. A.

Eng

Schulz

A. J.

Parker

(2012). Methods in community-based participatory research for health (2nd ed.). John Wiley.

Joly

(2015a). Transition to adulthood for young people with medical complexity: An integrative literature review. Journal of Pediatric Nursing, 30, e91–e103. doi:10.1016/j.pedn.2015.05.016

10.

Joly

(2015b). Access to services for young adults with medical complexity. Nursing Ethics, 24, 329–336. doi:10.1177/0969733015602053

11.

Kane

Rosas

(2018). Conversations about group concept mapping: Applications, examples, and enhancements. Sage.

12.

Kane

Trochim

(Eds.). (2007). Concept mapping for planning and evaluation (1st ed.). Sage.

13.

Kauffman

Dosreis

Ross

Barnet

Onukwugha

Mullins

C. D.

(2013). Engaging hard-to-reach patients in patient-centered outcomes. Journal of Comparative Effectiveness Research, 2, 313–324.

14.

Kraus de Camargo

(2011). Systems of care: Transition from the bio-psycho-social perspective of the international classification of functioning, disability and health. Child: Care, Health and Development, 37, 792–799. doi:10.1111/j.1365-2214.2011.01323.x

15.

LaNoue

Mills

Cunningham

Sharbaugh

(2016). Concept mapping as a method to engage patients in clinical quality improvement. Annals of Family Medicine, 14, 370–376.

16.

Mills

S. L.

Bergeron

Pérez

. (2015). Using concept mapping to develop a strategy for self-management support for underserved populations living with chronic conditions, British Columbia, August 2013-June 2014. Preventing Chronic Disease, 12, E173–E178. doi:10.5888/pcd12.150183

17.

Ogden

Barr

Greenfield

(2016). Determining requirements for patient-centered care: A participatory concept mapping study. BMC Health Services Research, 17, 780–792. doi:10.1186/s12913-017-2741-y

18.

Patient Engagement Resource Centre. (2017). PERC advisory board. Building patient engagement in research: A guide for research teams. A product of the INSPIRE-PHC Patient Engagement Resource Centre (PERC). Retrieved from https://www.patientengagement-phcresearch.com/

19.

Patton

M. Q.

(2014). Qualitative research & evaluation methods: Integrating theory and practice (4th ed.). Sage.

20.

Rising

K. L.

LaNoue

Gentsch

Doty

A. M. B.

Cunningham

Carr

B. C.

… Mills

(2019a). The power of the group: Comparison of interviews and group concept mapping for identifying patient-important outcomes of care. Medical Research Methodology, 19, 1–9. doi:10.1186/s12874-018-0656-x

21.

Rising

K. L.

Doyle

S. K.

Powell

R. E.

Doty

M. S.

LaNoue

Gerolamo

A. M.

(2019b). Use of group concept mapping to identify patient domains of uncertainty that contribute to emergency department use. Journal of Emergency Nursing, 45, 46–53. doi:10.1016/jen.2018.05.015

22.

Rosas

S. R.

(2012). The utility of concept mapping for actualizing participatory research. Cuadernos Hipanoamericanos de Psicologia, 12, 7–24.

23.

Rosas

S. R.

(2017). Group concept mapping methodology: Toward an epistemology of group conceptualization, complexity, and emergence. Quality & Quantity, 51, 1403–1416.

24.

Rosas

S. R.

Kane

(2012). Quality and rigor of the concept mapping methodology: A pooled study analysis. Evaluation and Program Planning, 35, 236–245.

25.

Vaughn

L. M.

Jones

J. R.

Booth

Burke

J. G.

(2017). Concept mapping methodology and community-engaged research: A perfect pairing. Evaluation and Program Planning, 60, 229–237. doi:10.1016/j.evalprogplan.2016.08.013

26.

Velonis

A. J.

Molnar

Lee-Foon

Rahim

Boushel

O’Campo

. (2018). “One program that could improve health in neighbourhood is ____?” using concept mapping to engage communities as part of a health and human services needs assessment. BMC Health Services Research, 18, 150–162.

27.

Windsor

L. C

. (2013). Using concept mapping in community-based participatory research: A mixed methods approach. Journal of Mixed Methods Research, 7, 274–293. dio:10.1177/155869813479175