Calling All Stakeholders

Abstract

Group-level assessment (GLA) is a qualitative and participatory large group method in which timely and valid data are collaboratively generated and interactively evaluated with relevant stakeholders leading to the development of participant-driven data and relevant action plans. This method is useful across a wide range of evaluation purposes in many environments. GLA involves bringing a large group of participants together to build a common database through the co-identification of relevant needs, judgments, and priorities. The GLA process proceeds through the following seven steps: climate setting, generating, appreciating, reflecting, understanding, selecting, and action. This article describes the methodological development and process of conducting a GLA and its various applications across the evaluation spectrum. We highlight several exemplars where GLA was used in order to demonstrate the particular nuances of working with different sizes and types of groups and to elaborate on our learnings from the wide applicability of the method.

Keywords

methodological development content area qualitative methodology

Introduction

Popular, “tried and true” methods commonly used in program planning, evaluation, and related disciplines include focus groups and one-on-one interviews with an expert evaluator/facilitator. Although each method certainly has advantages when executed by well-trained evaluators in suitable settings, they have a tendency to be overused and/or used exclusively. Focus groups and one-on-one interviews have disadvantages including the significant amount of time required for completion, potentially small sample size, lack of participant involvement, lack of participant “buy-in,” expert-driven, emphasis on socially desirable responses, limited scope, and no action as a result (Britten, 1995; Cornwall & Jewkes, 1995; Willis, Green, Daly, Williamson, & Bandyopadhyay, 2009).

In particular, focus groups are more likely to be effective and result in robust data when participants are articulate, extroverted, and responsive to the moderator’s questions. Thus, focus groups may be inadvertently biased against timid and less vocal participants (Beyea & Nicoll, 2000). Although focus groups rely on group interaction as the source of data (Kitzinger, 1995; Morgan, 1998), the ideal size for a single focus group is 6 to 10 participants (Morgan, 1998; Powell & Single, 1996) which not only limits the method to one small group at a time but also makes it difficult to recruit diverse stakeholders within a single focus group. Because the moderator directs the discussion and determines the timing of the questions and probes, the inherent nature of focus groups can prevent participants from synergistically interacting with each other and driving the direction of the results. A major disadvantage of exclusively utilizing one-on-one interviews is the interpersonal disconnection and dislocation from social context that occurs in the interview setting (Wilkinson, 1998). For some individuals, one-on-one interviews may feel confessional or therapeutic in nature with participants unconsciously using the interviewer as a sounding board to vent private frustrations and complaints or share secrets (Britten, 1995). This can result in an overemphasis on negative data that participants may not be willing to own or act upon publicly within a group. In addition, the private nature of interviews can easily result in potentially relevant data being rendered unactionable because the findings are perceived to be imposed from a source external to the group. Even with a small number of participants, the interview method is extremely time consuming often taking months to schedule, conduct, and then even longer to analyze the data (Draper & Swift, 2011). Further, data analysis is subject to the interpretative perspective of a sole interviewer and devoid of a larger stakeholder group’s contextual knowledge of the issue.

Both focus groups and individual interviews are resource-intensive and slow with regard to the vast amount of data generated and the hours of transcription required before data analysis can occur (Mansell, Bennett, Northway, Mead, & Moseley, 2004; Wilkinson, 1998). Analysis of focus group and interview data is evaluator-led, occurs post-data collection without the participants, and can be cumbersome and time-consuming (Powell & Single, 1996). Both focus groups and interviews tend to be researcher-centric and “one-sided” whereby an outside “expert” designs the questions to be used and conducts the primary data analysis (Mansell et al., 2004; Wilkinson, 1998).

Given the limitations and nonparticipatory nature of focus groups and one-on-one interviews, an evaluation method is needed where relevant, timely, actionable, and valid data are collaboratively generated and interactively analyzed among the group itself. Group-level assessment (GLA) is one qualitative and participatory large group method that collaborates directly with relevant stakeholders and leads to the development of participant-driven data and relevant action plans (Vaughn, Jacquez, Zhao, & Lang, 2011). Within the realm of evaluation, GLA can be used as a “systematic process to determine merit, worth, value, or significance” across the spectrum of evaluation from program planning/management and needs assessment to designing relevant next steps for action (American Evaluation Association, 2014). GLA is well suited for formative evaluation to assess needs and design a plan for the next iteration of programs in a participatory, stakeholder-engaged manner. As an evaluation tool, GLA best fits within the empowerment evaluation approach because of the focus on “helping people help themselves” and the ideas of self-determination and community action (Fetterman, 1994, 1). Akin to other large group stakeholder methods such as World Café (Brown & Isaacs, 2005), Future Search Conference (Weisbord, Weisbord, & Janoff, 2000), design charrettes (Condon, 2007), community summits (Steil Jr. & Watlington, 2007), and search conferences (Emery & Purser, 1996), GLA allows for the identification of needs and priorities within a large group setting where the relevant stakeholders/consumers/community members have the knowledge, experience, motivation, and expertise to inform and direct the program, intervention, action, and/or outcome. Thus, GLA can be used to answer a wide variety of questions and address issues related to evaluation practice. Like other forms of participatory evaluation and research, the GLA emphasizes “improvisation and personal responsibility in a manner which embraces and affirms multiple realities and local diversity” (Campbell, 2002, 22). The GLA is positioned as a socially relevant evaluation method that encourages those traditionally studied to have voice in the evaluation and research process (Hickey & Mohan, 2005; Pain, 2004). Thus, the GLA emphasizes “knowledge for action” with relevant stakeholders and informed consumers as active contributors within a nonhierarchical process (Cornwall & Jewkes, 1995; Lister, Mitchell, Sloper, & Roberts, 2003; Pain, 2004).

The GLA was initially developed as a modification of the Group Level Team Assessment (Reddy, 1996) which was used as a 3-day training and organization development tool to enhance the function of intact organizational teams. Since the development of the GLA in 1998 (Vaughn, & Lohmueller, 1998), we have used the method in many facets of evaluation-related projects including program evaluation, research, needs assessment, intervention planning, project management, curriculum development, and community building. However, to date, we have not published a detailed account of how to conduct a GLA nor the underlying characteristics and benefits of GLA. This article advances the previous literature on GLA by describing a step-by-step account of the process for conducting a GLA and its various applications. We also highlight several exemplars where GLA was used in order to demonstrate particular nuances of working with different size and type of groups and to elaborate on our learnings from the wide applicability of the method.

GLA Process

The GLA is best suited for groups of 15–60 although the process can be adapted for smaller groups or larger groups. The GLA typically proceeds through the following seven steps (see Figure 1) and normally requires a minimum of 3 hr unless the steps are conducted in sequential sessions or unless the group is small and the steps are abbreviated. Participants are recruited for GLA through established partnerships with stakeholders and/or through snowball sampling where one person recommends another person who may be interested and so on (Goodman, 1961). Participants are told ahead of time that all who attend the GLA session are expected to actively participate in the process and that there will be different levels and statuses of attendees (e.g., community members, professionals, business people, students, etc.). Typically, we use an RSVP system for attendees, so that we will know how best to proceed through the steps described subsequently. By asking invitees to confirm acceptance, we can cancel a session if too few register.

Figure 1.

Group-level assessment (GLA) process.

Preparation for GLA

Most of the preparation for GLA involves writing the prompts on the flip charts and placing them on the walls prior to the GLA session. In general, large conference rooms without excessive furniture and plenty of open wall space work best for GLAs. In addition, we have found that nonpermanent, nonbleeding markers (e.g., Crayola^®), and self-stick, adhesive back flip charts prevent damage from occurring to walls. Depending on the size of the group, the number of prompts should be adjusted. A rule of thumb is to have about 1.5 flip charts per participant with a maximum of about 35 prompts regardless of group size. For instance, the authors used 32 flip charts with a group of 65 participants in a program evaluation/planning session conducted with faculty of a university department. Some flip chart prompts may contain more than one statement (e.g., “We tend to overlook/We tend to focus on…”).

Prompts on the wall charts should be balanced between open-ended and structured, strengths and weaknesses, positive and negative, specific and broad, silly and serious (Reddy, 1996). Redundancy is built into the prompts by asking similar questions in different ways, but the similar charts should be spaced apart from each other in the room. Depending on the type of group and purpose of the GLA, prompts may need to address covert issues/hidden agendas, assets and visions of a group, and conflict (Vaughn, Jacquez, Zhao, & Lang, 2011). See Table 1 for common examples of prompts we have used in GLA. These prompts can be taken as is, modified, or serve as a springboard for evaluators to develop relevant open-ended statements that more closely meet the specific needs of their group. Also depending on the group, we have found that the inclusion of imaginative and creative prompts can contribute to a more dynamic and energetic atmosphere (i.e., if our group was an animal; theme song for our group; draw a bumper sticker for our group).

Table 1.

Examples of GLA Prompts Organized by Category.

Open Ended	Structured
We need to do more…. We need to do less…. Hopes/Dreams (related to group or topic) _______(group or topic) is really about…	Our potential for action would be enhanced if we…. Most important areas to address (list areas with Likert-type scale 1 to 5)
Strengths	Weaknesses
As we move forward, we need to consider… Best part of our group? We are awesome because ….	The worst mistake we could make… Our toughest problem to address is… In order to be more effective we should… Biggest barriers to success?
Positive	Negative
We can be proud we…. We are exceptional because we…. Potential assets?	We usually remain silent about… What’s hidden? (in our group or about the topic)
Specific	Broad
The roadblock to our success is… Best way for us to support people? Key collaborators include:	Let’s start/Let’s stop…. We definitely need… Related to (topic or group), I wish….
Silly	Serious
Draw a bumper sticker (group or related to the topic) If our group was an animal… Theme song (represents our group or the topic)	In 3 years, we need to be… ____________is usually ignored (in group or within topic) To improve the functioning of our group, we need to…

Step 1: Climate Setting

During climate setting, the facilitator provides an overview of the GLA agenda and philosophy and a description of each participant’s role in the process. The facilitator then conducts a brief interactive icebreaker or warm-up. Facilitators may be inclined to minimize this step or skip it altogether. However, given the nature of GLA, it is critical that evaluators start the process with this step to build trust while encouraging participants to openly share what they know. Evaluators involved in an ongoing and established partnership with a community group may find it necessary to complete this step only the first time they are in contact with the group. Wall charts used in Step 2 should be covered up during Step 1 to avoid distraction. After the icebreaker, the wall charts are uncovered one at a time while the prompts are read aloud by the facilitator who seeks to clarify each chart. This component prepares participants to respond while still seated. Depending on the size of the group, Step 1 generally takes at least 40 min.

Step 2: Generating

During the generating phase, the cornerstone of the GLA method, each member of the group stands and concurrently responds in writing to the open-ended prompts on the wall charts. Participants are given colored markers (the same color of markers can be used to prevent a comment from easily being attributed to one respondent) and asked to simultaneously walk around the room responding in writing to each heading in any order they choose, with any pictures, words, or phrases they deem relevant (see Figure 2). We typically see a significant increase in energy and interactivity during the generating phase. Depending on group size, Step 2 takes approximately 45 min.

Figure 2.

Participants responding in writing during the generating phase of GLA.

Step 3: Appreciating

Participants are instructed to mill around the room looking at the collective data written on the wall charts with the intent to absorb the totality of responses. During this step, participants can discuss what they observe and add additional comments or stars to responses with which they agree. During Steps 2 and 3, we encourage people to interact as they are responding to, and appreciating the wall charts. Step 3 usually takes a maximum of 10 min for most groups.

Step 4: Reflecting

During the reflecting step, participants are asked to spend time alone thinking about what the data as a whole mean to them. Depending on the group and time constraints, we sometimes ask participants to jot down initial thoughts of what they have observed during the appreciating step. We generally plan a maximum of 5 min for Step 4.

Step 5: Understanding

In Step 5, participants are randomly divided into smaller groups of 5–8 and assigned 5–6 nearby flip charts divided evenly among the small groups (e.g., 49 people and 35 flip charts translates to seven groups of seven people with five charts each). Sometimes the same flipchart prompts are assigned to more than one group depending on the number of participants. Another hallmark of GLA, the understanding step, is designed for participants to discuss responses on the charts and look for common themes to emerge from the data, essentially “analyzing” the data from their perspectives. Each small group is instructed to identify 3–5 common themes across their assigned charts. The small group phase can include small group facilitators. After each small group identifies salient themes, the group reunites and each small group reports verbally on their most commonly occurring themes. As the themes are reported, the facilitator records the major themes for the larger group to see. Step 5 requires a total of 50 min for most groups to complete. We allow 30 min for small groups to develop themes and then 20 min for the large groups to report out.

Step 6: Selecting

In the selecting step, participants clarify the most important ideas by discussing and further distilling the themes that were generated in Step 5. This step can be conducted as a large group with the primary facilitator directing the discussion for the purpose of allowing participants to evaluate the whole of what they are seeing. The other option sometimes used is for self-facilitated small groups to be formed to further discuss the themes from Step 5. For themes that are reported out in the Step 5 but not emphasized in Step 6, facilitators may want to include these themes in a separate “parking lot” list to be addressed by the group at a later time. Although not required (depending on the goal of the project or specific questions), the final stage of Step 6 can be some type of prioritization process using either a voting dot process where participants place colored dots by their top priorities or a rating system where participants rate each of the final themes by importance and feasibility, for example. Step 6 can occur in a short time frame or can be made up of a more in-depth discussion depending on time constraints and future plans. We customarily allow a minimum of 30 min for this step.

Step 7: Action

In Step 7, the large group considers possible next steps based on priorities. One of the distinguishing features of the GLA is the opportunity for the group to be involved in action planning. Like Step 6, the action step is not necessarily required depending on the specific program goals. However, we highly recommend that evaluators carry the participatory component throughout the process and have the group engage in this final step of action planning. This allows informed group members who have generated and analyzed their own data to actively participate in making decisions about how the data can inform relevant future programs, interventions, developments, and/or social change. For Step 7, we typically break into smaller groups and work on action planning for each of the identified priority themes. Because action planning is typically intensive and ongoing, Step 7 can be scheduled at a different time for interested group members. In many cases, the action groups and GLA data become the working structure for groups that continue to meet and work on future efforts.

Uses of GLA

GLA is a qualitative and participatory large group method that is flexible and readily adapted for use in various aspects of program development and evaluation. GLA moves from what Schein (1969) calls the “Dr/Patient model” to what Weisbord (1987) refers to as getting “everybody working to improve whole systems.” GLA rapidly results in concrete, meaningful ideas including perceived barriers to outcomes, inadvertent oversights, and participant preferences that allows for group engagement and the potential for capacity building and collaborative action because each participant has a piece of the puzzle. The group of inside stakeholders, rather than an outside “expert,” is responsible for data generation, data revision, data prioritization, and presentation of relevant information. When brought together and guided through GLA, the group publicly and synergistically shares information and comes to own the data they generated and evaluated. The benefits to using GLA are numerous, including (a) hidden agendas and problematic issues are often revealed, (b) stakeholders are responsible for honesty in presentation and validity of data, (c) phenomena under investigation are shaped by the interaction of multiple purposes and agendas, (d) and stakeholders are more likely to “buy into” data they generate and evaluate, therefore they are more ready to act on the outcomes of the process.

Depending on the purpose of the evaluation, the breadth of research questions, and the size of the stakeholder group/community, GLA can be used as a stand-alone session, or several GLAs can be combined to address the particular issues. For instance, an evaluation of a regional program might require 1–2 GLAs conducted in multiple counties in order to hear from a wide range of stakeholders and to ensure saturation of themes across the GLAs. If several GLAs are conducted, we combine and condense final themes from each individual GLA and present primary themes and secondary themes based on frequency of occurrence.

GLA Characteristics, Benefits, and Limitations

We have frequently used GLA in a wide range of settings and types of groups for a variety of evaluation-related purposes (see Table 2). In each of our examples, the GLA has produced meaningful qualitative results and numerous benefits including enhanced participant involvement and awareness, confirmation of mutual values, and increased commitment to future action (Vaughn, & Lohmueller, 1998; Vaughn, Jacquez, Zhao, & Lang, 2011). One particularly successful GLA was conducted with 63 stakeholders of a large, urban county’s foster care system. The goal of the GLA was to surface the needs of the foster youth as they transition to adulthood and evaluate the barriers within the foster care system. Stakeholders included foster youth, foster parents, court advocates, judges, researchers, caseworkers, and social service providers. Foster youth received a small gift card for their attendance at the GLA, but the remaining participants were not incentivized. Steps 1 through 6 of the GLA were conducted during an initial 3-hr session and five major themes/priorities were identified by the participants. Stakeholders were invited to return to a second session to work in action planning teams based on the five priorities identified in the first session. The action teams have continued to meet separately over the last year and continue to be involved in various projects and initiatives that address each of the themes/priorities.

Table 2.

Examples of GLA Settings, Types of Groups, and Purposes.

Group Composition	Purpose	Setting	Particulars
∼20 minority men	Empowerment evaluation; program planning and formative evaluation	Part of a regular support group meeting at a church	Diversity of group (i.e., literacy, poverty)
80+ students and faculty of one university department	Assessment and ongoing evaluation; evaluation of mission/vision alignment	University large meeting room	Conducted five consecutive years
21 person leadership team at a for-profit organization	Evaluate current state of leadership practices	Conference room on-site	Two consecutive 2-hr sessions, 2 weeks apart
18 patient services professionals	Evaluate the action plan they put in place 12 months ago to determine progress	Small conference room on their patient floor	Classic method utilized over 3 hr
∼15 doctoral students in one class	Mid-semester evaluation	Online using adobe connect	Nuances of using online
∼50 members of teams from Fortune 500 Corporation	Empowerment evaluation	Various large meeting rooms	Various cross functional teams involved
100+ medical professionals and parents	Intervention planning and ongoing empowerment evaluation	Part of a conference held at a hotel	Group split into two conference rooms for Steps 1–5; Steps 6 and 7 in large group
50+ stakeholders from all levels of the foster care system	Needs assessment	Reserved community meeting room	Wide variety of stakeholders participating
35 stakeholders associated with an academic–community partnership group	Evaluation and research for grant report	Medical center meeting room	Some participants had cognitive, physical, and/or developmental disabilities
68 minority and immigrant youth Kindergarten–eighth grade	Needs assessment	School library	Conducted separate GLAs by grade level; included small group facilitators
15–30 service providers from nine different counties (total = 200+)	Health needs assessment	Various large meeting rooms	Separate GLAs by county
Physicians and nurses	Program planning and ongoing evaluation	Medical center meeting room	Group had bias toward quantitative data so more time spent explaining purpose and benefits of GLA
100+ middle school students	Needs assessment	School classrooms	Separate GLAs conducted by classroom; Steps 5–7 scheduled at different time
All members (∼100) of a clinical research organization	Intervention planning and formative evaluation	Open board room and meeting rooms	Extended GLA conducted over several meetings

Note. GLA = group-level assessment.

Social Nature of Interactions and Talk Within GLA

The GLA supports a social constructionist view within evaluation which recognizes the social nature of interactions namely that opinions and ideas are collaboratively constructed as individuals relate to others socially (Gergen, 1985; Smithson, 2000). Participants in a GLA context collaboratively discuss data and then interactively analyze and prioritize the data; this process emphasizes the co-construction of meaning behind the ideas that participants share. In more individualistic forms of research and evaluation like one-on-one interviews, the social nature of talk and interaction is often ignored or glossed over (Wilkinson, 1998).

Collaboration, Action, and Buy-In

In order to facilitate lasting change in any group, evaluators need to find a way to truly collaborate with diverse stakeholders all of whom provide much-needed perspectives on contextual relevance and feasibility. By getting a large number of diverse participants to the metaphorical table in GLA, there is more likely to be buy-in and collective responsibility from all sides for future directions (Israel et al., 2010; Monroe, Oxarart, Mcdonell, & Plate, 2009; Wallerstein, Yen, & Syme, 2011). Because the GLA is focused on “end users,” there is emphasis on action in the interpretation, dissemination, and use of the data (Brannen, 2005).

Participant-Driven Process

The GLA allows the opportunity for equitable input by all relevant parties who have the knowledge and expertise to inform the process and develop relevant action plans toward future programming, interventions, and curricula. The group of stakeholders rather than an individual “expert” is responsible for data generation and analysis. Compared to other evaluation methods that are more evaluator-centric, the GLA is participant-centric encouraging interaction among participants. Participants are motivated to be involved because they are engaged in the data generation and analysis that will drive their own project forward. Through a collaborative process, all levels of stakeholders can work together to understand a complicated issue while considering the full social and cultural context of the issue. In an attempt to make the data credible, the group directs the content and the evaluator facilitates the process (Vaughn, & Lohmueller, 1998).

Multifaceted Data Analysis

The GLA allows for multiple levels of data generation and participant-driven data analysis. In response to the various prompts, individual-level qualitative data are generated and reflected upon by each participant in Steps 2 through 4 of the GLA as they interact within a social context. Then in Step 5, small groups of participants analyze, distill, and summarize themes from the wall charts. In Step 6, the large group as a whole identifies, discusses, and prioritizes themes. As such, the GLA process honors the messiness and complexity of attempting to capture simultaneously individual and social level data (Beckett & Clegg, 2007; Clark, Holland, Katz, & Peace, 2009).

Group-Owned, “Public” Data

The GLA is unique because (1) it uses the verbatim words and themes of the participants co-constructed in real-time interaction versus the evaluator reporting back the participant words and interchanges devoid of the social context; and (2) the generation of data is somewhat anonymous (not clear who wrote what on the wall charts) yet issues that are most important to a group will be endorsed by many (through checkmarks or stars by the written ideas) or repeated across multiple wall charts. Thus, the common ideas that are most pressing to the group as a whole are likely to be represented as themes and priorities. The psychology underlying GLA is that only the issues that a group is ready and able to address as a group will be endorsed and prioritized. It is possible that individual opinions written on wall charts within a GLA may not be incorporated into final themes or priorities by the group. However, as compared to individualized opinions shared privately with an interviewer for example, the intent of GLA is for a large group to have a common, publicly acknowledged database from which participants can analyze, prioritize, and decide to act upon their own data. The structured steps of the GLA allow for a multitude of deeply important issues to be raised by individuals without having one individual dominate the whole group or GLA process with his or her ideas.

Versatility and Flexibility

GLA easily works as an evaluation method across a variety of diverse groups, allowing trust to build quickly. Because of the variety of communication methods (i.e., verbal, written, drawing, and auditory) and formats (i.e., individual, small group, and large group) used in the GLA process, common participant barriers are diminished (e.g., participants with low literacy, shyness, verboseness, different ages, different educational levels, etc.). Thus, inherent in the design of the GLA is the “reflexive, flexible and iterative” process (Cornwall & Jewkes, 1995, p. 1668). The versatility and flexibility of the process especially within Steps 2, 5, 6, and 7 recognize the importance of getting all stakeholders in the room interacting to generate, analyze, and prioritize data. For instance, in Step 2, wall chart prompts can contain more pictures and simple words if children or people with cognitive disabilities are participating. In terms of accessibility, wall charts can be placed lower to the ground if people with wheelchairs are participating. In one of our projects, participants who were developmentally disabled brought aides to help them understand the prompts and write responses on the wall charts. In Step 5, there is flexibility with how the small groups are organized regarding whether a facilitator is assigned to each group or whether the small group facilitates itself in the identification and recording of themes. Steps 6 and 7 can be modified to reflect the specific goals and desired outcomes of the project. As stated earlier, Step 7 can be scheduled at a different time and only include those participants who are motivated to continue with action planning and next steps.

Limitations and Bias in GLA

Like any method used in evaluation, GLA is not without limitations and bias. In particular, GLA is designed for large groups, and large groups can quickly become unwieldy and hard to manage without a clear understanding of and adherence to the methodical steps of GLA. GLA is participant-driven, which may be uncomfortable or unfamiliar to certain groups who would prefer to have less responsibility by having an outside evaluator collect and analyze data for the group. Although some individuals within a GLA session may withhold information from the wall charts for a variety of reasons, the purpose of GLA is to have the large group distill, discuss, and prioritize data that they have consciously chosen to share publicly. Finally, we have found that some groups have difficulty understanding exactly what constitutes a theme. Therefore, during Step 5, some participants attempt to find a main idea on each assigned chart versus across their assigned wall charts. To address this confusion about development of themes, we usually clearly explain the instructions for Step 5 in multiple ways using several relevant examples.

Conclusion

GLA offers professionals a useful tool for a wide range of evaluation purposes particularly in projects requiring formative evaluation or self-determination of participants when using an empowerment evaluation approach. Compared to traditional qualitative methods, GLA is a participatory methodology directly involving a large group of stakeholders in the data collection, data analysis, data review, and action planning. Collaborating actively with participants in GLA rather than conducting research and evaluation on or about them offers an opportunity for evaluators to partner with large groups within communities and organizations. The GLA method is designed to acquire timely and relevant data that can be used by participants themselves in future endeavors.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

American Evaluation Association. (2014). What is evaluation? Retrieved March 25, 2014, from http://www.eval.org/p/bl/et/blogid=2&blogaid=4

Beckett

Clegg

(2007). Qualitative data from a postal questionnaire: Questioning the presumption of the value of presence [Article]. International Journal of Social Research Methodology, 10, 307–317. doi:10.1080/13645570701401214

Beyea

S. C.

Nicoll

L. H.

(2000). Learn more using focus groups. AORN Journal, 71, 897–900. doi:https://dx-doi-org.web.bisu.edu.cn/10.1016/S0001-2092(06)62276-X

Brannen

(2005). Mixing methods: The entry of qualitative and quantitative approaches into the research process. International Journal of Social Research Methodology, 8, 173–184. doi:10.1080/13645570500154642

Britten

(1995). Qualitative interviews in medical research. British Medical Journal, 311, 251–253.

Brown

Isaacs

(2005). The world café: Shaping our futures through conversations that matter. San Francisco, CA: Berrett-Koehler Publishers.

Campbell

(2002). A critical appraisal of participatory methods in development research. International Journal of Social Research Methodology, 5(1), 19–29.

Clark

Holland

Katz

Peace

(2009). Learning to see: Lessons from a participatory observation research project in public spaces. International Journal of Social Research Methodology, 12, 345–360. doi:10.1080/13645570802268587

Condon

P. M.

(2007). Design charrettes for sustainable communities. Washington, DC: Island Press.

10.

Cornwall

Jewkes

(1995). What is participatory research? Social Science & Medicine, 41, 1667–1676. doi:https://dx-doi-org.web.bisu.edu.cn/10.1016/0277-9536(95)00127-S

11.

Draper

Swift

J. A.

(2011). Qualitative research in nutrition and dietetics: Data collection issues. Journal of Human Nutrition and Dietetics, 24, 3–12. doi:10.1111/j.1365-277X.2010.01117.x

12.

Emery

Purser

R. E.

(1996). The search conference: A powerful method for planning organizational change and community action. San Francisco, CA: Jossey-Bass.

13.

Fetterman

D. M.

(1994). Empowerment evaluation. Evaluation practice, 15, 1–15.

14.

Gergen

K. J.

(1985). The social constructionist movement in modern psychology. American Psychologist, 40, 266–275. doi:10.1037/0003-066x.40.3.266

15.

Goodman

L. A.

(1961). Snowball sampling. The Annals of Mathematical Statistics, 32, 148–170.

16.

Hickey

Mohan

(2005). Relocating participation within a radical politics of development. Development and Change, 36, 237–262. doi:10.1111/j.0012-155X.2005.00410.x

17.

Israel

B. A.

Coombe

C. M.

Cheezum

R. R.

Schulz

A. J.

McGranaghan

R. J.

Lichtenstein

… Burris

(2010). Community-based participatory research: A capacity-building approach for policy advocacy aimed at eliminating health disparities. American Journal of Public Health, 100, 2094–2102. doi:10.2105/ajph.2009.170506

18.

Kitzinger

(1995). Qualitative research: Introducing focus groups. BMJ, 311, 299–302. doi:10.1136/bmj.311.7000.299

19.

Lister

Mitchell

Sloper

Roberts

(2003). Participation and partnerships in research: Listening to the ideas and experiences of a parent-carer. International Journal of Social Research Methodology, 6, 159. doi:10.1080/13645570110118692

20.

Mansell

Bennett

Northway

Mead

Moseley

(2004). The learning curve: The advantages and disadvantages in the use of focus groups as a method of data collection. Nurse Researcher, 11, 79–88.

21.

Monroe

M. C.

Oxarart

Mcdonell

Plate

(2009). Using community forums to enhance public engagement in environmental issues. Journal of Education for Sustainable Development, 3, 171–182. doi:10.1177/097340820900300212

22.

Morgan

(1998). The focus group guidebook (Vol. 1). Thousand Oaks, CA: Sage.

23.

Pain

(2004). Social geography: Participatory research [Article]. Progress in Human Geography, 28, 652–663. doi:10.1191/0309132504ph511pr

24.

Powell

R. A.

Single

H. M.

(1996). Focus groups. International Journal for Quality in Health Care, 8, 499–504. doi:10.1093/intqhc/8.5.499

25.

Reddy

W. B.

(1996). Group-level team assessment: A 10-step sequence to a committed team. San Diego, CA: Pfeiffer.

26.

Schein

E. H.

(1969). Process consultation: Its role in organization development. Reading, MA: Addison-Wesley.

27.

Smithson

(2000). Using and analysing focus groups: Limitations and possibilities. International Journal of Social Research Methodology, 3, 103–119. doi:10.1080/136455700405172

28.

Steil

Jr Watlington

(2007). Community Summits. In Holman

Devane

Cady

(Eds.), The change handbook: The definitive resource on today’s best methods for engaging whole systems, (2nd ed., pp. 365–369). San Francisco, CA: Berrett-Koehler.

29.

Vaughn

L. M.

Lohmueller

(1998). Using the group level assessment in a support group setting. Organization Development Journal, 16, 99–99.

30.

Vaughn

L. M.

Jacquez

Zhao

Lang

(2011). Partnering with students to explore the health needs of an ethnically diverse, low-resource school: An innovative large group assessment approach. Family & Community Health, 34, 72–84.

31.

Wallerstein

N. B.

Yen

I. H.

Syme

S. L.

(2011). Integration of social epidemiology and community-engaged interventions to improve health equity. American Journal of Public Health, 101, 822–830.

32.

Weisbord

M. R.

(1987). Productive workplaces: Organizing and managing for dignity, meaning, and community (1st ed.). San Francisco, CA: Jossey-Bass.

33.

Weisbord

M. R.

Weisbord

Janoff

(2000). Future search: An action guide to finding common ground in organizations and communities. San Francisco, CA: Berrett-Koehler.

34.

Wilkinson

(1998). Focus groups in feminist research: Power, interaction, and the co-construction of meaning. Women's Studies International Forum, 21, 111–125.

35.

Willis

Green

Daly

Williamson

Bandyopadhyay

(2009). Perils and possibilities: Achieving best evidence from focus groups in public health research. Australian and New Zealand Journal of Public Health, 33, 131–136. doi:10.1111/j.1753-6405.2009.00358.x