Applying the Participatory Evaluation Measurement Instrument in the Evaluation of a Public School Program for Teachers

Abstract

Although most evaluators are familiar with participatory evaluation (PE), the ability to measure stakeholder participation in an evaluation remains challenging. Based on Cousins and Whitmore’s (1998) PE theoretical model, Daigneault and Jacob (2009, 2012, 2014) developed an instrument for measuring the degree to which an evaluation can be considered participatory. The present study used Daigneault and Jacob's Participatory Evaluation Measurement Instrument (PEMI) in the evaluation of a public school district's educator mentoring program. The evaluators examined the benefits and challenges of using the instrument in public school settings. This paper details the procedure of using PEMI to explore the level of stakeholder participation in the evaluation case and uses a process similar to the mini-ethnography method to reflect on the experiences of the evaluators through the procedure. Findings show that in addition to accurately capturing the perceived level of participation, PEMI may be utilized in the planning phases of evaluation.

Keywords

participatory evaluation participatory evaluation measurement instruments PEMI stakeholder participation evaluation theory evaluation practice

Stakeholder participation has gained popularity in the evaluation community since the mid-1970s (Mark, 2001; Mathison, 2004; Taut, 2008). The development of theoretical frameworks and practical applications has demonstrated that involving stakeholders enhances the acceptance and utilization of the evaluation results (Alkin et al., 1998; Alkin et al., 1979). In various approaches, including Participatory Evaluation (PE), Collaborative Evaluation, Democratic-Deliberative Evaluation, Empowerment Evaluation, and Utilization-Focused Evaluation, nonevaluative stakeholders (program stakeholders without formal evaluation training or explicit involvement in the evaluator team) play a central role in the evaluation process (Bryson et al., 2011; Cousins & Whitmore, 1998; Fetterman & Wandersman, 2005; Shea & Lewko, 1995). Because stakeholder involvement is essential in each of these evaluation approaches, the approaches are often referred to interchangeably, even though they have distinguishing characteristics (Daigneault & Jacob, 2009). To distinguish PE from other forms of stakeholder-centered evaluation, researchers developed the theoretical frameworks to identify different forms of PE and outline the PE process (Cousins & Earl, 1992; Cousins & Whitmore, 1998; Weaver & Cousins, 2004). PE is defined as a program evaluation “in which trained evaluators work in partnership with [program] stakeholders to produce evaluative knowledge” (Cousins & Chouinard, 2012, p. 10). Essentially, PE can be differentiated from other evaluation approaches based on the extent of stakeholder involvement, the diversity among stakeholders, and the level of evaluator control over the process (Cousins & Whitmore, 1998).

Most recently, evaluation researchers went further to conceptualize PE and adapt it to practical evaluation activity. Daigneault and Jacob (2009) developed a framework for measuring the degree to which an evaluation can be considered participatory. In 2012, they created and validated “a measurement instrument that fully covers the content of participation, namely the Participatory Evaluation Measurement Instrument” (PEMI) (Daigneault et al., 2012, p. 243). The instrument measures stakeholder participation levels in three dimensions of PE: (1) Extent of involvement, (2) diversity of participants, and (3) control of the evaluation. The instrument was further revalidated and revised based on the combined qualitative and quantitative data from the 2014 mixed-method study (Daigneault & Jacob, 2014). The PEMI enables researchers to score the level of stakeholder involvement in the three dimensions of PE and measure the degree of participation in a particular evaluation.

Since 2009, there have been only a few evaluation research studies that applied the framework or the PEMI to test the alignment between the PE practice and the evaluations of education and social services programs (Connors & Magilvy, 2011; Jacob & Desautels, 2013; Jacob et al., 2011; Pietiläinen, 2012) or to assess whether the evaluation of a new faculty onboarding program can be considered participatory (Saalman, 2020). Although these studies described detailed application procedures and reported successful results of the applications, none of them shared their experience with the use of the PEMI or prescribed their recommendations and warnings on the utilization of the instrument.

This mixed-method study examined the pertinence of using Daigneault and Jacob's (2009) PE measurement model and the PEMI (Daigneault et al., 2012; Daigneault & Jacob, 2014) to assess the alignment of the evaluation of a public school district mentoring program with the PE framework. In addition, to gain practical insight into the application process's benefits and challenges, the researchers engaged in reflective practice, common in ethnographic studies, regarding their experiences utilizing the PEMI. Specifically, the study addressed two questions:
To what extent did stakeholders participate in the evaluation of a public school district's new educator induction and mentoring program, according to Daigneault and Jacob's PEMI?

What are the benefits and challenges of using the PEMI in the evaluation of public school programs, such as teacher mentoring and induction programs?
The study starts with a review of the PE theoretical framework, Daigneault and Jacob's PE measurement model, and the PEMI instrument. Next, an overview of the evaluation process provides background into the examined case, followed by a discussion of the application of the model and instrument to measure the PE level of the evaluation case.

Theoretical Framework

PE invites program stakeholders to participate in several evaluation tasks ranging from identifying evaluation questions and methodology design, data collection and analysis, and recommendation formulation to reporting and disseminating results. The primary goal of stakeholder involvement is to bring together the varied experiences, perspectives, and values of multiple participant groups to ensure the program participants’ buy-in to the evaluation, utilization of the evaluation result, and implementation of program changes (Cousins & Whitmore, 1998; Stufflebeam & Coryn, 2014). There are two forms of PE. The most commonly used in the U.S. and Canada is the practical PE, which focuses on fostering the use of evaluation results by enhancing evaluation relevance and ownership. The other form of PE is the transformative PE, which arises from the evaluation works in developing countries and emphasizes participatory principles and actions toward democratic social changes (Cousins & Whitmore, 1998). This research case analyzed the evaluation of the practical PE framework.

All program evaluation requires some degree of stakeholder participation (Alkin et al., 1998; King et al., 2007). Cousins and Whitmore (1998) proposed a foundation framework to help distinguish PE from other forms of non-PE and classify various types of PE. The framework allows researchers to locate other forms of non-PE and types of PE in a continuum along each of three distinct dimensions: (1) stakeholder selection for participation, (2) depth of participation, and (3) control of the evaluation process. The stakeholder selection for participation dimension looks at the types of stakeholders involved in the evaluation, ranging from only selecting the primary participant to including all the program's legitimate groups. The depth of participation dimension describes the involvement of stakeholders in the evaluation process, from providing only consultation to taking part in all aspects of the evaluation. Finally, the control of the evaluation process dimension presents the range of evaluation technical decision-making from entirely in the hands of the evaluator to wholly controlled by the stakeholders. (Cousins & Whitmore, 1998). Since its creation, this framework continues to be used as the theoretical concept for PE research studies (Chouinard, 2013; Cullen & Coryn, 2011), the development of measurement tools (Daigneault & Jacob, 2009; Smits & Champagne, 2008), and PE evaluation research (Saalman, 2020; Shaw et al., 2016).

Participatory Evaluation Measurement Instrument

Daigneault and Jacob (2009) pointed out the limitations of Cousins and Whitmore's framework as helpful for theorists but providing practitioners with no clear, explicit threshold between PE and non-PE. They proposed the theoretical development in the conceptualization of participation (Daigneault & Jacob, 2009) and went further to develop a measurement instrument that allows evaluation researchers and practitioners to quantify the degree to which an evaluation is PE (Daigneault et al., 2012). They relied on the conceptualization theory and method developed by Gerring (1999) and Goertz (2006) along with Cousins’ and Whitmore's (1998) concept of the three dimensions of PE to design The PEMI. Cousins’ and Whitmore's three PE dimensions—depth of participation, stakeholder selection for participation, and control of the evaluation process were mapped to PEMI's three process dimensions—Extent of the Involvement, diversity of participants, and control of the evaluation. Daigneault and Jacob later validated the instrument twice with sufficiently large samples of evaluation cases (Daigneault et al., 2012; Daigneault & Jacob, 2014).

PEMI measures stakeholder participation in the three dimensions of PE, each on a 5-point ordinal scale ranging from 0.00 (no participation), 0.25 (limited participation), 0.50 (moderate participation), 0.75 (substantial participation), to 1.00 (full participation). The scoring for the first two dimensions operates through four dichotomous indicators. For the Extent of the Involvement dimension, the indicators measure the involvement or noninvolvement in four steps of the evaluation process. For the Diversity of Participants dimension, the indicators determined the presence or absence of four types of participating stakeholders. The last dimension is subjectively scored on the level of control of evaluators and participants. An evaluation must score equal to or greater than 0.25 on each dimension to qualify as PE (Daigneault et al., 2012). Finally, the evaluation's overall level of stakeholder participation is determined by the average score of the three dimensions (Daigneault & Jacob, 2014). The measurement of stakeholder participation has been found advantageous to assess the participatory implementation of program evaluations on teacher/faculty induction programs (Saalman, 2020). Hence, the research under consideration attempted to apply the measurement framework in exploring the involvement of the stakeholders in the evaluation of the new educator mentoring program in a public school district.

Overview of the Case

The evaluation case in this research is the Mentoring and Induction Program for New Teachers and Administrators (the Mentoring Program) in a public school district. The district is in a suburb of Greater Boston, serving about 5000 students and employing approximately 385 licensed educators and administrators. The stated goals of the Mentoring Program focus directly on providing support as new teachers and administrators transition to teaching roles that should contribute to the development and retention of high-quality educators. Under the oversight of the district's assistant superintendent, this program is administered through a committee (the Mentoring Committee) that includes representatives from district leadership, school leadership, and teachers. The program matches experienced teachers or administrators (mentors) with new educators or administrators (protégés) and provides structured activities to support the professional growth of mentors and protégés. Program activities, such as orientation events and workshops, offer opportunities for mentors and protégés to develop professional skills. In addition, the pairs are required to meet, observe each other, and discuss professional practice. Mentors and protégés complete an annual survey previously developed by the Mentoring Committee, indicating the usefulness of various program components.

While the Mentoring Program is based on guidelines from the state education department, its details and implementation are at the school district's discretion. This evaluation examined the effectiveness of this district's program. The evaluation team comprised three student evaluators pursuing a Ph.D. in Research and Evaluation in Education at the University of Massachusetts Lowell. During the spring semester of 2023, they requested and were granted permission to conduct a PE of the Mentoring Program as part of coursework in program evaluation. The evaluation focused on the program's development of important mentor and new educator skills and the retention of beginning educators. The primary reason for choosing the participatory approach for this evaluation was pragmatic. The committee members are representatives of all program stakeholders: Teachers, school administrators, and district administrators. Hence, their input was invaluable in providing an understanding of the Mentoring Program. Moreover, the Mentoring Committee oversees program execution, and its members are invested in maintaining an effective and efficient program. The involvement of the Mentoring Committee was considered instrumental in increasing the likelihood of implementing recommendations from the evaluation.

The evaluators engaged the Mentoring Committee as participants in the evaluation through a series of structured evaluation meetings. At the first meeting, the committee evaluated and revised the program's theory of action/logic model, which had been drafted by the evaluators based on program descriptions provided by the district. The Mentoring Committee was also asked to provide development feedback on new survey instruments developed specifically for this evaluation to be used later in the evaluation. Mentors and protégés completed the survey before the second structured evaluation meeting with the Mentoring Committee. At the second meeting, the Mentoring Committee engaged in the analysis of survey data, both qualitative and quantitative, to begin to arrive at evidence-based recommendations for strengthening the program. Committee members were also regularly reminded that their questions and comments about the evaluation were always welcome.

Recommendations from the evaluation focused on strengthening adherence to the theory of action and building activities that would increase educator retention. The program should prioritize matching mentors and protégés with similar work assignments and ensure the training opportunities are appropriate for each type of assignment. The Mentoring Program must work to continually grow and diversify its pool of trained mentors to support the variety of new staff entering the district while also ensuring the language of the program is inclusive of this diversity. Finally, the evaluators also recommended strengthening the implementation of the expectation that a team, including school administrators, the mentor, and other colleagues, actively support all new staff.

Methods

PEMI Application Procedure

Each of the evaluators in the team independently scored the evaluation case using the coding conventions for PEMI (Daigneault et al., 2012; Daigneault & Jacob, 2014) and plotted their scores on coding scheme tables (see Tables 1, 2, and 3). A final score was produced based on a discussion leading to consensus. Although no formal reflection protocol was employed during this discussion, each researcher shared their observations and reflections that contributed to their score on each dimension. As each researcher described their interpretation of the dimension components, the conversation converged to a shared understanding of each dimension and consensus on their ratings. The scoring procedure is detailed below.

Table 1.
Coding Scheme for Extent of Involvement Dimension.

Description: The number of tasks in which at least one nonevaluative stakeholder is involved

Criteria: Is any nonevaluative stakeholder involved in this step?
Evaluation questions and methodological design

Data collection and analysis

Judgments and recommendation formulation

Report and dissemination of evaluation findings

Score: Involvement of nonevaluators at each of the steps receives 0.25 points. The score of the dimension is the total score from 4 evaluation steps.
0.00 = No involvement

0.25 = Limited/weak involvement

0.50 = Moderate involvement

0.75 = Substantial/strong involvement

1.00 = Full involvement

Table 2.
Coding Scheme for Diversity of Participants Dimension.

Description: The number of types of nonevaluative stakeholders involved

Criteria: Is this type of stakeholder involved?
Policy-makers and decision-makers

Implementer/deliverers

Target populations and intended beneficiaries

Civil society and citizens (people and organizations with interest in the evaluation)

Score: Involvement of each nonevaluator type receives 0.25 points. The score of the dimension is the total score of 4 nonevaluator types.
0.00 = No diversity

0.25 = Limited/weak diversity

0.50 = Moderate diversity

0.75 = Substantial/strong diversity

1.00 = Full diversity

The Extent of Involvement measures non-evaluative stakeholder participation in the four evaluation steps. Each step is scored 0.25 points for the involvement or 0 points for the noninvolvement of nonevaluative stakeholders in the step. The total score for the Extent of Involvement dimension is computed by adding the score of each evaluation step. When at least one evaluation step involves a non-evaluative stakeholder, this signals the presence of PE (Table 1).

The Diversity of Participants measures the total number of stakeholder types involved in an evaluation. Each of the four categories is scored on whether that type of non-evaluative stakeholder is involved in the evaluation. If the type of stakeholder is involved in the evaluation, 0.25 points are scored, and 0 points are scored if the type of stakeholder is not involved in the evaluation. The total score for this dimension is calculated by adding each score of the four types of stakeholders. The inclusion of at least one type of stakeholder involvement in the evaluation can be considered PE (Table 2).

Control of the evaluation measures the level of control of the evaluation process that non-evaluative stakeholders have compared to the control that the evaluators have. The measurement score reflects the stakeholder control level at different moments of the evaluation process. There are two main indicators to measure control of the evaluation process. The control of technical decisions refers to the authority to determine the course of action in each evaluation task. The source of influence refers to indirect power that stakeholders can use to influence the evaluation process, including, but not limited to, expertise, money, values, and organizational norms. The stakeholders’ level of control and influence may vary throughout the evaluation stages. A clear-cut scoring rule is rather tricky, but the measurement procedure is intuitive and flexible (Daigneault & Jacob, 2009). The minimum 0.25 points score, where stakeholders hold at least a limited control of the evaluation, signals eligibility for categorization as PE (Table 3). Scores of the three dimensions are then combined to determine an overall level of participation by averaging the scores of the three dimensions (Daigneault & Jacob, 2014).

Table 3.
Coding Scheme for Control of the Evaluation Process Dimension.

Description: The way in which control is exercised at different moments of the evaluation process

Criteria:
Control is measured throughout the whole evaluation process.

Score:
0.00 = Exclusive control by evaluators and/or nonparticipating sponsors

0.25 = Limited/weak control by participants

0.50 = Shared control between participants and evaluator and/or nonparticipating sponsor

0.75 = Substantial control by participants

1.00 = Exclusive control by participants

Reflective Research Practice

To reflect on using PEMI to assess the PE dimensions of the evaluation case, researchers followed an approach similar to a mini-ethnographic case study, which combines ethnography and case study design to take advantage of the strengths and mitigate the weaknesses of each research approach (Dobbins et al., 2021; Fusch et al., 2017). Miniethnography is often used when research focuses on a narrow area of inquiry and has time constraints (Bayeck, 2023; Dobbins et al., 2021; Fusch et al., 2017). This approach has been used extensively in the health and education fields (Hegade et al., 2023; Hsieh et al., 2016; Uddin, 2021; Witt, 2019; Ziegler & Bozorgmehr, 2023; Zirkus, 2019). The current study investigated the nonevaluative stakeholder involvement in the evaluation case, which lasted for 2 months. Researchers implemented data collection methods, including observation, field notes, and reflective journals (Bayeck, 2023; Fusch et al., 2017). Informal field notes were maintained during the evaluation process of the case and the process of applying PEMI scoring to the case. Individually, evaluators kept reflective journals to capture personal feelings and statements about the scoring process, assessment of the PEMI usage, and challenges. Each of these data sources informed the scoring on the PEMI and subsequent discussion. Data analysis was conducted at the end of the evaluation and during the scoring process when evaluators compared notes and journals. Rather than organize data into themes, the team identified and adopted similar data points (observation or judgment) in each other's notes. Conflicting data was judged based on evidence from evaluation documents, evaluation meeting transcripts, and correspondent emails. The data analysis results in the agreement on the PEMI score of the evaluation and the common experience of the scoring process.

Results

Measurement of Stakeholder Participation

The scoring results for the evaluation of the Mentoring Program are presented in Table 4. The extent of involvement dimension received a score of 0.75, indicating the substantial involvement of nonevaluative stakeholders (Table 4). In the first structured evaluation meeting, the committee members reviewed and gave feedback on the theory of action/logic model and provided input to the development of the survey. This involvement of the committee earned 0.25 points for the first evaluation step (evaluation questions and methodological design). During the second structured evaluation meeting, the committee helped analyze responses from the evaluation survey, identify patterns, interpret the survey data, and suggest recommendations. Hence, the committee members’ participation was observed in the second (data collection and analysis) and the third (judgment and recommendation formulation) evaluation step. Each step received 0.25 points. In the fourth evaluation step (report and dissemination of evaluation findings), the committee was not involved in writing the final evaluation report, and the evaluation team provided the evaluation report to the committee and the assistant superintendent. Therefore, there were no points awarded to this step. The total score of this dimension was the sum of the scores in each step.

Table 4.
Scoring the Evaluation Case Using Participatory Evaluation Measurement Instrument (PEMI).

Extent of involvement dimension Diversity of participants dimension Control of the evaluation process dimension

Description
The number of tasks in which at least one nonevaluative stakeholder is involved Description
The number of types of nonevaluative stakeholders involved Description
The way in which control is exercised at different moments of the evaluation process

Criteria Score Criteria Score Criteria Score

Evaluation questions and methodological design 0.25 Policymakers and decision-makers 0.25 Participants’ control is measured throughout the whole evaluation process 0.25

Data collection and analysis 0.25 Implementer/deliverers 0.25

Judgments and recommendation formulation 0.25 Target populations and intended beneficiaries 0.25

Report and dissemination of evaluation findings 0.00 Civil society and citizens (people and organizations with interest in the evaluation) 0.00

Total score of the dimension 0.75 Total score of the dimension 0.75 Total score of the dimension 0.25

Scoring scheme0.00 = No involvement0.25 = Limited/weak involvement0.50 = Moderate involvement0.75 = Substantial/strong involvement1.00 = Full involvement Scoring scheme0.00 = No diversity0.25 = Limited/weak diversity0.50 = Moderate diversity0.75 = Substantial/strong diversity1.00 = Full diversity Scoring scheme0.00 = Exclusive control by evaluator and/or nonparticipating sponsor0.25 = Limited/weak control by participants0.50 = Shared control between participants and evaluator and/or nonparticipating sponsor0.75 = Substantial control by participants1.00 = Exclusive control by participants

Overall score of the evaluation: 0.58The overall score of the evaluation is the average of the scores of the three dimensions.

The Diversity of Participants dimension received a score of 0.75, which signified the strong diversity of nonevaluative stakeholders involved (Table 4). The diversity of roles among the committee members contributed positively to this dimension. The assistant superintendent was the chair of the committee, and her responsibility and authority allowed her to make and implement policies and decisions that affected the program. Hence, her involvement in the evaluation, categorized as a high-level policy and decision-maker type of stakeholder, earned 0.25 points for this dimension. Because the committee was charged with implementing and managing the program, all committee members were “implementers and deliverers” of the program. This type of stakeholder participation received 0.25 points. Additionally, the committee members who were teachers also served as mentors; hence, they could be categorized as potential beneficiaries (according to PEMI's rules). Building principals represented on the committee also benefited when new educators participated in the program. Having intended beneficiaries participate in the evaluation, an additional 0.25 points were added to this dimension. Finally, the evaluation did not involve any stakeholders external to the school district. The lack of participation of this type of stakeholder scored 0.00 points. The total score of this dimension was the sum of the scores for each type of stakeholder involved in the evaluation.

Finally, the control of the evaluation process dimension score was 0.25 points, corresponding to the program participants’ limited or weak control of the evaluation (Table 4). The evaluation team held dominant power over the technical evaluation aspects, including drafting a logic model, proposing evaluation questions, designing preliminary survey tools, facilitating data analysis, generating findings, and writing an evaluation report. The Mentoring Committee only assisted in finalizing the logic model, confirming the evaluation questions, and reviewing the survey questionnaires. The members then participated in the data analysis with the facilitation of the evaluation team. The assistant superintendent exerted minimum influence over how the evaluation was conducted by participating in the capacity of a member of the Mentoring Committee.

Adhering to the 2014 PEMI's coding and interpreting conventions, the evaluation of the public school district's new educator induction and mentoring program was considered as a PE since each of its participatory dimensions scored at least 0.25 points. The overall level of participation in this evaluation was 0.50 which was rounded down from 0.58 (as recommended by Daigneault & Jacob, 2014). The overall score put this program evaluation at the moderate level for PE (Table 4). Although the extent of involvement and diversity of participants dimensions reflected stronger levels of participation, the PEMI requires evaluators to make the final judgment based on the average of the three dimension scores.

Benefits and Challenges of PEMI Application

Regarding what can be learned from the PEMI application process to score the evaluation, each evaluator's field notes and reflections yield similar insights into the benefits and challenges of the instrument and the process. The use of PEMI has proven advantageous in understanding how well the PE was used in the evaluation of the mentoring program. The scoring of the Diversity of Participants and the Extent of Involvement dimensions were particularly helpful in explaining the efficient design of the evaluation. By breaking down and assigning a score for each type of stakeholder involved and each evaluation step with stakeholder involvement, the scoring confirmed the importance of the participation of the Mentoring Committee and their roles at each meeting where key evaluation steps took place. Each committee member played multiple roles; for example, the members who implemented the program were also mentors or the organization leader. Their combined extensive program knowledge, direct beneficial interest, and implementing authority allowed the evaluation team to move from finalizing the program theory of change/logic model and optimizing the evaluation surveys to analyzing collected data and arriving at evaluation recommendations within a short time frame and limited meeting opportunities.

At the advent of the project, the evaluation team had planned to share the evaluation tasks with the Mentoring Committee equally; however, the scoring of the last dimension showed that the evaluators controlled most technical aspects of the evaluation. Retrospectively scrutinizing the implementation process, the team realized that it had scaffolded the technical process around the two evaluation meetings with the Mentoring Committee. This allowed the team to take advantage of PE elements in the restricted timeline and exposure with the key stakeholders. The team's decision to use the existing annual program questionnaire, which all committee members were familiar with, as the basis for the evaluation survey tool enabled its members to effectively assess evaluation data and provide great insight into underlying issues that the data revealed. The evaluation team's heavy control of the evaluation process might be a better-fit strategy for this evaluation after all.

The primary challenge of using PEMI was mapping the tool's discrete criteria of each dimension to the rather complex stakeholder participation of a real-world evaluation. First, the multiple roles of each stakeholder in the program required some investigation to arrive at an accurate rating for the Diversity of Participants dimension. As mentioned earlier, individual committee members played more than one role in the program. Hence, the member could represent more than one category of stakeholders. Second, the head of the committee was the assistant superintendent, who was not involved in writing the evaluation report but could decide whether to disseminate the evaluation findings beyond the committee members. Differences in opinions regarding whether decision-making ability warranted the member's involvement in the dissemination task resulted in varied initial ratings of the extent of involvement dimension among the evaluation team. Third, the high ratings for the first two dimensions might signal a high rating for the control of the evaluation process dimension. As the team discussed the balance of control in the evaluation, which seemed to lean toward the evaluation team throughout the evaluation process, and accepted each evaluator's subjective judgment on the level of control at each evaluation task, the lowest rank for this dimension was finally selected. The lack of structure in rating this dimension required self-reflection and a review of process notes to arrive at a conclusion. This came as no surprise, as Daigneault and Jacob have warned about the intuitive and flexible nature of the rating on the control of the evaluation process dimension (Daigneault et al., 2012).

Although coding conventions for using PEMI are available, the actual evaluation case can be so complicated that simple scoring rules may even create more confusion among the users. The evaluation team chose to score the evaluation individually first and then consolidate the results. Instead, scoring the evaluation together might be helpful, allowing analysis of the case and development of a consensus understanding of each PEMI's dimension characteristics. As the analysis breaks down different types of stakeholders and charts out the activities under each evaluation step, a simple grid can capture three layers of data: stakeholder types, activities, and evaluation steps. Such a tool may help clarify the coders’ understanding of the case and visualize the three dimensions for easy scoring.

Discussion

PEMI was used to assess the degree of realization of PE in the evaluation of the Mentoring Program. The tool assisted the evaluators in deconstructing the necessary dimensions and components of PE and carefully rating each using established criteria. The current study shows that the evaluation of the Mentoring Program was considered a PE, and its overall level of stakeholder participation was moderate. The PEMI scoring result reflected the evaluation model selection and represented the intended level of stakeholder participation. From the start, evaluators intentionally selected a PE model for practical purposes by planning to involve critical stakeholders in most of the evaluation steps so evaluation results and recommendations would be more likely to be adopted and implemented. With limited time and resources, training the stakeholders in evaluation techniques and methodologies, which would have enabled them to control more evaluation processes, was not an option. The evaluation team prioritized controlling the technical decisions at each step over maximizing stakeholders’ involvement to ensure the quality and timeliness of the evaluation.

During the scoring process, team members had a rich discussion to clarify and resolve the differences in opinions and ratings of each dimension. Some preparation and agreement on the rating procedure among the evaluation team should occur early in the use of PEMI. Differences in the opinions on accounting for various components of the program evaluation, available data, and under or overestimating stakeholder interest in the evaluation may result in the different scoring of the PEMI criteria. Having these discussions early will help minimize disagreement, increase consensus, and facilitate reconciliation in rating the criteria for the specific domains. When using PEMI, it is worth considering the intent of the tool and how the dimensions’ scores are connected. The PE-qualification score level of each dimension and how the overall score is eventually determined by the average score across all three domains promote the idea that all dimensions and criteria are equally important. Increasing stakeholder participation from the lowest to highest levels must be done so that all criteria are equally considered.

PEMI has only been used to test evaluations for their realization of PE after the evaluations are completed (Connors & Magilvy, 2011; Jacob & Desautels, 2013; Pietiläinen, 2012; Saalman, 2020). Instead, the tool can also be useful in guiding the evaluators to plan for evaluation implementation so that it will align with the PE framework and/or the desired level of stakeholder participation. As evaluators identify and analyze each group of stakeholders, they can also assess different design options to include that stakeholder group in one or more evaluation steps. Technical knowledge in evaluation is the critical factor in deciding how much control one has over the evaluation process (Cousins & Whitmore, 1998). A plan for training stakeholders and/or customizing evaluation tasks to fit the skill levels may need to be put in place before the start of the evaluation to increase their participation. At the early stage of the evaluation, the tool can aid evaluators in mapping out and providing each stakeholder group with a clear role and responsibility in each evaluation step. This can empower the stakeholder to take charge and be more active in the evaluation process (Jacob et al., 2011). Furthermore, PEMI scoring results provide insight into how ongoing evaluation operations, such as internal evaluations, can be redesigned to increase stakeholder participation. Evaluators can develop baseline scores and establish targets for evaluation design options with different levels of stakeholder involvement, as demonstrated to some extent by Connors and Magilvy's (2011) evaluation scenarios.

PE is an arduous stakeholder- and evaluator-engaged process. The nature of this type of evaluation model coupled with the limited use of PEMI are likely factors that contribute to its limited published use cases. This study and previous use cases of the PEMI show that the tool can be used to positively influence the evaluation process. Future research may consider the impact PEMI can have on stakeholder participation when it is used during the planning stage or to redesign the evaluation process.

Description: The number of tasks in which at least one nonevaluative stakeholder is involved
Criteria: Is any nonevaluative stakeholder involved in this step? Evaluation questions and methodological design Data collection and analysis Judgments and recommendation formulation Report and dissemination of evaluation findings
Score: Involvement of nonevaluators at each of the steps receives 0.25 points. The score of the dimension is the total score from 4 evaluation steps. 0.00 = No involvement 0.25 = Limited/weak involvement 0.50 = Moderate involvement 0.75 = Substantial/strong involvement 1.00 = Full involvement

Description: The number of types of nonevaluative stakeholders involved
Criteria: Is this type of stakeholder involved? Policy-makers and decision-makers Implementer/deliverers Target populations and intended beneficiaries Civil society and citizens (people and organizations with interest in the evaluation)
Score: Involvement of each nonevaluator type receives 0.25 points. The score of the dimension is the total score of 4 nonevaluator types. 0.00 = No diversity 0.25 = Limited/weak diversity 0.50 = Moderate diversity 0.75 = Substantial/strong diversity 1.00 = Full diversity

Description: The way in which control is exercised at different moments of the evaluation process
Criteria: Control is measured throughout the whole evaluation process.
Score: 0.00 = Exclusive control by evaluators and/or nonparticipating sponsors 0.25 = Limited/weak control by participants 0.50 = Shared control between participants and evaluator and/or nonparticipating sponsor 0.75 = Substantial control by participants 1.00 = Exclusive control by participants

Extent of involvement dimension	Diversity of participants dimension	Control of the evaluation process dimension
Description The number of tasks in which at least one nonevaluative stakeholder is involved	Description The number of types of nonevaluative stakeholders involved	Description The way in which control is exercised at different moments of the evaluation process
Criteria	Score	Criteria	Score	Criteria	Score
Evaluation questions and methodological design	0.25	Policymakers and decision-makers	0.25	Participants’ control is measured throughout the whole evaluation process	0.25
Data collection and analysis	0.25	Implementer/deliverers	0.25
Judgments and recommendation formulation	0.25	Target populations and intended beneficiaries	0.25
Report and dissemination of evaluation findings	0.00	Civil society and citizens (people and organizations with interest in the evaluation)	0.00
Total score of the dimension	0.75	Total score of the dimension	0.75	Total score of the dimension	0.25
Scoring scheme0.00 = No involvement0.25 = Limited/weak involvement0.50 = Moderate involvement0.75 = Substantial/strong involvement1.00 = Full involvement	Scoring scheme0.00 = No diversity0.25 = Limited/weak diversity0.50 = Moderate diversity0.75 = Substantial/strong diversity1.00 = Full diversity	Scoring scheme0.00 = Exclusive control by evaluator and/or nonparticipating sponsor0.25 = Limited/weak control by participants0.50 = Shared control between participants and evaluator and/or nonparticipating sponsor0.75 = Substantial control by participants1.00 = Exclusive control by participants
Overall score of the evaluation: 0.58The overall score of the evaluation is the average of the scores of the three dimensions.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Ha Pho

References

Alkin

Hofstetter

(1998). Stakeholder concepts in program evaluation. Advances in Educational Productivity, 7(1998), 87–113.

Alkin

M. C.

Daillak

White

(1979). Using evaluations: Does evaluation make a difference?. Sage.

Bayeck

R. Y.

(2023). Is microethnography an ethnographic case study? And/or a mini-ethnographic case study? An analysis of the literature. International Journal of Qualitative Methods, 22, 160940692311720. https://doi.org/10.1177/16094069231172074

Bryson

J. M.

Patton

M. Q.

Bowman

R. A.

(2011). Working with evaluation stakeholders: A rationale, step-wise approach, and toolkit. Evaluation and Program Planning, 34(1), 1–12. https://doi.org/10.1016/j.evalprogplan.2010.07.001

Chouinard

J. A.

(2013). The case for participatory evaluation in an era of accountability. American Journal of Evaluation, 34(2), 237–253. https://doi.org/10.1177/1098214013478142

Connors

S. C.

Magilvy

J. K.

(2011). Assessing vital signs: Applying two participatory evaluation frameworks to the evaluation of a college of nursing. Evaluation and Program Planning, 34(2), 79–86. https://doi.org/10.1016/j.evalprogplan.2010.12.001

Cousins

J. B.

Chouinard

J. A.

(2012). Participatory evaluation up close: An integration of research-based knowledge. Information Age Press.

Cousins

J. B.

Earl

L. M.

(1992). The case for participatory evaluation. Educational Evaluation and Policy Analysis, 14(4), 397–418. https://doi.org/10.3102/01623737014004397

Cousins

J. B.

Whitmore

(1998). Framing participatory evaluation. New Directions for Evaluation, 1998(80), 5–23. https://doi.org/10.1002/ev.1114

10.

Cullen

Coryn

C. L.

(2011). Forms and functions of participatory evaluation in international development: A review of the empirical and theoretical literature. Journal of Multidisciplinary Evaluation, 7(16), 32–47. https://doi.org/10.56645/jmde.v7i16.288

11.

Daigneault

P.-M.

Jacob

(2009). Toward accurate measurement of participation: Rethinking the conceptualization and operationalization of participatory evaluation. American Journal of Evaluation, 30(3), 330–348. https://doi.org/10.1177/1098214009340580

12.

Daigneault

P.-M.

Jacob

(2014). Unexpected but most welcome: Mixed methods for the validation and revision of the participatory evaluation measurement instrument. Journal of Mixed Methods Research, 8(1), 6–24. https://doi.org/10.1177/1558689813486190

13.

Daigneault

P. M.

Jacob

Tremblay

(2012). Measuring stakeholder participation in evaluation: An empirical validation of the participatory evaluation measurement instrument (PEMI). Evaluation Review, 36(4), 243–271. https://doi.org/10.1177/0193841X12458103

14.

Dobbins

C. E.

Edgar

L. D.

Dooley

K. E.

(2021). Facilitating the scholarship of discovery: Using the mini-ethnographic case study design. Journal of Experiential Education, 44(4), 395–408. https://doi.org/10.1177/1053825921999685

15.

Fetterman

D. M.

Wandersman

(2005). Empowerment evaluation principles in practice. Guilford Press.

16.

Fusch

P. I.

Fusch

G. E.

Ness

L. R.

(2017). How to conduct a mini-ethnographic case study: A guide for novice researchers. The Qualitative Report, 22(3), 923–942.

17.

Gerring

(1999). What makes a concept good? A criterial framework for understanding concept formation in the social sciences. Polity, 31(3), 357–393. https://doi.org/10.2307/3235246

18.

Goertz

(2006). Social science concepts: A user's guide. Princeton University Press.

19.

Hegade

Aryan

O. P.

Netekal

Shettar

(2023). A Mini-ethnography study of teaching C programming to first-year non-circuit branch students. Journal of Engineering Education Transformations, 36(Special Issue 2), 504–511. https://doi.org/10.16920/jeet/2023/v36is2/23077

20.

Hsieh

J. G.

Hsu

Wang

Y. W.

(2016). An anthropological approach to teach and evaluate cultural competence in medical students—the application of mini-ethnography in medical history taking. Medical Education Online, 21(1), 32561. https://doi.org/10.3402/meo.v21.32561

21.

Jacob

Desautels

(2013). Evaluation of Aboriginal programs: What place is given to participation and cultural sensitivity? The International Indigenous Policy Journal, 4(2), 1–29. https://doi.org/10.18584/iipj.2013.4.2.1

22.

Jacob

Ouvrard

Bélanger

J.-F.

(2011). Participatory evaluation and process use within a social aid organization for at-risk families and youth. Evaluation and Program Planning, 34(2), 113–123. https://doi.org/10.1016/j.evalprogplan.2010.08.002

23.

King

J. A.

Cousins

J. B.

Whitmore

(2007). Making sense of participatory evaluation: Framing participatory evaluation. New Directions for Evaluation, 2007(114), 83–105. https://doi.org/10.1002/ev.226

24.

Mark

M. M.

(2001). Evaluation’s future: Furor, futile, or fertile? American Journal of Evaluation, 22(3), 457–479.

25.

Mathison

(2004). Encyclopedia of evaluation. Sage.

26.

Pietiläinen

(2012). Testing the participatory education evaluation concept in a national context. Studies in Educational Evaluation, 38(1), 9–14. https://doi.org/10.1016/j.stueduc.2012.02.003

27.

Saalman

D. R.

(2020). Objectives driven participatory evaluation model. Journal of Modern Applied Statistical Methods, 18(2), 14. https://doi.org/10.22237/jmasm/1604190900

28.

Shaw

Campbell

Hagstrom

O’Reilly

Kreiger

Cain

Nye

(2016). Bringing research into practice: An evaluation of Michigan’s sexual assault kit. Journal of Interpersonal Violence, 31(8), 1476–1500. https://doi.org/10.1177/0886260514567964

29.

Shea

M. P.

Lewko

J. H.

(1995). Use of a stakeholder advisory group to facilitate the utilization of evaluation results. The Canadian Journal of Program Evaluation, 10(1), 159–162. https://doi.org/10.3138/cjpe.010.012

30.

Smits

P. A.

Champagne

(2008). An assessment of the theoretical underpinnings of practical participatory evaluation. American Journal of Evaluation, 29(4), 427–442. https://doi.org/10.1177/1098214008325023

31.

Stufflebeam

D. L.

Coryn

C. L.

(2014). Evaluation theory, models, and applications (Vol. 50). John Wiley & Sons.

32.

Taut

(2008). What have we learned about stakeholder involvement in program evaluation? Studies in Educational Evaluation, 34(4), 224–230. https://doi.org/10.1016/j.stueduc.2008.10.007

33.

Uddin

M. S.

(2021). A mini ethnography on virtual elementary student teaching during the COVID-19 pandemic with a focus on the subject of mathematics (Doctoral dissertation. Morgan State University).

34.

Weaver

Cousins

J. B.

(2004). Unpacking the participatory process. Journal of Multidisciplinary Evaluation, 1(1), 19–40. https://doi.org/10.56645/jmde.v1i1.144

35.

Witt

H. R.

(2019). What is a healthy body? Asking for a friend. A mini ethnography about undergraduate women at Seattle university and their relationships with food and exercise. SUURJ: Seattle University Undergraduate Research Journal, 3(1), 19.

36.

Ziegler

Bozorgmehr

(2023). Mini-ethnography for medical students. Inquiry-based learning to foster transcultural competences. European Journal of Public Health, 33(Supplement_2), ckad160–804. https://doi.org/10.1093/eurpub/ckad160.804

37.

Zirkus

K. J.

(2019). Informal Interactions: A Mini-Ethnographic Case Study of Students with Emotional and Behavioral Disorders.