“Expectations to Change” (E2C)

The E2C Process

The E2C process is a six-step, interactive, workshop-based method used to guide stakeholders from establishing performance standards (i.e., expectations) to formulating action steps toward desired programmatic change. The process can be completed in one or more working sessions with those evaluation stakeholders best positioned to put the evaluation findings to use within their organization (i.e., the “primary intended users”; Patton, 2008). Each of the six steps is described in detail below. While the specific activities used to carry out each step should be tailored to the evaluation setting, we provide example activities based on various implementations of the process to date.

Step 6: Plan for Change

The objective of this step is for stakeholders to formulate an action plan for implementing their recommendations. Evaluators should select an activity that enlists all of the stakeholders and result in concrete next steps. One approach is to use a combination of sticky wall activities (Michigan State University Extension, 2010) and small group work to accomplish the objective. Transfer the final set of recommendations onto large sticky notes and post the sticky notes randomly across a wall. Provide each stakeholder with one red sticky dot and a set of green sticky dots (the exact number of green dots depends on the size of the group and the number of recommendations the group has the capacity to take action on). Introduce the red dot as their “passion dot” and ask the stakeholders to approach the wall and place their passion dot on the recommendation toward which they are willing to dedicate leadership energy. Next, introduce the green dots as “helper dots” and ask the stakeholders to place those dots on the recommendations toward which they are willing to direct helper energy. The red and green dots represent the team that will work on the corresponding recommendation. Place any recommendation without a team off to the side and explain that a list will be retained so they can attend to those recommendations in the future.

Once the teams have been formed, give each an opportunity to come together to talk more about their recommendation, generate some tangible next steps, and identify a date for their next meeting. Then, adhere a timeline with designated increments (e.g., 1 month, 3 months, 6 months, 9 months, and 1 year) to the wall and ask the team leaders to report on their team’s next steps and then place their recommendation on the timeline based on when their team commits to having it completed. During the reporting, have a stakeholder document the recommendations (including those without a team), team leaders and helpers, next steps, and timeline.

As shown in Figure 1, after the plan for change step, the process cycles back to “set expectations.” The E2C process is designed to support continuous program improvement. Once the plan for change has been implemented, the stakeholders can set new expectations, collect data, and then work through the process again, either with a facilitator or on their own.

OPEN IN VIEWER

Figure 1.

Expectations to Change (E2C) Process.

Implementing the E2C Process: Considerations

There are several important considerations when implementing the E2C process. The first consideration is who to include. To make this decision, the evaluator, in consultation with their client, should consider who within or outside the organization is best positioned to put the evaluation findings to use. In other words, the goal is to include individuals who bring diverse perspectives and can translate the findings into action to improve the operation of the program. This may include, for example, direct services staff, administrators, funders, or service recipients. Depending on the setting, it may or may not be advisable to assemble a heterogeneous group. Varying levels of authority within a group can create a dynamic where individuals with more power dominate the process while those with less power feel uncomfortable sharing their opinions and ideas. In some situations, the influence of authority imbalances among a group can be addressed by establishing ground rules that encourage full, equal participation and voice. Alternatively, it may be necessary to select one key group with whom to move forward or hold separate sessions for each unique stakeholder group.

A second consideration is the amount of time and number of working sessions needed. There is no set amount of time required for the E2C process; the process can be tailored to fit any evaluation context. We have facilitated the process with a group as small as five with a set of findings for nine quantitative indicators in one 1.5-hr session and with a group as large as 15 with multiple evaluation questions, each with a substantial number of findings in three, 3-hr sessions (9 hr total). The total amount of time needed depends on the number of findings that are going to be processed in the workshop and the estimated amount of time needed to work through those findings with the planned activities. It is helpful to start by identifying the activities that are going to be used to carry out each step of the E2C process, estimating the amount of time needed for each step and then calculating the total time based on those estimates, also accounting for time required for introductions/background and closing activities. If the organization cannot dedicate the amount of time required, adjust the plan by further limiting the number of key findings that will be processed during Steps 4–6.

In addition to the total amount of time, the number of working sessions requires consideration. It may be easier for some groups to come together for a full-day workshop, while others may prefer to break up the work into multiple, shorter sessions. Either plan can be accommodated. If multiple, shorter sessions are preferred, efforts will need to be made to retain participants across all sessions. Gaining participants’ commitment to attend all sessions and spacing the sessions no more than 1 week apart may aid in preventing scheduling conflicts and staff turnover from significantly effecting group membership.

Space is a third consideration when implementing the E2C process. Ideally, the workshop sessions should help in a space away from the organization’s daily activities to minimize distractions and interruptions. The room should have tables and chairs that can be arranged in a semicircle or U-shape so that participants can easily engage with one another. The room must be large enough to accommodate an instruction area in front of the tables and space for each of the planned interactive activities, such as rotating flip charts. The room will also need adequate wall space for hanging posters and other materials needed for activities, such as the sticky wall.

A final consideration to note when implementing the E2C process is the skills and roles of the facilitators. An E2C facilitator needs strong interpersonal and communication skills. They must be able to lead activities, teach, and act as technical consultant. Further, they need the skills to encourage participants’ full, equal participation, manage disruptive behavior, and address technical issues that arise such as the misinterpretation of the findings. These are essential skills of effective “interactive evaluation practice” (King & Stevahn, 2013) that can be strengthened through training and experience.

Implementing the E2C Process: A Case Study

The E2C process was originally developed for use in a participatory evaluation with a large domestic violence shelter program. This evaluation began during the third year of a 5-year initiative to build the organization’s research and evaluation capacity. The impetus for the evaluation of the shelter program was an administrator’s concern about the extent to which services were being delivered in accordance with the organization’s empowerment philosophy. At the outset of the project, a work group consisting of the evaluators, key program administrators, and select direct service staff was formed. The work group’s aim was to determine the evaluation questions, design the data collection instruments, make decisions about the data collection procedures, and oversee the finding’s interpretation and reporting process. Following data collection, the work group discussed options for interpreting and reporting the findings and elected to engage all of the direct service staff who would ultimately use the evaluation findings to improve service delivery. With guidance from the evaluation practice literature and a process facilitation expert (third author), the evaluators developed the E2C process and implemented it in three, 3-hr workshop sessions with 13 direct service staff and 2 program administrators.

The workshop sessions were held at the local public library to ensure adequate space for activities as well as provide an environment free from work-related distractions and interruptions. Tables were arranged in a U-shape so that participants could see one another and the facilitators could easily lead the group. At the front of the room, two large posters were displayed, one with the evaluation questions and one with the E2C diagram (Figure 1). The workshop began with a review of the evaluation purpose, questions, and methods, as well as an introduction to each step of the E2C process, referencing the posters displayed on the wall. We then established ground rules to help facilitate effective group dynamics and manage behaviors that hinder productive group work (Michigan State University Extension, 2010). Our ground rules included participate fully, everyone has an equal voice, keep an open mind, respect each other, one person at a time, and accept the data as valid. The purpose of the last ground rule was to minimize participants’ desire to explain away negative findings by finding fault with the study methodology or questioning respondents’ (i.e., program recipients’) motives or perceptions.

Once the ground rules were agreed upon, each stakeholder was provided with a binder that contained a printed copy of the E2C diagram, worksheets, and the data organized by evaluation question. To maximize readability and build capacity, frequency data were presented in tables structured to replicate the format of the reports generated by the organization’s client information system. Analysis of variance and correlation results were displayed in summary tables and graphs. Qualitative data were provided in raw format for the purpose of enriching stakeholders’ understanding of the quantitative data.

After orienting stakeholders to the materials contained in their binders, we started the E2C process. At each step, we explained the rationale and objective of the step and then guided them through the planned activities. During the first session, we worked through the first three steps for each evaluation question. For Step 1, set expectations, stakeholders were guided through the completion of the expectations statements listed on worksheets in their binders, and a facilitated group discussion was used to reach consensus. Step 2, review findings, began with a lesson on how to read the data as presented and then stakeholders were given ample time to digest and record their reactions to the findings before sharing and discussing their thoughts with the larger group. During the large group discussion, attention was drawn to findings that exceeded expectations and stakeholders were encouraged to reflect on the meaning and implications of the positive findings. To identify key findings (Step 3), stakeholders were asked to come up to a flip chart pad at the front of the room and write the three to five findings that had not met their expectations and that they felt required the groups’ immediate attention. To prioritize the generated list of findings, each person was then given three sticky dots (one red, one blue, and one green) and directed to read the entire list and place their red dot on the finding they believed needed the most urgent attention, their blue dot on the second most urgent finding, and their green dot on the third most important urgent finding.

Between the first and second session, the evaluators ranked the findings for each evaluation question based on the stakeholders’ sticky dot voting. Then, at the second session, the group was provided with a summary of their rankings organized by evaluation questions. Starting with the key findings identified for the first evaluation question, stakeholders were guiding through a discussion of what the findings meant and their interpretations were documented on flip chart paper (Step 4). Next, the large group was divided into smaller groups and each small group was assigned a finding for which they brainstormed recommendations for change and then presented those recommendations to the large group for discussion (Step 5). Finally, the full list of recommendations was posted and the large group completed a worksheet detailing plans to put the recommendations into action (Step 6). For each recommendation, a team was identified and concrete next steps were discussed. Steps 4–6 were repeated for each evaluation question over the course of the second and third workshop sessions.

Evidence of Evaluation Use Following the E2C Process

One year after the E2C workshop, the 12 workshop participants still working at the organization (of the original 15) were asked to complete an online survey, administered via Survey Monkey, to assess the extent to which the evaluation had been used. Participation was voluntary, no financial compensation was provided, and informed consent was obtained prior to entering the survey. The survey consisted of six open-ended questions written to assess conceptual use, instrumental use, and process use. For example, participants were asked the following questions, “how did the [evaluation] affect the way you think about domestic violence, clients’ needs, or the work you do at [the organization]?” “What changes have been made in response to the findings from the [evaluation]?” and “By participating in the workshop did you learn new things about program evaluation that you did not know before? If so, what?” All 12 staff members completed the survey, resulting in a 100% response rate.

To analyze the qualitative data, we utilized a thematic analysis described by Braun and Clare (2006). First, we (the first and second authors) independently read the responses to familiarize ourselves with the data. We generated a list of initial codes separately and then came together to develop a coding scheme based on agreed upon codes. We recoded the entire data set based on the coding scheme and developed themes. Finally, themes were reviewed, defined, and named through a shared iterative process between coders.

Overall, the survey findings showed that the evaluation findings were used by the service providers in the year after the workshop. As shown in Table 1, a majority of the stakeholders identified conceptual, instrumental, and process uses of the evaluation. Conceptual use took the form of improved understanding or modification of thinking about aspects of the program. Eighty-three percent (n = 10) of stakeholders described how the findings increased their knowledge and awareness of clients’ experiences receiving services from the shelter program. “It gave me more information about what survivors think and gain/don’t gain from the services we provide. This gave me a more informed lens for direct service and program planning” (Rachel ¹ ). Additionally, six stakeholders discussed how the evaluation revealed the programmatic weaknesses and areas that they needed to improve when delivering services to survivors. The E2C process helped stakeholders see the gap between current and desired state of service delivery.

OPEN IN VIEWER

Table 1.

Evidence of Evaluation Use.

Type of Evaluation Use	Evidence
Conceptual	83% discussed increased knowledge and awareness of clients’ experiences
	50% gained insight about programmatic weaknesses and areas needing improvement
	42% discussed increased understanding of theory underlying service delivery and how to put that theory into practice
Instrumental	54% described using the findings to add or improve components to the program
	54% described how the findings lead them to change the way they interact with clients
Process	75% discussed the value of evaluation data
	58% discussed learning specific skills such as collecting data systematically and interpreting findings

I have learned that while we are an agency that provides voice to those who otherwise don’t have one within the community; we sometimes create an environment within that causes survivors to not feel they could speak. That bothered me. (Michelle)

Forty-two percent of stakeholders stated that the evaluation increased their understanding about empowerment theory and provided guidance about how to utilize this theory when delivering services.

Instrumental uses were also identified by stakeholders in the form of organizational- and individual-level changes within the shelter. At the organizational level, over half (54%) of the stakeholders surveyed described using the findings to add or improve components to the shelter program. For example, stakeholders described posting a visual depiction of the dynamics of an abusive relationship or offering life skills workshops in order to strengthen educational components of the program. On an individual level, 54% described how the findings lead them to change the way they interact with clients. Participants discussed making themselves appear more available to clients by shifting their body language or making it a priority to introduce themselves to clients soon after they arrive at shelter. They also integrated the dynamics of domestic violence into the conversations that they have with women. The quote below typified the various organizational- and individual-level changes that were made following the E2C workshop:

Staff has been more aware of the literature and information that is put out in shelter about DV [domestic violence]. Large “Power and Control Wheels” have been posted in shelter. Staff has also been doing workshops to help residents with life skills. I personally now make sure to introduce myself to new residents as soon as they come into shelter. (Sophia)

In addition to conceptual and instrumental use, stakeholders described specific ways their participation in the E2C workshop contributed to process use. Their participation fostered service providers’ appreciation for, increased their knowledge of, and enhanced their ability to engage in evaluation activities. Stakeholders extensively discussed the value of evaluation data, and specifically, how beneficial data could be to providing perspectives that service providers had not always considered.

Yes, I did learn new things pertaining to how the clients were surveyed at the end of their shelter stay and feedback was created for the shelter staff. The program evaluation taught each of us ways that we can improve and things the clients would like to see from us more often. Also, we learned what has worked in the past and things we can continue to do to empower the women and families we serve. (Mary)

Seven stakeholders (58%) also discussed learning specific skills such as collecting data systematically and interpreting findings. Christine summarizes briefly, “Yes, how to interpret the findings instead of simply looking at the results.” Veronica said, “I try to be more consistent when collecting data.” Finally, Sherry offered, “Yes, I learned this really great way to do program evaluation where staff come up with some of the questions that are asked and they are involved in the process.”

Conclusion

The E2C process is designed to engage stakeholders with their evaluation findings as a means of promoting evaluation use and building evaluation capacity. The distinguishing feature of this process from other available frameworks is that it is uniquely suited for contexts in which the aim is to assess performance on a set of indicators by comparing actual performance to planned performance standards for the purpose of program improvement. In the E2C process, stakeholders are guided through establishing standards, comparing the actual results to those standards to identify areas for improvement, and then generating recommendation and concrete action steps to implement desired changes. At its core, E2C is a process of self-evaluation and the role of the evaluator is that of facilitator, teacher, and technical consultant.

There are two key limitations to the E2C process that should be considered. First, the process can be resource intensive. The use of the E2C process requires the continued engagement of the evaluator beyond a final written report and a willingness on the part of the stakeholder group to commit time, energy, and money to this activity. For the evaluator, planning the workshop and preparing materials can be time intensive. For instance, we have found that writing the facilitation plan and preparing materials for a full-day workshop can take up to 16 hr. Also, we have found it beneficial to have two facilitators—co-facilitating activities or one as the primary facilitator and one assisting—which introduces further time investment and cost. The stakeholders have to see the value in covering the cost of the evaluators’ time, materials, and the time of participating staff members. The organization may also have to consider how to manage service provision in the absence of their direct service staff. Finally, the organization or evaluator must have access to adequate space to accommodate the group and planned activities. In some cases, resource constraints may render this process impracticable.

A second limitation is that the E2C process was developed for use with quantitative data, and it is currently unclear how the process would be carried out with qualitative or mixed-method data. It is possible that deductive, theoretical or data-driven qualitative analytic techniques (i.e., deductive content analysis or theoretical thematic analysis) could align with the standards-based (i.e., hypothesis-driven) E2C process (Braun & Clarke, 2006; Elo & Kyngas, 2008). We are currently in the process of testing this assertion.

Limitations notwithstanding, the primary strength of the E2C process is that it provides evaluation practitioners with a clear set of steps and suggested activities to guide stakeholders from establishing performance standards to generating tangible action steps for putting the evaluation findings to use within their organization. While a formal written report alone can lead to evaluation use, the E2C workshop affords stakeholders the dedicated time and structure to collectively process the evaluative information and develop a plan for taking action. We have seen the evaluation come to life for stakeholders during the E2C process (Hwalek & DeSole, 2012; Hwalek & Williams, 2012); they expressed genuine interest in, excitement about, and commitment to understanding and using the information that came from the evaluation, and the energy generated during the process carried through as they put the evaluation to use within their organization. Future research on evaluation practices should explore the unique contribution of interactive processes such as this on evaluation use, above and beyond the influence of traditional reporting methods.

Another strength of the E2C process is that it can be adapted for use in a variety of evaluation settings. One setting in which we believe the process has particular utility is that of nonprofit evaluation. Most nonprofit organizations engage in process and outcome monitoring and evaluation activities to gauge program performance. Due to resource and capacity constraint, often the data are collected and reported to funders and other external stakeholders for the purpose of accountability but not fully harnessed for internal program improvement purposes. By providing direct service staff with the opportunity to systematically compare their evaluation results to agreed-upon performance standards, celebrate successes, and address weaknesses, the E2C process facilitates self-evaluation for the purpose of continuous program improvement.