Exploring Alternative Approaches for Presenting Evaluation Results

Presenting Evaluation Results in “Alternative” Ways

Alternative presentations of results employ forms drawn primarily from the humanities and the arts, such as poetry, performance, narrative, and imagery as well as art and dance. The basic justification for these forms is their power to capture more completely and to evoke more powerfully the full dimensionality of human experience. Alternative presentations convey multiple voices, expand the range of relevant standpoints, and legitimize diverse experiences. Poetry, fiction, and drama offer opportunities that include, even foreground, the emotional, moral–ethical, psychological, and value strands of human action and its meanings. Dance and art offer parallel presentations that are explicitly interpretive and evocative; they require an audience, a viewer, an interpreter. As Paget has commented about performance as a text of presentation:

The multiple interpretive acts of performance enhance, rather than diminish, the intelligibility of the text as a scientific account because these multiple interpretations enhance our understanding of the complexity of the reality to which the text and the science of the text allude. (Paget, 1995, p. 241)

The Argument for Alternative Presentations

The idea of presenting evaluation results in alternative ways emerged from the constructivist traditions of social inquiry as part of the “interpretive turn” of the latter part of the last century (Schwandt, 2007). At the core of a constructivist worldview are the assumptions that the social world is different from the physical world and that what most importantly guides human action are the constructions of meaning that people make in interaction with others in particular contexts. That is, the social world is not fixed, stable, or predictable but is contextual, dynamic, and filled with multiple and ever-changing actions, interactions, and socially constructed meanings. Knowledge of this world does not come in the form of general abstract propositions but rather is itself a temporal and contextual construction of the knower.

These assumptions extend to social inquiry, such that constructivist inquirers are inevitably implicated in their knowledge claims. Social knowledge is inescapably constituted, in part, by the particular lenses of the inquirer, by the inquirer as knower. Extending this core notion to the ways in which we represent what we know from social inquiry, such representations are also constituted, in part, by the particular lenses of the inquirer, by the inquirer as author.

What kind of representations should we as social inquirers author? Abma (1997) recommends that we consider constructing “writerly” instead of “readerly” texts. Readerly texts are closed and authoritative. They are structured and predictable with clear directions on how to understand the text, inviting no interpretation, use, or transformation other than that intended by the author who remains masked and hidden by third-person language and passive voice, among other linguistic strategies (Abma, 1997). Writerly texts, in contrast, are open. They invite multiple interpretations; in fact, they insist that the “reader” actively participate in meaning making. And they unmask the author.

We find Abma’s vision of writerly texts appealing because of their dialogic and educative potential, and thus their congruence with our values-engaged, educative ambitions for evaluation. We further believe that alternative presentations are one legitimate form of writerly texts and that the alternative presentations included in this article are examples of them.

Alternative Presentations in Evaluation

A number of evaluators have used alternative forms for presenting findings to and communicating with stakeholders (e.g., Lawrenz, Gullickson, & Toal, 2007; MacNeil, 2000). Perhaps distinctive to evaluation, the preferred mode is a combination of traditional and alternative ways of presenting findings rather than one or the other (Goodyear, 2001). Alternative formats include brochures, poetry, cartoons, skits, photography, music, and song, among others (Bessell, 2007; Chandler, Walker, & Greene, 2002; Lawrenz et al., 2007; MacNeil, 2000; Mathison, 2008; Rogers, 2003; Simons & Greene, 2010, 2011; Torres, Preskill, & Piontek, 2005). Further, evaluators have argued that, compared with their traditional counterparts, alternative presentations can provide increased accessibility to data, more personalized interpretations, and a more intense interactive experience (Torres et al., 2005). These offerings encourage deeper contemplative experiences for both the evaluators and the stakeholders and deeper levels of critical reflection and engagement, particularly when there are multiple audiences to consider. In addition, alternative presentations can facilitate stakeholders’ learning by reducing language barriers. Symbols and cartoons are particularly valuable for conveying complex issues in a symbolic and concise way (Torres et al., 2005).

Our Values-Engaged, Educative Evaluation Approach

The alternative approaches described here were developed as part of a set of field tests of the EVEN approach to evaluation in STEM education contexts. This approach (1) seeks to explicate and elevate the value dimensions of STEM education evaluation to a central position in the evaluation’s planning, implementation, and reporting; (2) explicitly advances engagement with the values of equity and inclusion, in part by defining program quality at the intersection of STEM content, pedagogy, and equity (see Figure 1); and (3) endeavors to catalyze critically reflective learning on the part of key stakeholders about the evaluand and about their own professional practice, in the educative evaluation traditions of Cronbach (Cronbach & Associates, 1980; Greene, 2012; Weiss, 1998) and Schwandt (Schwandt, 2002). Application of the framework should provide answers to questions such as, “How well does this STEM program serve the educational needs of students who are least well served in this context?” and “What STEM education programs work well (generate meaningful learning, boost self-efficacy, open new educational doorways) for which individuals under what circumstances and in what ways?”

OPEN IN VIEWER

Figure 1.

The values-engaged, educative framework for defining program quality.

As a values-engaged practice, this approach unmutes the value dimensions of evaluation and seeks to provide safe spaces for value discussions as part of the evaluation enterprise. Values engagement also means providing for the articulation of multiple, diverse stakeholder values related to the program. One tenet of the EVEN approach is inclusion of all legitimate stakeholders. This valuing of inclusion follows both responsive (Hood, 1998; Stake, 1973, 2004) and democratic (House & Howe, 1999; MacDonald, 1976) traditions in evaluation. In addition, it highlights the democratic value of equity, infusing throughout the evaluation considerations of how well the program fosters equity in access, participation, and accomplishment for all learners, in particular those least well served in the contexts at hand.

As an educative practice, this approach promotes learning about the contextual character and contours of effective STEM education programs. It seeks to describe and assess the quality of the learning experiences of students and teachers, to understand the STEM thinking and pedagogical perspectives offered by the program, and to document a multidimensional conceptualization of what was actually learned. Stakeholders’ critical reflections on their own professional practices as related to the evaluand are explicitly encouraged.

Alternative forms for presenting results—especially interim but also final results—are particularly well matched to our aspirations to engage meaningfully with values as part of the evaluation process and to catalyze key stakeholders to do the same. Poetry, drama, narrative, and music all evoke dimensions of experience that are closely related to values—emotions, cherished beliefs, moral–ethical principles, dreams, and aspirations. These forms thus offered vehicles through which we could effectively enact our commitments to values engagement. We believed that such presentations could also fulfill our educative ambitions in deeper and more critical ways than would more conventional presentations of results and that they could help maximize our commitment to equity by creating a nonthreatening space for conversing about values-related issues that are often considered sensitive (e.g., the extent to which the evaluand promoted diversity and equity). Each alternative form was purposely data based and often included intentionally the experiences of those least well served (e.g., student participants from underrepresented groups), thereby giving voice to the typically voiceless. It was for these reasons that we anticipated qualitatively different, more values-engaged stakeholder conversations about the evaluand following each alternative presentation.

We offer four approaches, two from a middle school summer math program, one from a nanoscience initiative with high school students, and one from an evaluation of science outreach at a middle school for girls. The four approaches were (1) linking challenging evaluation findings to the values-engaged evaluation framework, (2) crafting and performing data-based skits to convey distinctive stakeholder experiences and perspectives, (3) presenting multiple program theories schematically to raise critical questions about the program’s underlying logic, and (4) writing data-based poems to capture students’ program experiences. The description of each approach includes sample excerpts from our fieldwork.

These approaches were selected and implemented at the discretion (and invention) of the evaluation team. We used our professional judgment to determine which presentation strategies might be appropriate for engaging stakeholders at particular junctures of a given evaluation, to achieve particular evaluative purposes (e.g., drawing the stakeholders’ attention to important topics or judgments). Stakeholders’ reactions and comments pertaining to each presentation were recorded and monitored, and successive efforts to utilize alternative approaches were informed by our collective understanding of the value that these efforts added to our evaluation ambitions. Notably, at the conclusion of the 2-year evaluation of the middle school Math Champs summer program, stakeholders had come to expect and even look forward to the use of alternative formats during the presentation of findings.

Multiple meetings and reports from meta-evaluators and advisory board participants provided documented assessments of how the EVEN approach was experienced by evaluands. These external reports, together with our own experienced-based questions and critiques, guided many of our team meetings and meta-evaluator–directed reflective sessions. For example, prompts provided by a meta-evaluator used to guide a reflective session included questions such as, “How—and how well—has it gone so far in putting the EVEN ideas into practice? How does EVEN validate or check its own understanding of program theory? How do you address and represent the multiple program theories in any context? If there are differences in how and how well the [EVEN] framework seems to be carried out, what might account for these differences? Where might the fit be particularly good? Not so good?” (Notes from Core Team Meeting, June 4, 2007). During such sessions and meetings, responsive actions to emerging issues were critically discussed, pursued, recorded, monitored, and adjusted (as needed). Meta-evaluators also collected data from our evaluation clients, providing important insights into the impact of our evaluative efforts at each site. This continuous process of reflection and review helped determine and increase the quality of the evaluation. Although the external reporting and questioning primarily targeted the processes, practices, and products of the EVEN approach in general, all of the aforementioned strategies supplied important insights concerning how the alternative approaches were experienced by stakeholders and helped to establish the credibility of our work.

The alternative presentations described here were used during various evaluation phases (i.e., beginning, mid-point, and as part of the final evaluation report) and based on empirical evidence particular to each setting. Regardless of the context in which the approach was used, each presentation illuminated certain aspects of the program, allowing program implementation tensions and achievements to be made visible as they unfolded. Thus, the presentations became a mechanism for enhancing our understanding of the program and for providing (formative) feedback to stakeholders.

In this process, stakeholders and evaluators learned from each other, and these shared understandings informed the development and use of more traditional evaluation approaches (i.e., subsequent data collection, the write-up of the final report, etc.). Each of the alternative approaches was coupled with more traditional presentations of findings (e.g., formal oral and written reports), as we believe that evaluation contexts do call for standard forms of reporting. The alternative presentations, and what was learned from them, effectively illustrated linkages between stakeholder experiences and program goals. Thus, we view our presentations not only in interaction with traditional evaluation methods but also as part of a series of complex, engaging evaluative interactions aimed at increasing the utility of our work.

The Math Champs Program

As a part of our field testing of the EVEN approach, the evaluation team collaborated with diverse groups of STEM educators and program coordinators to evaluate a variety of STEM programs. One of these, the Math Champs program, encouraged underrepresented (specifically African American) middle school students to participate in advanced math courses through a supportive preparatory intervention. Our evaluation was conducted in three phases over 2 years. Prolonged engagement at this site afforded us multiple opportunities to present both interim and final evaluation reports, and the latitude to explore the use of alternative approaches for presenting findings. The Math Champs program, held during a 5-week summer session, was designed to prepare rising sixth, seventh, and eighth grade students for successful participation in advanced mathematics courses. Priority invitations for program participation were extended to African American students with strong mathematics skills who were currently not in the school’s advanced math track. The program had an explicit agenda of changing the school’s norms about who was smart enough to be an advanced math student. Most students who participated in the Math Champs program registered for advanced mathematics courses the following academic year. These students were strongly encouraged to also register for a math support course that was offered concurrently during the school year.

From the perspective of the educational staff within the school, the overall objective of the program was to increase minority representation in advanced mathematics courses. Specific summer program content was developed and implemented by the program instructors and designed to be congruent with the content and goals of the advanced math course appropriate to each grade level. The program focused primarily on developing students’ skills in algebra, geometry, and statistics as well as increasing their skills and confidence with related technologies (e.g., graphing calculators, statistical software). In addition, the program emphasized hands-on learning and the incorporation of technology whenever possible and appropriate.

Presentation #1: Linking Challenging Evaluation Findings to the Values-Engaged Evaluation Framework

Early in our Math Champs evaluation, we presented an interim evaluation report to school and program administrators and to math faculty members. This presentation had multiple purposes: To familiarize stakeholders with core aspects of the EVEN approach and especially the central role of values therein; to describe our recent and forthcoming evaluation activities; and to provide a snapshot of the evaluation data collected to that point (i.e., foreshadowing some of the evaluation findings). To accomplish these tasks, evaluation team members prepared a presentation that included (1) a diagram describing key components of the evaluation framework (see Figure 1); (2) a diagram demonstrating how the evaluation framework was used to collect, analyze, and interpret observation data (Figure 2); and (3) a set of prompts for guiding a reflective discussion of the observational data.

OPEN IN VIEWER

Figure 2.

Snapshot of students’ experiences. This figure presents a “snapshot” of students’ experiences in their advanced math course as captured by initial observation data. The “content” and “pedagogy” excerpts are representative of the data collected, and the “equity” excerpt offers an observed instance of equity challenges. Brief summative statements precede each excerpt, and each excerpt provides an empirical basis that supports each summative statement.

We chose the data to include in Figure 2 to signal emerging issues of possible importance in our Math Champs evaluation. Notably, “content” data represented our observations of the high level at which the advanced math courses were being taught. The “pedagogy” data came from our observations of the math support class offered to program participants during the school year. The routine character of this class—sometimes more like a study hall or even a free period than an environment in which program participants could get targeted expert support in math—had surfaced as a possible concern. And the racial incident conveyed in the “equity” circle (which we presented in ways that safeguarded confidentiality of the participants involved) signaled our emerging concerns about cross-race peer relations within the advanced math classes.

Following the presentation of Figure 2, the evaluation team offered a series of reflective prompts designed to provoke reflection and engagement. An excerpt from the exchange that followed is reproduced below.

Excerpt from reflective discussion

Evaluator    We are interested in understanding the intended role of the advanced math courses, and their connections to the Math Champs program. Can you tell us about this intended role (focusing here on program participants)? What is most important about students’ experiences in your advanced math course?

Stakeholder #1 I don’t see program participants’ role in this class any differently than I see other students’ role. Students need to be engaged and participating. The program does provide some students with an opportunity that they might not otherwise have… The program is intended to provide students with encouragement and engagement. The program can offer kids the opportunity to be proficient in math and to feel good about themselves as math learners. This encouragement and support helps lift them out of the “quagmire” that they are in with math.

Stakeholder #2 There were, in the past, the “five bad boys in the first row.” These kids misbehave because they are not understanding the material being taught. The Math Champs program gives them a chance to “get it,” to be like other kids, to have the same chances as other kids to be engaged and to be acknowledged that they can do math.

Stakeholder #3 And there has been a perceptual change on the part of teachers. Previously, they would say that these kids do not have upper level class behaviors. This program changed that paradigm …

In a team debriefing, we agreed that using the EVEN framework to organize this interim report was an effective strategy for introducing stakeholders to key components of the evaluation approach and, even more importantly, for catalyzing thoughtful discussions among them about selected substantive (and potentially sensitive) program issues. We believe the ensuing discussion was deeper, more engaged, and more critical than would have occurred if we had presented a more traditional qualitative summary of our observation data. For example, a meta-evaluator noted that Math Champs staff “indicated that the graphic was a useful supplement to verbal descriptions [of the EVEN approach].… They liked it, found it valuable to them” (Meta-Evaluation Report to Board, 2008).

We theorized that the alternative presentation provided a concrete framework, a targeted set of categories (program content, pedagogy, and equity), and a specific instance of each category for discussion. Further, this presentation inserted some cognitive, social, and emotional space between the stakeholders (most of whom were highly committed to the Math Champs program) and the issues. The diagram itself (Figure 2) created separation between the stakeholders and the issues, rather than keeping them emotionally close, making it safer for them to engage with the issues openly rather than defensively. In short, we argue that it was these features of the alternative presentation—structure, specificity, and space—that contributed to the thoughtful and critical discussion that took place. It was also an excellent start to our 2-year evaluation work with the educators of the Math Champs program, during which we experimented with other forms of alternative presentations.

The NanoDARE Program

Another context in which we tested our values-engaged approach was the NanoDARE program. This program targeted high school juniors and seniors from groups underrepresented in scientific fields who have aptitudes and interests in STEM and the potential to pursue these subject areas in higher education and in their careers. These youth were provided interactive research experiences in a nanolab on a university campus guided by two university scientists and two science educators. Because it was the program’s inaugural year, the evaluation focused on program theory components: inputs, outputs, outcomes, and program experiences. We paid particular attention to the ways in which the NanoDARE educational staff articulated their program logic (the program as intended) and the ways in which the high school student participants made sense of their program experience (the program as experienced).

Presentation #3: Organizing Evaluation Results in a Program Theory Framework to Raise Critical Questions About the Program’s Underlying Logic

We presented our understanding of the staff and student program logics by juxtaposing both in a program theory representation (see Figure 3). Our intent was to display the differences in perspective to the program director and science educators to catalyze reflections on the program logic and design. We first reminded the staff about the program theory focus of the evaluation and explained that the diagram was constructed with data collected through various evaluation activities (document reviews, staff and youth interviews, and nanolab observations). We then provided time for the staff to silently review the diagram. Next, we invited the staff to ask questions based on what was presented and engaged them in an extended conversation about the principles and assumptions underlying the program theories.

OPEN IN VIEWER

Figure 3.

NanoDARE program theory. Four elements of program theory are included in this diagram: Program inputs, experiences, outputs, and outcomes. Items listed inside the circle represent data drawn from data collected from program staff. Items listed outside the circle are drawn from data collected from students.

We believe that presenting both student and staff program theories in a single diagram provided a visual stimulus that effectively captured key aspects related to what took place during the first year of the program. For example, during an interview as part of a meta-evaluation, a NanoDARE program staff member stated that the “diagram was very helpful” and that “it is a powerful way to connect concepts.” He continued, “people can find it easier to accept information in a visual format. This is a good tool. It could be applied to a number of different programs, and it captures subtleties.”

Presenting both student and staff program theories in a single diagram also created an opportunity for the staff to reassess the goals of the program. For example, because many of the youth participants indicated that they already had a significant amount of laboratory experience, spending a lot of time conducting hands-on activities in the lab was not necessarily viewed as a benefit of the program. This led to the staff considering questions such as, “Should this program provide laboratory experiences for youth with little to no laboratory experience? If this is our goal, how might this focus impact our recruitment of youth program participants? How might it impact our program curriculum?” Finally, these experiences helped to refine our evaluation report (including targeted recommendations), which in turn—and perhaps more significantly—informed the second iteration of the NanoDARE program. For example, during the first year of the program, one goal had been to increase the likelihood that the youth would become scientists themselves. However, a staff member indicated that, “meaningful youth development was a priority, over intended program goals of cultivating a career in science, especially nanoscience.” This program staff member further “noted that if a youth participant realized he/she did not want to be a scientist, that would be a very important outcome” (NSF review project July 8 document; reflective session #6). This reflection became a more explicit youth development goal during the second year of the program. That is, the program still targeted students with aptitudes in STEM but shifted from striving primarily to increase their likelihood of pursuing careers in science to also providing authentic science laboratory experiences as an opportunity for participants to explore and discover their own unique interests in science, or even in other subject areas.

Science Outreach at the Girls Middle School

A third context in which we used alternative presentations was the Science Outreach program conducted at the small, private Girls Middle School (GMS). As required by their grants from the National Science Foundation, and also guided by their own values and commitments to K-12 and community outreach, selected STEM faculty from the area university participated in the Science Outreach at GMS to provide girls with a wide range of unique, state-of-the-art science learning experiences to complement and enrich their regular science curriculum. With the recognition of the underrepresentation of women in STEM fields, and in tandem with the main evaluation priorities of the school administrators who requested the evaluation, our work focused on assessing the girls’ experiences in and perceptions of the program, especially in relation to possible longer term outcomes of enhancing and nurturing girls’ interests, motivation and self-efficacy in science learning. In accordance with our own equity-oriented agenda, the evaluation also sought to address the relevance and importance of the GMS Science Outreach program to the broader STEM education communities’ mission to increase diversity and equity in STEM fields.

Presentation #4: Crafting a Data-Based Poem to Capture Students’ Program Experiences

As with our other field test sites, the evaluation team explored the possibilities of using alternative formats for presenting the multiple data sets gathered from various stakeholders. Specifically, we wished to foreground the diversity of views, voices, and values that were captured in our work. That is, “Science Outreach” looked and felt quite different from the standpoint of the girls, the GMS teachers, and the university science faculty. We featured several alternative presentation formats in our final written evaluation report. In particular, we presented the experiences and perspectives of the student participants—their views of science and science learning, how they experienced and understood the Science Outreach program, and what they valued in their Outreach experience—in poetic forms (see Figures 4 and 5).

OPEN IN VIEWER

Figure 4.

In their own words: Science Outreach activities and interactions.

OPEN IN VIEWER

Figure 5.

Students’ views on the value of their Science Outreach experience.

These poems were crafted using the words of students, either direct quotes or close paraphrases, from the evaluation data gathered. The poetic form was intended to convey students’ views and experiences in their own words, rather than being filtered through our analysis or interpretations. The poems were also designed to enliven and authenticate the perceptions of the GMS girls, whose voices had rarely been heard before in that context. For example, we included short one-line responses that are often encountered when interviewing teenage students. We had hoped that the nature and character of poetry—– which allows for multiple meanings and interpretations—would evoke diverse reactions and further enable a more dialogic engagement with the issues presented.

At the closing of the evaluation, we reserved time during one of the school’s weekly staff meetings to present our report and discuss the key findings with the school’s administrators and faculty. In general, our evaluation was well received, including the poems. The GMS stakeholders said they appreciated hearing what the students had to say in their own words. The poems affirmed the benefits of their Science Outreach experience at GMS and also underscored the importance of hands-on activities, experiments, and projects in science learning. The poems generated a discussion about the value of STEM outreach more broadly, not only for the girls at GMS but also for general student populations in public schools, especially those traditionally underrepresented in STEM and those most underserved in a given school context.

Reflections From Evaluation Team Members

Table 1 briefly summarizes the field trials of the four alternative presentation approaches that are described in this article.

OPEN IN VIEWER

Table 1.

Summary of Field Trials.

Program context	Alternative presentation form	Outcomes
Math Champs Program: Designed to prepare underrepresented (specifically African American) middle school students (grades 6–8) for participation in advanced mathematics courses	Linking evaluation findings to the values-engaged evaluation framework of program content, pedagogy and equity, accompanied with a set of reflective discussion prompts	Catalyzed thoughtful discussions among the key program staff about selected substantive (and potentially) sensitive program issues
Math Champs Program (See above)	Crafting and performing data-based skits to convey distinctive stakeholder experiences and perspectives	Engaged stakeholders in values-based discussions about issues central to the program with a concrete and specific example of the issues through skit presentation and performance
NanoDARE Program: Designed to provide interactive research experiences for high school juniors and seniors from groups underrepresented in scientific fields who have aptitudes and interests in STEM	Presenting program theories of multiple stakeholders schematically in a data-based diagram to raise critical questions about the program’s underlying logic	A single diagram representing multiple program perspectives provided a visual stimulus that effectively catalyzed a reflective discussion about important program issues, and created an opportunity for the program staff to reassess the goals of the program, which in turn significantly informed the second iteration of the program.
Science Outreach at the Girls Middle School: Intended to provide girls with a wide range of unique, state-of-the-art science learning experiences	Crafting data-based poems to foreground diversity of views and values captured in the evaluation, specifically to capture students’ voices and their program experiences	The poems were well received by the stakeholders, affirmed the benefits of students’ program experiences, and generated discussions among program staff about the value of STEM outreach more broadly, especially to those traditionally underrepresented and underserved in STEM education.

Note. STEM = science, technology, engineering, and mathematics.

Each evaluation presentation is unique. The character of the participants, as well as the character and tone of the evaluation findings themselves, can influence the ways in which a discussion unfolds. In addition, the professional evaluator often has an agenda or a preconceived notion of how a meeting should progress. Utilizing alternative approaches may help evaluators navigate some of the tensions that are inherent in such activities (e.g., Whose voice should be privileged? How should data be summarized and presented? What should be emphasized?). These approaches were generally effective in promoting active engagement, critical reflection, and shared dialog among various stakeholder groups. For example, Math Champs staff said:

[The evaluators are] exceptionally skilled at leading a self-reflective process for us. What we get from EVEN is packaged in a practical way, in a way useful for us. It is extremely valuable. It is a growth model, and like having an outside facilitator and coach, who is positive, who helps us do better.

The conversations that resulted from the use of alternative formats frequently included stakeholders asking about, or reflecting upon, the potential utility of our work. Sometimes this connection was intentional (e.g., we used program theory, in part, to help stakeholders identify logical missteps), and sometimes it was not (e.g., we were asked to present our skits at a district professional development session). It is not clear whether the level of stakeholder involvement and engagement we experienced directly resulted from the implementation of alternative presentation approaches, although we are confident that they contributed to it. For example, the Math Champs Staff said:

We have enjoyed having them [the evaluators] here. We have appreciated their asking questions. […] Teachers are pleased when they hear about a feedback session. This makes them feel excited, and even more positive. Meeting with [the evaluators] is something our teachers like, something to which they look forward.

Our experiences indicate that these presentation strategies have the potential to add richness and value to the evaluative process for stakeholders and evaluators alike.

Moreover, the use of alternative approaches served as an effective tool for fulfilling the educative intent of our evaluative efforts. That is, the evaluation team endeavored to inform stakeholders about the nature and quality of these particular STEM programs from multiple perspectives, including our own. Alternative approaches allowed us to craft presentations in much the same way that a teacher prepares a lesson. We were able to highlight main points, probe points of (possible) tension, and facilitate meaningful and productive conversations about the program. We could educate stakeholders about the major components and tenets of our evaluation approach more deliberately. It is not uncommon for evaluation clients and stakeholders to be uncertain or unclear about the purpose or nature of the evaluation in which they are participating. Our goal was to help evaluation clients and stakeholders understand the intent, process, and utility of the EVEN approach.

Alternative approaches provided us with a unique opportunity to privilege voices that may have otherwise been marginalized or very quiet in the evaluative process. In this way, we were able to tailor the presentation of findings to match the character of the stakeholder group, the evaluation approach, and the evaluation findings themselves.

Admittedly, we did not employ alternative approaches without some degree of hesitancy. We considered these approaches as one strategy to encourage diverse stakeholders to participate in meaningful conversations about their program, with a focus on surfacing and engaging diverse value stances relevant to key program assumptions, activities, goals, and outcomes. To this end, it was important to ensure that stakeholders had a basic understanding of the context, meaning, and implications of the findings before engaging them in critical and reflective conversations. We determined that this could be best accomplished by coupling alternative approaches with more traditional presentations of evaluation findings. To accomplish this task, a significant amount of time was spent designing and preparing each alternative presentation. The content of each presentation was tailored to fit each audience (for logistical reasons, for privacy, and to best facilitate program learning while being inclusive of diverse stakeholders). Although this experience is not uncommon (i.e., presentations of evaluation findings are frequently tailored for different audiences), the nature and the degree of modification required a considerable amount of forethought, reflection, and creative energy.

Concluding Comments

Crafting a cogent presentation of evaluation results is a critical step in program evaluation. Evaluators have a responsibility to provide diverse audiences with clear information and to represent their evaluative conclusions accurately and defensibly. However, evaluators also have an obligation to ensure that the information they present is understood and to promote critical reflection regarding the real and potential implications of their work. Indeed, critical reflection is an important component of professionalism, and evaluators are uniquely positioned to promote such practices (Schwandt, 2007). Accordingly, an important question for evaluative research becomes evident: How can evaluators effectively engage diverse audiences for the purpose of promoting such reflection? Part of the answer, we believe, is to develop and employ alternative approaches for presenting the information and judgments formed through our evaluative efforts. This expansion of our professional practice increases the quality of our interactions with clients and stakeholders, thereby making our work more relevant, more salient, and more likely to be used.