Abstract
The challenge and opportunity for the field of evaluation is to prepare evaluators with sophisticated interpersonal competence alongside technical research skills. This article shows that service-learning, a “real-world” pedagogical model, holds promise as a way to educate students in evaluation while developing their interpersonal skills. While other community-based approaches offer experiential applications of evaluation methods, service-learning adds the important element of reflection to deepen student learning. A course that employed the “client-based” service-learning approach, individual-to-group student assignments, practice with interpersonal skills, and ongoing reflection activities is presented as a pedagogical approach to expand the range of interpersonal skills that can be developed among students. Student surveys showed that the course conveyed many benefits beyond technical knowledge of evaluation: use and awareness of important interpersonal skills, growing insight into their community and its people, growth of professional self-confidence, and kindling an interest in the field of evaluation.
Keywords
As they confront entrenched and emerging societal challenges, public and nonprofit leaders can be more effective if they make decisions based on a foundation of solid data with which to analyze community needs, implement evidence-based programs, improve program performance, and achieve impact for citizens and funders. The demand for individuals who can work with such data is strong. Innovation Network’s 2016 State of Evaluation found that 92% of responding nonprofit organizations engaged in some form of evaluation and 85% agreed that evaluation was important to understanding the impact of a program (up from 68% in 2010). Most respondents (85%) used evaluation results to develop new initiatives (State of Evaluation, 2016). However, nearly half (48%) agreed that insufficient knowledge and skills of staff limited their ability to engage in evaluation. Looking at the pipeline for potential job applicants trained in evaluation, Lavelle (2011, p. 363) concluded that “there is significant demand for evaluators trained in diverse forms, perspectives, and processes.” LaVelle found thousands of employment listings that used the words evaluate, assess, measure, and data. A quick replication of his search reveals this demand for evaluation continues to this day.
However, these organizations need more than data technicians. Since evaluation takes place in community settings, the profession has begun to highlight the importance of interpersonal competence: listening and building trust, working in a team, engaging in culturally responsive interaction, facilitating shared decision-making, managing conflict, and more (American Evaluation Association [AEA], 2018; Stevahn & King, 2016). The challenge and opportunity for educators of evaluators is to prepare individuals with well-developed interpersonal competencies to complement technical evaluation skills. Interpersonal competencies take time to develop, but by making them more of an explicit focus of evaluation training and education, learners can be well equipped to implement effective evaluations.
This article describes a service-learning approach to educating evaluators that infuses development of interpersonal skills. The approach places learners in community settings where they practice evaluation skills with heightened emphasis on interpersonal interactions. Service-learning, a high-impact approach that connects a curriculum with hands-on practice and reflection, shows promise as a technique to develop well-rounded technicians (American Association of Colleges and Universities, n.d.). After reviewing the recent research on the competencies needed by evaluators and the pedagogical model known as service-learning, this article describes how the instructional approach infuses interpersonal awareness and shares feedback from students on the multiple benefits they reported for their emerging development as evaluators.
What Does It Take to Be a Competent Evaluator?
For more than two decades, evaluation organizations, educators, and practitioners have worked to discern the knowledge, skills, attitudes, and practices one needs to conduct strong program evaluations. King, Stevahn, Ghere, and Minnema (2001) compared common core competency domains of the leading evaluation taxonomies and continued to develop a competency taxonomy in the years that followed (Stevahn, King, Ghere, & Minnema, 2005). These core competency domains included professionalism, technical methods, situational aspects, management aspects, and interpersonal competence.
Building on this and other research, the AEA (2018) recently engaged its membership to identify a set of evaluator competencies shown in Table 1. This new taxonomy described five domains of competence for evaluators: adherence to professional standards, use of systematic inquiry and methodological skills, understanding of the unique context of the evaluation, skills in evaluation planning and management, and interpersonal competence, defined as “human relations and social interactions that ground evaluator effectiveness for professional practice through the evaluation” (AEA, 2018).
American Evaluation Association Evaluator Competencies.
Note. AEA = American Evaluation Association
In evaluation, technical competencies such as designing evaluations, sampling, and data collection approaches are utilized in community settings. Employing these methods requires broad-based skills in evaluation techniques and competency in data analysis (Bakken, Nunez, & Couture, 2014; Davis, 2006; Johnson, McGuinness, McCorkendale, & Laney, 2007). However, application of technical methods often is mediated by the interactions enabled by the evaluator’s interpersonal competence (Stevahn & King, 2016). “Realization that clearly communicating purpose, conveying information, respectfully seeking access, deeply listening to concerns, attending to cultural norms, and constructively dealing with conflicts or tensions that may surface will be key to conducting tasks successfully” (Stevahn & King, 2014, p. 154). The AEA (2018) taxonomy not only emphasizes this interpersonal role exclusively in Domain 5 (Interpersonal), but it includes nontechnical skills within Domains 3 (Context) and 4 (Management). These two domains refer to competence in interacting with others in unique organizational circumstances, navigating the situations and motives within the context of an evaluation, fostering use of results, and networking and negotiating collaborative work plans.
As project leaders, evaluators must monitor the human dynamics as well as the technical quality of work completed by a team. For example, in community settings, evaluators must manage the give and take of competing goals held by stakeholders and must adapt to situational limitations as the work proceeds. “Whether a study is more evaluator-directed, collaborative, or participant-directed, all evaluations require evaluators to interact and/or facilitate interaction at some level with and among people” (Stevahn & King, 2016, p. 69). Evaluators plan and lead meetings, share information, organize the tasks of others, identify the unique talents of team members, recognize barriers and overcome them, encourage the contributions of diverse actors, and at times, refocus the team. The evaluator keeps members of the group on the same page, balances competing interests, and facilitates the project by developing trusting relationships, initiating constructive dialog, listening to stakeholders, and applying conflict resolution approaches. Cross-cultural skills enable the evaluator to “be responsive and inclusive to the values that those of differing cultures bring into the evaluation of programs” (N. Christie, 2012, p. 744). Frequently, stakeholders and evaluators differ in terms of age, place of residence, education, race, or social class). Cultural competence leads to better working relationships with a program’s staff or clients, affects decisions about how to collect data, and encourages use of culturally appropriate methodology and inclusion of stakeholders at all stages. Moreover, evaluations are often political: They have the potential to alter program resources, jobs, and future opportunities. All of these important activities involve interpersonal dynamics; it is essential that learners have a chance to develop interpersonal awareness and competence as they learn to apply evaluation methodologies.
King and Stevahn (2015) encouraged greater use of the competencies by instructional programs and professional learning communities, urging the field to examine the best practices for preparing well-rounded evaluators with these competencies. They wrote that “research on the effects of specific approaches to competencies instruction…would mark a helpful beginning in learning ways to teach novice evaluators the competencies that can lead…to good practice” (p. 34). Lee, Wallace, and Alkin (2007, p. 537) urged evaluation programs to “move beyond traditional teaching methods to those that are more engaging and ‘hands on’ to help students understand the interactive nature of program evaluation.”
This article presents an instructional strategy, service learning, that enables learners to develop technical competence while practicing interpersonal approaches needed by evaluators. These nontechnical competencies may be difficult to teach in classrooms because they are unpredictable and dynamic. The article illustrates a viable pedagogical technique to develop a wide range of evaluation competencies with a focus on interpersonal interactions.
Developing Evaluators’ Interpersonal Competencies
Are interpersonal skills being taught sufficiently in evaluation programs? And if so, do students demonstrate these competencies? Dewey, Montrosse, Schroter, Sullins, and Mattox (2008) reported that while employers and job seekers agreed on the importance of interpersonal skills for evaluators, job seekers had not adequately developed them. Davies and MacKay (2014, p. 425) surveyed university faculty who had taught advanced evaluation courses and found a strong emphasis on technical evaluation skills but that “topics regarding professional issues, interpersonal skills, and evaluator dispositions were less likely to be addressed.” While well equipped to teach the technical methods, educators may give less focus to development of interpersonal skills. Although students pick up some of these skills as they discuss ethical concerns, cultural competence, planning an evaluation, and other evaluation topics, they may not have sufficient engagement to develop interpersonal competence.
Various models of hands-on learning can help students develop evaluation competencies, including interpersonal skills (Davies & MacKay, 2014). Classroom-based experiences such as simulations, case studies, collaborative learning, role plays, and other adult-learning strategies simulate actual evaluation circumstances. In these approaches, as they interact with each other and the educator, learners develop insight into dynamics of evaluations. However, based in the classroom, students cannot fully grasp certain evaluation challenges without experiencing them. For example, how might one collect data from someone with dementia (or should this be attempted at all?) or from youth with a very different cultural background than the student? Likewise, a role play may not sufficiently replicate interactions where the evaluator wants to implement a control group and address the ethical concerns of staff who do not wish to randomize individuals at intake. Finally, a classroom presentation for peers (all of whom are studying evaluation) may be quite different than one in a community setting in front of managers and staff with little knowledge of evaluation.
Community-based learning pedagogies can enhance development further than the classroom experience by exposing novice evaluators to the “intricacies of conducting evaluations in real-world settings” (Trevisan, 2004, p. 256). These experiences enable students to sharpen evaluation skills by partnering with agencies through community internships, problem-based learning activities, collaborative learning, and client-based projects with program partners (Bakken et al., 2014; Dillman, 2013; Johnson et al., 2007; Lee, Wallace, & Alkin, 2007; Oliver, Casiraghi, Henderson, Brooks, & Mulsow, 2008; Stevahn, Gere, & Minnema, 2005; Waldner & Hunter, 2008). Internships offer opportunities to apply evaluation skills independently with feedback from a site supervisor and faculty advisor but with fewer opportunities to learn a range of interpersonal concepts.
In a “problem-based learning” (PBL) course design, Lee et al. (2007) advocated hands-on work with a single program throughout a course to expose students to a wide range of evaluation challenges. Reflection focused on critical thinking was a core component of PBL as students responded to complex and changing situations requiring professional decision-making. Goals of the problem-based approach are to develop a knowledge base, problem-solving skills, self-directed learning skills, collaboration, and intrinsic motivation. While PBL enabled students to interact with people as part of the evaluation and addressed the key skill of collaboration, development of broader competence in interpersonal skills was not an explicit goal.
Typically, these hands-on instructional strategies add value by deeply engaging learners in critical thinking, collaboration, and decision-making of evaluation. The approaches offered a richer learning opportunity in which to develop technical competence, but only one model described in the literature explicitly focused on an interpersonal skill (collaboration). Did students understand and retain the interactional skills they had used? Was their learning about interpersonal dynamics explicit? What was the range of interpersonal competencies did they achieve? The service-learning approach adds an important instructional technique that was not described in the literature describing the other pedagogies: reflection.
The service-learning pedagogy shows promise to expose students to a wide range of evaluation competencies as it integrates (1) learning of technical evaluation methods, (2) practice/engagement in real-world settings, and (3) regular reflection by students on interpersonal competence. While reflection may be used at times in the other hands-on pedagogies, it is a central component of service learning. Coupled with community engagement, reflection may be an important way to help students become aware of interpersonal and contextual dynamics that can be practiced and learned. The next section describes core components of service learning followed by description of a particular course in program evaluation that focused on initial development of interpersonal skills.
The Service-Learning Approach
Dicke, Dowden, and Torres (2004, p. 201) described service-learning as “an academically rigorous instructional method that incorporates meaningful community service into the curriculum.” In essence, service-learning is a pedagogical framework that can be applied across a wide variety of disciplines and educational settings (Kronick & Cunningham, 2013). Professional training programs can incorporate its core strategies to develop the contextual awareness and interpersonal skills of practicing evaluators.
Service learning combines three important components: curricular instruction, field experience, and reflection. “Academic work and service are completed together so students study issues and become participants in addressing them in a particular community setting” (Meyers, 2009, p. 373). As they engage in the community, students interact with unfamiliar people, settings, and tasks that generate a level of “disequilibrium” and anxiety that stimulates deep learning (Hullender, Hinck, Wood-Nartker, Burton, & Bowlby, 2015; Sessa, London, & Hopkins, 2010). As they interact and navigate the community service experience, students have opportunities to get outside of the academic setting and to communicate with a variety of individuals and practice a variety of interpersonal skills such as negotiation, communication, cross-cultural awareness, and empathy.
As students reflect on the experience (through guided writing and discussions), they integrate the curriculum with real life, reduce anxiety, and increase deep learning. Hatcher, Bringle, and Muthiah (2004) found that regular, well-structured reflection activities created meaningful learning experiences for participants. Findings showed that when reflection activities engage the learner in examining and analyzing the relationship between relevant, meaningful service and the interpretive template of a discipline, there is enormous potential for learning to broaden and deepen along academic, social, moral, personal, and civic dimensions. (p. 39)
Research has begun to accumulate on the positive effects of service-learning on students. In a meta-analysis, Warren (2012) reported that service-learning had statistically significant and positive effects on student curricular learning outcomes, exceeding impacts of traditional instructional methods. Engaging in service-learning as an undergraduate can substantially advance students’ academic development and lead to a deeper understanding of concepts, critical and higher order thinking, and problem-solving (Palombaro, Dole, & Black, 2015). Students participating in service-learning experiences expand a sense of civic responsibility, grow in cultural awareness and competence, show increased civic engagement, deepen their understanding of community, develop a future commitment to volunteering, and show a more positive attitude toward service. Finally, students develop personally through service learning and its impacts on moral development, communication, interpersonal, and leadership skills (Ahmed & Oberling, 2007; Astin & Sax, 1998; Imperial, 2007; Palombaro et al., 2015). Next, this article describes a client-based service-learning course where students learned basic evaluation methods applied to real-world programs. This course made learning of interpersonal skills explicit through assigned readings, written reflection, and class discussion.
A Service-Learning Program Evaluation Course
“Evaluating Public and Nonprofit Programs” is a one semester, upper level required course within an undergraduate Public Administration curriculum offered at a Midwestern public university with a typical enrollment between 22 and 30. The course contributes to a broader aim of the Network of Schools of Public Policy, Affairs, and Administration (2009), that is to develop “the ability to communicate and interact productively with a diverse and changing workforce and citizenry.” Most enrolled students were senior public administration or political science majors with little or no prior awareness of evaluation. Most students entered this course with varying levels of professionalism and interpersonal competence; some were returning adults while others were traditional students. Most were preparing for careers as managers of public service organizations.
The primary goal of the course was to prepare future public and nonprofit administrators to implement basic evaluation techniques in pursuit of effective programs while developing confidence and interpersonal awareness. As a one-semester overview course, most learning objectives addressed early development of technical competencies, and interpersonal competence was embedded throughout. Students interacted with each other along with community stakeholders as they learned the following technical competencies: understand and use important evaluation terms; develop appropriate evaluation methods for a given program and context; analyze a program and its components and map them in a logic model; develop evaluation questions and define the purpose of an evaluation; identify, operationalize, and measure program outcomes; and discuss how evaluation can be utilized to make decisions and improve programs.
The course had several defining features: an evaluation curriculum, client-based service-learning projects completed by groups, a series of individual assignments, and structured reflection through writing and class discussions. In combination, these learning activities conveyed certain basic technical evaluation skills and provided opportunities for students to start to develop the interpersonal skills so important to evaluators.
Evaluation Curriculum
Since the goal of the course was to prepare future public service managers to measure program outcomes, it did not delve deeply into evaluation theory or history of the field. Instead, it focused on the front end of planning an evaluation, identifying program outcomes and basic measurement techniques. A selection of textbook chapters, journal articles and cases, Internet websites, and classic evaluation guidebooks (e.g., Centers for Disease Control and the Kellogg Foundation) were assigned as “just-in-time” readings aligned with the tasks of the evaluation field project. Additionally, students reviewed literature for information about the community needs served by their programs and evaluations of similar programs. For example, when a student group worked with a local veterans’ court, they read about homelessness, incarceration rates, and mental health issues of some returning veterans. They discovered evaluation literature relevant to diversion court programs. Finally, students read about and discussed the interpersonal skills required by evaluators.
Client-Based Service-Learning Project
At the outset, the instructor explained the service-learning pedagogy to students and discussed goals and expectations to help students understand that this course would be different from typical college courses; it would challenge them to learn and apply the material in a way that yielded a product that was important to a community agency. Making this clear up-front motivated students and reduced anxiety as they attempted to reconcile what was going in the class with more traditional courses. The professor urged students to remain flexible and open minded.
Employing a “client-based” approach, the course organized students into teams which completed projects for community partners (Waldner & Hunter, 2008). Working with program staff, students’ end goal was to produce a complete outcome measurement plan for an actual program, including a logic model, set of priority outcomes and indicators, measurement tools, broad data collection plan, ethical protections, limitations or weaknesses of the measurement system, and a memo that overviewed everything. Based on student interest and input, the instructor formed groups of three to five students to partner with one program the entire semester who would meet with them 3 or 4 times. Within groups, the instructor attempted to balance student characteristics (such as maturity, perceived confidence, academic skills, and motivation). Student groups did not collect data but prepared the agency partner to do so as they delivered a “shovel-ready” system and tools to employ. Achieving this goal required developing technical and interpersonal skills as they worked with their peers and community professionals.
Securing the participation of high-quality community partners was an essential foundation for the course. Prior to each semester, the instructor recruited 6–10 managers of public or nonprofit programs who were interested in developing outcome measures, often to satisfy funders or policy makers. Area agencies saw this as an opportunity to build evaluation capacity and jump-start evaluation work, heretofore pushed off due to limited time, insufficient evaluation skills, and lean staffing. Ideal programs for this course had the following characteristics: simple interventions or delivery models; a target audience with perspectives or circumstances that were new to students (e.g., veterans with arrest records, individuals with memory loss or cognitive disabilities, homeless persons, and children with parents who did not speak English); observable group-based programs (not confidential or individual services) where students could visit onsite; ability to dedicate two or three staff who were receptive to working with college students and who were available to visit campus three times over the semester; and goal of developing evaluation capacity of staff.
Over time, several community agencies (such as the local YMCA) participated repeatedly. Table 2 describes a sampling of programs with the projects completed by student teams. Community partners received a written letter of agreement (shown in Table 3) outlining the expectations for the semester. The instructor promised an efficient use of their time, a chance to develop a partnership with the university, and a major step forward in developing outcome measures of their own programs. Program representatives visited the classroom midsemester and at conclusion of the course for presentations; some met with students again outside of class.
Examples of Partner Programs and Service Learning Projects.
Letter of Agreement With Community Partners.
Individual Assignments
Since most students entered this course with minimal knowledge of evaluation, they needed support to learn evaluation skills and then apply them effectively. A set of four individual assignments shown in Table 4 developed students' basic evaluation skills and provided scaffolding to support their learning. At the same time, these assignments contributed meaningful content for the group project. As the semester progressed, students completed each of the four assignments individually and, after receiving feedback (and a grade) on their work, combined their ideas to develop a product as a group which would be received by the agency partner. In this way, all students made the initial effort, learned from it, and had the chance to revise and improve the work collaboratively. As the course proceeded, the responsibility for quality work shifted gradually to the students as they felt motivated to help the program. Each individual exercise related to a component of the evaluation plan shared with agency partner at the end of the course. Importantly, the first assignment required students to visit and observe their program and prepare a description. According to student feedback, doing so early in the semester generated interest and motivated students to work hard on behalf of their partner while offering insight into the program context, target audience, and operations. Students gathered two other items for the first assignment: Background information on the societal need addressed by the program and literature related to an evaluation of a similar program.
Individual Assignments Linked to Group Deliverables.
A second individual assignment required students to specify their program’s inputs, core activities, outputs, and outcomes. They developed outcome chains, beginning to work with logical progression. Using this initial work, groups engaged in substantive discussions regarding the formation of the logic model and prepared to share it with agency staff, including sending it ahead of time with an explanatory message, choosing handouts, and deciding which information would be presented by whom. Students discussed the importance of active listening and how to remain respectful of staff expertise while sharing the draft logic model. Staff and students then jointly revised the logic model and chose the outcomes to measure.
A third exercise required students to construct an evaluation question reflecting the selected outcomes and develop or adapt data collection tools and methods to measure the outcomes. Students utilized evaluation resource websites, books, and literature searches to locate valid, publicly available survey scales or observer’s guides. When appropriate items could not be found, students developed new survey items, having read several resources on survey item development and practiced some question-writing skills in class. With such limited experience in survey design, not all students mastered survey development. Regardless, with some support from the instructor, most times, the student groups provided a strong start for the agency in development of a data collection tool.
The fourth and final individual exercise encouraged students to explore more rigorous evaluation design possibilities for the program and reflect on their group's work in terms of strengths, weaknesses, and ethical concerns to address. Student groups then proposed a general quasi-experimental evaluation design to assess impact of the program, considering options for comparison groups and additional points of data collection. In some cases, a rigorous design could not be adapted to the program of interest—but in attempting to do so, students learned principles of design nonetheless. Finally, students reflected on the strengths and limitations of the groups’ outcome measurement plans, identified ethical challenges, and developed recommendations pertaining to data collection. Based on the final individual exercise, groups compiled a memo for the agency partner that discussed these topics and recommendations.
Reflective Writing and Discussion
Beyond this carefully structured sequence of individual-to-group assignments, reflective writing and discussion aimed to increase student awareness of effective interpersonal strategies, or at least to introduce them. With support from the instructor, the entire class regularly worked through the challenges faced by various groups. Students offered ideas to peers, heard the instructor give feedback, and considered issues related to the evaluation along with political, cultural, and interpersonal issues at play. By reflecting on the ethical situations facing certain student groups as they planned for data collection, learners considered the perspectives of program participants. For example, in one classroom discussion, after having observed a veteran treatment court program, students hotly debated the merits of collecting baseline data from participating veterans. Through their comments and suggestions, it was evident that students had developed empathy for these veterans and were weighing their desire for baseline data with the risks of retraumatizing these individuals who had recently been arrested and referred to the court. After this animated discussion, the class chose a nondisruptive way to incorporate a brief pretest into their data collection plan. This remedy integrated consideration of interpersonal issues along with optimal data collection.
In addition, writing prompts encouraged students to think about interpersonal dynamics such as empathy, effective communication, problem-solving, and team dynamics. To start a class or to consolidate the thoughts following a discussion, the instructor regularly asked students to write a paragraph on topics like the following: You may now know more about evaluation than your partner program staff currently does. How might you present the logic model so that they will understand and not feel intimidated? How might you couch your recommendations in a way that makes them easier to hear, especially coming from students? (empathy, communication, and negotiation) Why is this program (e.g., a support group for elderly Hmong immigrants) important in our community? How might the evaluation help/harm this program and/or its participants? How should you collect data in a respectful, culturally competent way? (cultural competence, empathy) What might an evaluator do if the program manager does not want to use the survey items that you feel are essential, or wants to change them in a way that creates bias? (conflict resolution, negotiation, and communication) Could teenage girls possibly be harmed psychologically by surveying them about their discomfort with starting middle school? How might your questions be set up to minimize these feelings? (empathy, communication skills) In what ways does your own group work together well? Not so well? What might be improved regarding the way your group works together? (team skills, problem-solving)
Integration of Interpersonal Competencies
Table 5 shows how the course directly and indirectly drew attention to interpersonal skills through various activities. As shown, the highly interactive group work and meetings with agency staff brought out most of the interpersonal competencies. Students prepared logic models and evaluation plans for their agencies with consideration of the staff perspective, that is, the knowledge and experience the visitors brought to the meeting, what they needed to know, and how they might be feeling. Students facilitated shared decision-making in these meetings as they communicated their ideas in a respectful, clear way and then actively listened for feedback from agency staff. Together, students and staff negotiated the outcomes to measure and sketched out a plan to do so. In the individual exercises, students considered client perspectives regarding potential data collection: How might the client react to the proposed data collection method, the survey items, and ethical risks? In doing so, they considered the impacts of power and privilege on the evaluation. Finally, students again became conscious of how they would communicate the evaluation plan as they wrote and visually presented their plans with a professional audience in mind. In sum, this course structure provided opportunities for students to learn some technical evaluation competencies while developing interpersonal competence as they produced meaningful work that was valued by community partners.
Course Learning Activities and Interpersonal Competencies.
Students’ Experiences of the Course
What did students say about their experiences? Over 3 successive years, students in three offerings of the course completed the same anonymous online survey (which had been approved by the Institutional Review Board). In all, 71 students responded to the survey while one completed an alternative online assignment. Sixty-nine percent of respondents (49) were female and 31% male (22); 82% were seniors. They included the following majors (some with double majors): public administration (47), political science (20), environmental policy and planning (15), arts management (10), and other (18).
The survey queried students’ perceptions of personal growth related to the service-learning experience. Warren (2012) conducted a meta-analysis of service-learning outcomes and found that self-reported measures are a legitimate indication of impact since they correlated well with objectively measured learning and that self-reported measures of student attitudes and perspectives. The survey (used over successive course offerings) asked students to rate, on a Likert-type scale, whether the following had increased a great deal or somewhat, stayed the same, or decreased somewhat/a great deal from the beginning to the end of the semester: writing for a professional audience, considering stakeholder interests, taking cultural factors into account, protecting clients from ethical risks, working well as team member, explaining outcome measurement to a novice, working with others to solve a challenging problem, and understanding the cultural context of data collection; community commitment and awareness of nonprofit organizations; and their technical skills and career interest in program evaluation.
An open-ended question asked students to describe how interpersonal skills are utilized in evaluation. Using the Qualtrics platform, 71 responses were gathered into a database and downloaded into SPSS v23. After cleaning and checking the data, simple descriptive statistics for the closed ended items were generated. Responses to the open-ended question on interpersonal skills were coded according to the eight categories of interpersonal competencies described in Domain Five (5.1–5.8) of the AEA Competencies for evaluators (positive relationships, listening, shared decision-making, trust/honesty, awareness of power, communicates clearly, cultural competence, and conflict management). In each response (typically two to three sentences), up to two interpersonal categories were coded. In the few instances when students penned lengthier responses, only the first two topics were coded. Seventy-one responses were analyzed, yielding 111 assigned codes. Following coding, frequency analysis revealed the number of times each interpersonal skill was noted by students.
Overall, students reported a high level of commitment to the service-learning project. Out of a possible high score of 100% commitment, they averaged 89% commitment to the project (SD = 14.5, n = 71). Likewise, near-perfect class attendance throughout the semester revealed motivation of students. Informal observations consistently found that many students worked diligently in groups during class, often having animated and substantive discussions. Finally, interactions outside of class with the instructor and teaching assistants showed that groups sought to deliver useful and accurate plans for their partner agencies. As deadlines approached, students visited office hours frequently and asked for feedback on their work.
Table 6 shows responses to closed-ended questions and the AEA interpersonal competencies that align with each (the data were collected prior to release of the 2018 competencies). Almost every student reported having increased (somewhat or a great deal) consideration of stakeholder needs when developing a work plan, showing that they had grown in the ability to consider different perspectives and utilize shared decision-making. Eighty-eight percent of students had increased awareness of how vulnerable individuals might be impacted by an evaluation, responses that point to awareness of different perspectives along with culturally responsive interactions. Eighty-four percent reported that they had increased their ability to take cultural factors into account in an evaluation somewhat or a great deal. Almost every student (97%) reported having increased the ability to explain outcome measurement to someone that knew little about it, showing that they could communicate meaningfully about evaluation. Three fourths increased their ability to work with others to solve a challenging problem, revealing development in conflict management and fostering positive relationships. Finally, 63% of students believed that their own ability to “work well as a member of a team” had grown. Working as a team member involves several competencies, such as maintaining positive relationships, shared decision-making, building trust, and meaningful communication.
Change in Self-Reported Interpersonal Skills of Students.
Note. AEA = American Evaluation Association.
Table 7 presents the frequency with which the specific interpersonal competencies were reported in the key open-ended question, “in what ways are communication and interpersonal skills used during a program evaluation”? A large majority (42 of the 71) of respondents referred to the importance of clear communication with stakeholders. A sampling of their comments illustrates students’ awareness of the importance of effective interpersonal communication in all stages of the work and in all evaluative settings: “…verbal and nonverbal communication, negotiation, teamwork, and cross-culture skills are needed during the entire evaluation process.” “[Evaluators need]…the ability to interact and integrate ideas with all participants.” “Learning to speak with program participants, the agency staff, and your own group are all important for evaluation.” “Part of a program evaluation is to find the things that are not working well, and that can cause tension, especially if this is not communicated with those involved from the beginning…the evaluator must be open to questions and suggestions and be respectful too.”
AEA Domain 5 Interpersonal Competences in Open-Ended Question.a
Note. n = 71. AEA = American Evaluation Association.
a Up to two responses were coded per respondent.
Responses relating to communication (as shown above) might be expected given that the question included the words “interpersonal” and “communication.” However, Table 7 shows that many wrote nuanced responses that referred to interpersonal competencies needed in evaluation. As shown, 24% of students addressed the importance of building trust and being honest with partner agency staff. Nearly as many (23%) wrote about building positive relationships with stakeholders to the evaluation, and 20% specifically noted the importance of active listening. One wrote, “the evaluator must listen to and understand the agency’s goals in order to better serve them.” Another stated, “I learned to be patient and listen to what the agency had to recommend in the logic model.” Thirteen percent of the students described the need for cultural competence, including empathy for participants from whom data would be collected. Eleven percent talked about the importance of shared decisions. Only three students noted the power dynamics that might exist in an evaluation, expressed thoughtfully by one student who said, “It’s important to work sensitively with the organization so they do not feel threatened by the evaluation.” Finally, only two students discussed conflict management in evaluation in the context of their own group work and how they had to continue to work together for the entire semester, an unusual length of time for a student project. The breadth of their statements revealed a broad awareness of how a variety of interpersonal skills are used in evaluation.
Service-learning often impacts students outside of the realm of course material. In this case, large majorities of students believed the course led them to increase their awareness of various community needs (90%), commitment to the community (76%), and belief in the value of public and nonprofit programs (77%). Not surprisingly, three fourths of students reported increased feelings of confidence and professionalism as they interacted with staff to develop the evaluation plan. This sense of professionalism was also observed in their written communication with agency partners, their dress, conduct during meetings with agency partners in class, and presentations.
Importantly, service-learning increased students’ self-reported interest in the field of evaluation. Seventy-two percent reported that the course increased their interest in developing a logic model in the future, and 59% increased interest in program evaluation as a potential career. One student wrote, “beginning this course, I was not familiar with the field of program evaluation. However, as the class progressed, I realized this is a great opportunity for [a job].” Another wrote, “by the end of this course, I began to consider how to apply the field of program evaluation to my other areas of interest, public administration and urban planning.” Finally, “[program evaluation] has actually become a lot more interesting to me, and it has opened my eyes to more career possibilities.”
Discussion
The need for expanded evaluation capacity puts pressure on educators to prepare individuals with interpersonal competence who can navigate the issues embedded in data-based projects. In community settings, evaluators must manage the give and take of competing goals held by stakeholders and must adapt to situational limitations as the work proceeds. Evaluators plan and lead meetings, share information, organize the tasks of others, identify the unique talents of team members, recognize barriers and overcome them, encourage the contributions of diverse actors, and, at times, refocus the team. The evaluator keeps members of the group on the same page, balances competing interests, and facilitates the project by developing trusting relationships, initiating constructive dialog, listening to stakeholders, and applying conflict resolution approaches.
Whether in undergraduate, graduate, or professional settings, evaluation coursework continues to focus on technical skills. Indeed, Christie, Quinones, and Fierro (2014) reported evaluators’ most frequently completed courses were statistics, research methods, and survey design. These approaches devote little attention to honing learners’ ability to interact productively with others. The reality is that evaluators facilitate interactions and must monitor the human dynamics as well as the technical quality of work completed by a team. “All evaluations require evaluators to interact and/or facilitate interaction at some level with and among people” (Stevahn & King, 2016, p. 69).
Furthermore, cross-cultural skills are important to enable the evaluator to “be responsive and inclusive to the values that those of differing cultures bring into the evaluation of programs” (Christie, 2012, p. 744). Frequently, stakeholders and evaluators differ in terms of age, place of residence, education, race, or social class. Cultural competence leads to better working relationships with a program’s staff or clients, affects decisions about how to collect data, and encourages uses of culturally appropriate methodology and inclusion of stakeholders at all stages. Moreover, evaluations are often political: They have the potential to alter program resources, jobs, and future opportunities. All of these important activities involve interpersonal dynamics; it is essential that learners have a chance to develop interpersonal awareness and competence as they learn to apply evaluation methodologies.
This article describes the promise of a pedagogical approach, service learning, to convey these essential interpersonal skills. Like other “real-world” course designs, service learning contributes to deep learning of evaluation techniques by enabling students to apply evaluation methods while working with community agencies. However, service-learning goes beyond hands-on application of evaluation concepts: it encourages students to reflect on and integrate these experiences. According to student feedback from three sections of the course, students reported learning outcomes such as an increased ability to maintain positive working relationships, listen to others’ perspectives, facilitate shared decision-making, communicate with clarity, and consider cultural context as they developed evaluation plans.
In what ways does service learning, in particular, lend itself to interpersonal skill development? Along with community engagement and a curriculum, an essential pillar of service learning is reflection. This component was an essential tool to bring out awareness of interpersonal dynamics. In this case, the carefully structured reflective writing and group discussion prompts seemed to increase learner awareness of interpersonal dynamics and enabled students to practice them.
Students considered others’ perspectives and could adjust communication or evaluation methods in meaningful ways. These reflections fostered early awareness and practice with skills such as active listening, shared decision-making, negotiation, cultural competence, empathy, perspective-taking, and facilitation.
Moreover, this course delivered other important outcomes aside from technical and interpersonal evaluation skills. Service learning has been shown to broaden and deepen learning along academic, social, moral, personal, and civic dimensions (Hatcher et al., 2004), and this seemed to play out in for students who participated in this course. As they discussed classmates’ evaluation projects, students gained awareness of community needs and the importance of the public-serving organizations addressing them. About half of these students indicated increased interest in the field of evaluation after the course ended. LaVelle (2011) investigated undergraduate students’ interest in evaluation as a career and found that students from a variety of disciplines had relatively equal levels of interest in program evaluation and possessed positive attitudes toward the career field. LaVelle (2011) concluded, “some undergraduates could be purposefully cultivated into future evaluators or users of an evaluative skill set” (p. 372). The same could be said for service-learning as an approach to teach evaluation: It could attract undergraduates into the career path.
Certain aspects of this service-learning methodology might be employed in continuing education settings to help learners develop the technical and interpersonal competencies. Typically, evaluation workshops take place over one or more days with a full slate of technical content to cover. According to service-learning principles, participants might be offered the opportunity to work on an actual program with infusion of reflective discussions and writing prompts. As they learn logic models, outcome measures and statistical analysis, training participants might be asked to reflect on or discuss situational interpersonal dynamics in their own evaluation settings.
While we know it when we see it, further research is needed to measure the actual interpersonal skills demonstrated by evaluators. As the field emphasizes development of effective social interactions, it must continue to operationalize what is meant by interpersonal competence. As shown in nuanced essay responses and self-report by students, nearly, all believed they had increased the ability to empathize with stakeholders in evaluations. Because of this awareness, students shared evaluation information with stakeholders differently than they might have otherwise.
Why don’t more evaluation educators employ the service-learning model? Colleges and universities are increasingly called to expand the number of high-impact experiences available to students (American Association of Colleges and Universities, n.d.). Teaching with a service-learning approach demands a different skill set than traditional classroom-focused models. Service-learning requires faculty and trainers to engage with community partners and to recognize programs that might provide a good match with the course goals. The instructor must be flexible, comfortable with a moderate level of uncertainty, willing to problem-solve as the semester goes along, and generally prepared to support students in a variety of ways. Although challenging to implement, especially at first, a service-learning approach can bring benefits and new learning opportunities for faculty, enabling them to keep current and fresh. It creates a dynamic classroom with extensive give and take and animated discussions. This way of teaching enables faculty to build relationships with students. As coaches and consultants, instructors give essential, anticipated feedback on the student projects.
There is clearly a learning curve involved for instructors when developing this hands-on approach to instruction, and rewards for those who do so may be more psychic than tangible at present. Developing a service-learning course can be time consuming and difficult for educators with uncertain payoff in terms of institutional reward systems (Darby & Newman, 2014). Research into service-learning and the promotion and tenure process revealed a major barrier to the expansion of service-learning in higher education. Although administrators believed the approach had value, “service-learning…is not highly valued in promotion and tenure decisions, unless it takes the form of a peer-reviewed publication” (Frank et al., 2010, p. 7). In fact, the majority of administrators in the study reported that they advised pretenure faculty against engaging in service-learning altogether. Frank et al. (2010) point out that tangibly rewarding the scholarship of engagement has indeed gained traction and that more institutions are interested in high-impact practices.
Conclusion
When it comes to educating evaluators, this article shows that well-designed service-learning experiences have the potential to begin to develop interpersonal skills of learners. In this instance, the service-learning approach led to deep learning of the curriculum, initial development of important interpersonal competencies, budding professional self-confidence, expanded awareness of the community and its people, and for some, kindling of an interest in the field of evaluation.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
