An Exploration of Two Novice Evaluation Educators’ Experiences Developing and Implementing Introduction to Evaluation Courses

Abstract

As the field of evaluation continues to expand, so do evaluation course offerings. However, little is known about the backgrounds of evaluation educators, how they go about planning their courses, challenges experienced, or how they attain competency as teachers. This study offers an initial understanding of novice evaluation educators’ experience in developing and teaching introduction to evaluation courses within a university setting. Factors that influence decisions around course content and pedagogical approaches are also examined. Data collection methods include dialogic conversation notes, reflective journals, course syllabi, and student evaluative feedback. Results indicate that perspectives on the purpose of evaluation and how evaluation should be taught are influenced by evaluation educator professional affiliation, experience with evaluation, and student audience. Implications for teaching and training in evaluation are discussed.

Keywords

evaluation training teaching evaluation evaluation courses introduction to evaluation

Overall accountability demands have spurred the significant expansion of the field of evaluation (Chouinard, 2013). In recent years, the need for evaluation as well as qualified program evaluators has increased markedly given global commitments to accountability in education, health, and science fields (National Academies Press, 2005, 2010). The need for evaluation and qualified program evaluators has also led to growth in regional, national, and international evaluation societies around the world (Rugh & Segone, 2013). An overwhelming majority of evaluators “stumble” into the field of evaluation (Christie, 2003), having begun their professional career journey in fields such as psychology, public health, sociology, and education. As such, a considerable portion of evaluation practitioners do not hold advanced degrees in evaluation and are motivated to seek formal training in order to expand their knowledge and competencies (Christie, Quiñones, & Fierro, 2014). While evaluation is becoming a distinct area of study, with over 48 university-based training programs currently offered (LaVelle & Donaldson, 2010), professional development sessions offered by universities, evaluation societies, and private entities have become increasingly popular as well.

The beauty and challenge of the field’s exponential growth lies in the notion that essentially anyone can be an evaluator. While many evaluation societies, including the American Evaluation Association (AEA; 2017), are adopting statements on necessary evaluator competencies, there are no standard requirements of competency attainment through training or practice to enter the field (Imas, 2010; Stevahn, King, Ghere, & Minnema, 2005). Further, scholarship on the teaching and training of evaluation continues to garner significant traction. Existing research on training in evaluation has focused on the number and content of training programs (Davies & MacKay, 2014; LaVelle & Donaldson, 2010), evaluator course taking and professional practice (Christie et al., 2014), and novice evaluators’ fieldwork experiences (Chouinard et al., 2017; Gredler & Johnson, 2001; Hurley, Renger, & Brunk, 2005). Prior work on teaching in evaluation studies has examined general teaching techniques (Conner, 2001; Oliver, Casiraghi, Henderson, Brooks, & Mulsow, 2008), the use of role playing (Alkin & Christie, 2002; Lee, Wallace, & Alkin, 2007), the need for long-term student evaluation projects (Trevisan, 2002), and techniques for designing evaluation courses and practicums (Levin-Rozalis & Rosenstein, 2003; Morris, 1994).

While there has been a surge in empirical effort to understand how, where, and what is being taught within evaluation courses, less is known about who is teaching, for what purposes, and how. To that end, this article answers the call for more empirical research based on systematic reflection for the purpose of disseminating knowledge gained (Becker & Renger, 2017; Mark, 2008). The purpose of this article is to explore the varied experiences of two novice evaluation educators developing and implementing two separate introduction to evaluation courses. Specifically, this comparative case study includes dialogic conversation notes, course syllabi, reflective journals, and course evaluation reviews as data sources. The following research questions guide this inquiry: (a) How and in what ways did novice evaluation educators make meaning of the development and implementation of introductory evaluation courses? (b) what factors influenced the decisions and pedagogical approaches associated with course design and implementation? and (c) to what extent did educational background, perspectives on evaluation, and research interests have an impact on the novice evaluation educator experiences? In the following sections, we offer a review of the literature on teaching and training in evaluation, thereby underscoring the importance of understanding the factors that influence the decisions and pedagogical approaches associated with course design. Next, we describe the study’s research methodology. The remainder of this article discusses the five themes that emerged during data analysis, a discussion about evaluation teaching and training, and implications for novice and expert teachers of evaluation alike.

Training and Teaching in Evaluation Literature

The most recent literature on evaluation training has focused on categorization and classification of evaluation course offerings, courses taken by evaluators, and training needs of evaluators. For instance, Lavelle and Donaldson (2010) provided evidence for what they purport is a dramatic increase in the number of evaluation trainings within university settings. They also found that while most universities only offer two to three specific evaluation courses, novice evaluators also take a range of quantitative, qualitative, and mixed methodological courses. Christie, Quiñones, and Fierro (2014) have categorized evaluator course-taking patterns into four distinct types of classes including quantitative, evaluation/qualitative, general research/quantitative, and general research. Within this investigation, the researchers revealed that very few evaluators have taken evaluation theory courses or advanced qualitative or advanced quantitative methods courses. An examination of topics covered in introductory evaluation courses affirmed that instructors spent the most time on evaluation approaches, practical issues of conducting evaluations, and evaluation planning (Davies & MacKay, 2014). Less than an hour or no time during courses were spent on the history of evaluation, metaevaluation, and contracts and budgeting. When evaluators were asked to identify competency areas that warranted their additional training, Galport and Azzam (2016) noted that 75% listed metaevaluation and 50% listed quantitative methods. These results are not surprising considering that so little time is spent on metaevaluation in introductory courses and that few evaluators take advanced quantitative methods courses. As a field, we should continue to explore the implications of the lack of variety of courses evaluators take prior to entering the profession.

Since 2004, evaluation scholars have reflected on and made suggestions about evaluation course pedagogy, values, and content. Oliver, Casiraghi, Henderson, Brooks, and Mulsow (2008) argued for the need to discuss pedagogy within program evaluation courses. They concluded that pedagogical choices such as cooperative learning, active learning, and self-directed learning were useful strategies to guide course activities to ensure learning outcomes are met. Thomas and Madison (2010) asserted that social justice should be included as a fundamental value within evaluation courses and field experiences. We agree with their argument that “evaluation students also must be inspired to challenge the status quo, to care about the interests of the disadvantaged, and to uncover weaknesses within the system that contribute to inequities within society” (Thomas & Madison, 2010, p. 571). Morris (1994) called for the need for evaluation educators to focus on teaching future evaluation practitioners both ethical knowledge and ethical practice. He urged instructors to employ multiple strategies that include providing tangible examples, having students keep reflective journals, and attempting to instill moral courage when discussing these issues. While courses are important ways for students to begin their training in evaluation, scholars have long recommended that evaluation students receive hands-on, practical training experiences (Hargreaves, Attkisson, Horowitz, & Sorenson, 1978).

Fieldwork experiences are often incorporated into graduate-level training in evaluation (Darabi, 2002; Trevisan, 2004). While adapting classroom knowledge or theory to the worksites/practice can be challenging for novice evaluators (Boyce & Chouinard, 2017; Chouinard & Boyce, 2017; Hurley et al., 2005), practical experiences for evaluation students including simulations, single course projects, practicums, and role playing (Trevisan, 2004) can have the potential to positively impact evaluation student learning. Long-term evaluation project participation is likely to have the most impact on the development of evaluation abilities (Gredler & Johnson, 2001; Trevisan, 2002). Three key components to successful fieldwork experiences are close communication between students and instructors regarding fieldwork projects (Gredler & Johnson, 2001; Levin-Rozalis & Rosenstein, 2003); the establishment of clear time lines and course expectations; and the development of trust, credibility, and project expectations with clients/agencies (Hurley et al., 2005).

While there have been some studies of the experience of university instructors (Endrizzi, 2011), there are no studies within the literature that examine the experiences of evaluation educators or factors that influence the decisions and pedagogical approaches associated with course design. Scant attention has been paid to backgrounds of evaluation educators, how they go about planning their courses, or challenges experienced. To our knowledge, nothing is known about the experiences of novice educators of evaluation. The current study is pertinent as the field continues to expand and professionalize because more evaluation educators, with varying backgrounds and perspectives on evaluation, will be called upon to teach evaluation courses and training seminars.

Method

Study Design

A comparative case study research design was employed to assess similarities, differences, and patterns across two or more cases (Campbell, 2010; Stake, 2006). Merriam (1998) defines case study as “an intensive, holistic description and analysis of a bounded phenomenon such as a program, an institution, a person, a process, or a social unit” (p. xiii). The two cases in this study were of our experiences in developing and implementing two evaluation courses, each by a different faculty member. We employed a constructivist epistemological approach, thus framing the construction of reality through meanings created both socially and experientially (Merriam, 1998). We developed our own understandings and assigned meanings across multiple sources of evidence. This research design was appropriate as it allowed data to be collected about each course; the design also allowed us to use our own experiences and reflections to assemble in-depth personal views. Doing so helped us better understand the larger issue of novice evaluation educators’ experiences with introduction to evaluation courses and factors that influence course design, implementation, and evaluation. Our ultimate goal for this research was to present case study findings in a reflective narrative format as they are useful when describing lessons learned for the improvement of practice (Becker & Renger, 2017).

Cousins and Chouinard (2012) described the emergence of reflective case narratives as an increasingly prevalent mode of empirical research on evaluation. Unfortunately, lack of attention to narrative structure, such as unclear prioritization, excessive length, and loose organization, often undermines the potentially rich insights a case study can offer (Brandon, 2014); this is especially true of reflective case narratives (Becker & Renger, 2017). As such, scholars have sought to establish clearer guidelines for the organization of reflective case narratives within the field of evaluation (Becker & Renger, 2017; Brandon, 2014; Cousins & Chouinard, 2012). To that end, we used tenets from Becker and Renger’s (2017) framework for structuring the remaining components of our methods section including (a) background and context; (b) motivation, scope, and review of subjectivity; and (c) study strategy, data collection methods, and analysis. Their framework also guided the Lessons Learned section of this article.

Background and Context

This study began as we both accepted and commenced with our tenure-track faculty positions in fall 2015, at the University of North Carolina at Greensboro (UNCG), a midsize public doctoral university with an R2 Carnegie Classification (Carnegie Classification of Institutions of Higher Education, 2016). During an informal conversation at the UNCG new faculty orientation program, we each acknowledged that we were teaching introductory evaluation courses in separate departments to different graduate student audiences within the School of Education (SOE). This was the first time either of us had taught an introduction to evaluation course. Upon this realization, we started sharing ideas about how we planned to approach our respective courses. Furthermore, we constructed a scholarly space where we openly shared research interests, teaching ideas, and professional aspirations. We then reviewed literature on teaching and training in evaluation and did not see our experiences reflected in these scholarly conversations. Consequently, as newer scholars in the academy, we viewed this discovery as a unique opportunity to fill this gap and collaboratively designed a research study to explore how we made meaning and factors that influenced the design and implementation of our respective introductory evaluation courses.

UNCG

The UNCG was originally established in 1891 as the first public college for women in North Carolina. The first African-American students were admitted in 1956 and the university became coeducational in 1962. Women comprise 67% of the student body with ethnic minority students making up 40% of the undergraduate and 36.5% of the graduate student body (UNCG, 2017). The UNCG is the most racially diverse of the UNC system’s historically White campuses and is designated as a minority serving institution. The UNCG SOE’s six departments, Counseling and Educational Development, Educational Leadership and Cultural Foundations, Educational Research Methodology (ERM), Library and Information Studies, Specialized Education Services, and Teacher Education and Higher Education (TEHE), are committed to advancing access to life’s opportunities by providing transformative learning, leading innovation and discovery, engaging communities, and promoting equity and diversity (UNCG SOE, 2017).

ERM 642: Evaluation of educational programs

Ayesha Boyce is an assistant professor in ERM. Over the course of 5 years, program evaluation has gained significant traction within UNCG’s SOE, particularly within the ERM department. Boyce is the second tenure-track assistant professor hired within the department whose research agenda has a focus on evaluation methodology. She was hired to work with ERM colleagues to build a program evaluation concentration for the department’s MS and PhD degree programs.

In fall 2015, Boyce taught Evaluation of Educational Programs which is housed in the ERM department. Graduate students within the SOE and from departments across the university enroll in this service course. For students enrolled in the Educational Research, Measurement, and Evaluation MS and PhD programs, this course is also the first in a sequence of evaluation and research methodology courses. The purpose of the course was to introduce students to program evaluation, evaluation theories, and overview the main commonplaces (Greene, Boyce, & Ahn, 2011) or components of an evaluation. A total of 25 students were enrolled in the course. For all but one student, this course was truly their introduction to program evaluation. Of the 25 students, there were 13 White women, 3 Black women, 4 White men, 2 Black men, 2 international women, and 1 international man. Thirteen students were from the ERM department, three students were from TEHE, three students were from Specialized Education Services, and two students were from Counseling and Educational Development. Four students were not formally enrolled in a UNCG department. Countries of origin represented were Nigeria, Saudi Arabia, Guyana, Bosnia, Turkey, and the United States. Both master’s and doctoral-level students were represented. In addition to articles, the required books for the course were: Alkin’s (2011) Evaluation essentials: From A to Z and Alkin’s (2013) Evaluation roots: A Wider Perspective of Theorists’ Views and Influences, Second edition. Course assignments included a reflective paper, a position paper about a controversial topic within the field of evaluation, an interview with a current evaluator about his or her theoretical and practical leanings, and a group presentation on an assigned data collection methodology. The final project was to complete an evaluation plan and presentation based on a fictional request for proposal (REP) and was completed in groups of four.

Higher education 663: Planning and evaluating programs in higher education

Brian McGowan is an assistant professor in TEHE. In fall 2015, he taught a course titled, Planning and Evaluating Programs in Higher Education, which is housed in the TEHE department. This course fulfills a core requirement for masters’ students in the student affairs administration and higher education (SAAHE) program. The purpose of the course was to provide students with choices available for and processes involved in conducting program evaluation by providing them with hands-on experience conducting a small-scale evaluation study relevant to their interests. In 2015, two professional associations, American College Personnel Association (ACPA)-College Student Educators International and NASPA-Student Affairs Administrators in Higher Education revised their professional competency areas for student affairs educators (ACPA/NASPA, 2015). One of the competency areas focuses on assessment, evaluation, and research, and this course was designed based on the foundational outcomes within this competency area. Twenty-eight students in the SAAHE program were enrolled in the course. With the exception of three students, this course was their first experience with program evaluation in any capacity. Of the 28 students, there were 5 White men, 14 White women, and 9 Black women. In addition to articles, the required book for the course was: Boulmetis and Dutwin’s (2011) The ABCs of Evaluation: Timeless Techniques for Program and Project Managers. Course assignments included a reflective paper, an ongoing reflective evaluation journal, and a small-scale evaluation study that included a prospectus, proposal, class presentation, and a final report.

Motivation, Scope, and Review of Subjectivity

Within the context of reflective case narratives, evaluators are encouraged to be attentive to the underlying impetus for the study, to account for their own subjectivities, and be attentive to how organizations and memberships can shape professional ideologies (Becker & Renger, 2017). We both employ similar philosophies to teaching and motivations for entering the academy. We are both passionate about issues related to diversity, equity, and access, as well as developing a climate of inclusion of historically marginalized populations within contexts of evaluation and higher education. Collectively, being in positions to assist the creation of change at an individual, institutional, and systemic level factored into our decisions to pursue the professoriat. Furthermore, given both of our field’s continued expansion into racially, ethnically, linguistically, economically, politically, and culturally diverse settings, we both believe that it is essential that educators and methodologists alike have a foundation in program evaluation techniques. We both have a social justice orientation that is reflected in the courses we teach and our research interests. While we share similarities and overarching professional beliefs, our experiences, education, and scholarship are markedly unique. We offer a brief overview below.

Boyce earned her PhD in educational psychology with an evaluation specialization from the University of Illinois at Urbana–Champaign. She is committed to developing the technical skills students need for evaluation practice, while also encouraging the adoption of a critical lens (Everitt, 1996; Fay, 1987) and attention to social justice (Thomas & Madison, 2010). She decided to pursue an academic career because she is committed to research on evaluation and to the teaching of the next generation of evaluators. Her hope is that students who complete her courses will have a strong methodological foundation and conduct evaluations that are based in democratic principles; engage with values; and promote equity, fairness, and diversity. She has over 10 years of professional evaluation experience and identifies herself as an evaluator who also conducts research and teaches. She is an active member of AEA.

McGowan earned his PhD in higher education administration from Indiana University and completed some coursework in evaluation during his doctoral training. His motivation to pursue an academic career stemmed from his prior administrative experiences in higher education coupled with research and teaching opportunities acquired in his doctoral program. His interest in teaching an evaluation course in his higher education program stemmed from a doctoral course on program evaluation where he conducted an evaluation of a mentoring program serving students of color. He identifies as a higher education researcher who has conducted evaluations and is not a member of AEA.

Given our understanding of our respective fields, we believe that our portfolios are likely similar to others who have taught introduction to evaluation for the first time. Some will be located with educational psychology, educational research, or other departments within an SOE and will have considerable experience practically and theoretically with program evaluation, the field of evaluation, and their primary allegiance and professional identification will be rooted in an evaluation society. Others will be housed within higher education, economics, psychology, or public health departments and will have affiliations first to their field and corresponding organizations and second to program evaluation. As previously mentioned, little is known about who is teaching introduction to evaluation courses and for what purposes. Moreover, to our knowledge, virtually nothing is known about populations unaffiliated with AEA that teach these courses. Our motivation for this study was to gain an in-depth understanding of similarities and differences in our experiences, thought process, and pedagogies, as well as the content and implementation of our introduction to evaluation courses.

Study Strategy, Data Collection, and Analysis

Data collection methods

Course syllabi, dialogic conversation notes, reflective journals, and course evaluations were collected to improve interpretability and meaningfulness of our novice evaluation educator experience. This approach was used to capitalize on strengths and counteracting biases in each of these data sources (Greene, Caracelli, & Graham, 1989). Data collection took place over the course of 6 months, August 2015–March 2016. Authors were the sole data collectors and reviewers. A description of each data collection method is included below.

Course syllabi

In August 2015, we began the study by sharing syllabi with one another to gain a sense of the content we were planning to cover in our courses. Both syllabi included standard content such as class format; course objectives and expectations; grading criteria; assignments; and statements about academic integrity, civility, and special needs.

Reflective journals

In September 2015, we began reflective journals. Skolits, Morrow, and Burr (2009) have suggested that “a meaningful strategy for self-reflection would be a systematic review of each activity of the evaluation, the demands that resulted, and the effectiveness of the associated role responses” (p. 291). To make the reflections as systematic and useful as possible, we wrote in our individual reflective journal once a week throughout the duration of the fall 2015 semester and specifically reflected upon each weekly class. Prompts were used with the following purposes to (1) gather information throughout the semester relevant to how the course was planned, designed, implemented, and evaluated with particular attention to logistics, pedagogy, and strategies; (2) capture information related to challenges and triumphs during the course; and (3) capture the experience of implementing an introduction to evaluation course for the first time from the instructor’s perspective. We also wrote final reflections in March 2016 that included us making suggestions for the next iterations of our courses.

Dialogic conversation notes

Also beginning in September 2015, we began our weekly dialogic conversations. During these conversations, we were able to voice our experiences, share resources, provide sources of support, express concerns, and offer constructive feedback as we prepared for the following week. For instance, there were a few topics where McGowan would share his lesson plans with Boyce for feedback prior to teaching that week. There were also moments where we cocreated and shared potential classroom engagement activities and would modify them to fit our respective audiences.

Course evaluations

In February 2016, we shared our feedback from student course evaluations with one another. These data allowed us to compare student perceptions of the course, course activities, and learning outcomes with our own and to develop an understanding of the quality of the courses.

Data analysis

We analyzed our respective reflective journals, dialogic conversation notes, and course evaluation feedback throughout the data analysis process. In case study research, data collection and analysis often happen simultaneously (Merriam, 1998). Therefore, we were purposeful in beginning data analysis while data collection was still underway. For instance, throughout the semester, we independently coded notes from the dialogic conversations before completion of data collection. We then came together to discuss our themes for the notes. Furthermore, we planned future dialogic conversations according to what we found in previous conversations. For example, our weekly dialogic sessions served as an opportunity for us to write memos, connect our experiences back to the literature, and begin noticing patterns across our data. However, we did not begin data analysis in earnest until summer 2016.

Initial independent analysis

Throughout the month of June 2016, we reviewed all individual data sets (course syllabi, reflective journals, and course evaluations) separately. We utilized Ryan and Bernard’s (2003) article as a guide as we searched for themes. We began by independently checking the dialogic conversation notes for accuracy and met briefly to discuss and summarize two dialogic conversation meetings where the notes were lacking. We then individually checked our own structured reflections and syllabi for accuracy in dates, events, referenced time frames, and other markers of origins. After accuracy checks, we manually analyzed each set of data individually.

This study also used analysis techniques that are consistent with case analysis research, specifically, searching for patterns, constructing themes, and identifying frequencies within the data (Stake, 1995). Stake (1995) notes “two strategic ways that researchers reach new meanings about cases are through direct interpretation of the individual instance and through aggregation of instances until something can be said about them as a class” (p. 74). Data were examined and searched for certain words, phrases, events, and themes that stood out. This was a three-step process to (a) search through data for regularities and patterns, as well as topics represented in the data, (b) write down topics and phrases to categorize these patterns/themes, and then (c) search for disconfirming data. Additional information about the coding and analysis process can be found within the respective sections below.

Reflective journal themes

Following the individual data set analyses for accuracy, all themes across all data sets were reviewed for our cases individually. We each began with the weekly reflective narratives because they served as our primary data source. We individually reread them line by line to generate initial codes. For Boyce, codes such as “running out of time,” “over planning,” “too much time on activities,” and “need to find other ways to fill class time” were themed as time constraints and course content. For McGowan, codes such as “following up with a peer about potential activities,” “contacting peers to guest lecture,” and “appreciating unsolicited advice” were themed as usefulness of informal/peer support. Codes from our reflective narratives were compared and recurring patterns within the data were sought out, which led to initial themes. Boyce’s themes from her reflective narrative were: need for formal support, anxiety about course implementation, time constraints and course content, and concerns about course management and peer review. McGowan’s themes from his reflective narrative were: usefulness of informal/peer support, concerns about lack of experience with evaluation, evolution of course content throughout the semester, formal mentoring, and semester time line and higher education student needs.

Dialogic conversation notes

We met to analyze and discuss the conversation notes together. After the discussion, we came to consensus on the following themes for the dialogic conversation notes: decisions around the amount of reading and reading content, reflections on syllabi changes during future iterations of the course, appropriateness of course texts and assignments, working through teaching and classroom management challenges, and aligning content to our respective field’s competencies and standards.

Course evaluation themes

We individually came up with themes for our own course evaluations. Boyce’s course evaluation themes included instructor enthusiasm, in-class storytelling, and changes for next iteration. McGowan’s course evaluation themes were: interaction with student, successful activities, and changes for next course iteration. We then each wrote up a brief summary of our individual cases based on the aforementioned individual themes.

Final themes

We then met in July 2016 to compare syllabi, course evaluations, and individual case study summaries including themes from the course evaluations and reflective narratives. Initial interrater reliability was high and consensus was achieved through dialogue. Course syllabi were compared and notes about similarities and differences in activities, content, and objectives were made together. We also reviewed and compared course evaluation feedback and scores and noted similarities and differences in students’ recommendations and comments. Including our previously constructed dialogic notes themes, we then constructed, refined, and combined themes across both cases and all data. Criteria for final themes included repetition, or topics that occur and reoccur (Bogdan & Taylor, 1975), searching for similarities and differences by making systematic comparisons across units of data (Glaser & Strauss, 1967), and linguistic connections or searching for words and phrases (i.e., “since” and “as a result”) which often indicate causal relations (Ryan & Bernard, 2003). The final themes were refined by combining themes with related meanings and deleting themes that had little data to support them or that had multiple counterexamples or disconfirming data. We finally reordered them into one cohesive final set of five themes. The findings are outlined in the Findings section.

Findings

Our analysis yielded five broad themes related to our experience as novice evaluation educators with developing and implementing introduction to evaluation courses:

While we both were apprehensive about teaching an introduction to evaluation course, the source of our anxiety was dependent upon our educational backgrounds and professional experiences.

There was a great deal of variation between objectives and content across courses. This variation stemmed from differences in the fields of higher education and program evaluation, student needs, and professor expertise and interest.

Pedagogical decisions and course content were restricted by semester time constraints.

It was decided that both courses would undergo significant changes in their second iteration.

Informal and formal structures were key in supporting both professors throughout the semester.

Each of these themes is presented in greater detail below.

Apprehension to Teaching Evaluation From Divergent Sources

We both had different apprehensions as we prepared our respective courses for the first time. The source of our anxiety was dependent upon our educational background and professional experiences. Having never served as the sole instructor of a course, Boyce was concerned about her limited experiences in the classroom. With over 9 years of practical experience, including 25 evaluations conducted and formal training in evaluation theory, she considered herself a competent evaluator. Additionally, she had never developed or assessed student knowledge for program evaluation. She commented in her reflective journal:

I am incredibly nervous to begin teaching Evaluation of Educational Programs next week. This is pretty much the fulfillment of a dream I envisioned for myself seven years ago. I have never taught a course from start to finish before. I endeavor to bring the field of evaluation to life in a way that engages and excites my students. I hope I can cultivate understanding with a small bit of passion. While I consider myself an experienced evaluator, ardor and content knowledge can’t replace strong pedagogy. (Boyce, reflective journal entry, one week before the start of the semester)

As McGowan had taught several courses prior to fall 2015, his concerns were rooted in his novice evaluation skills. He was confident in his ability to construct a syllabus, develop lesson plans, and assign and grade homework. However, he was less sure of his ability to explain logic models and evaluation plan development. His individual apprehensions for teaching evaluation are also exemplified in his initial journal entry:

I have to openly acknowledge that this course [Planning and Evaluating Programs in Higher Education] is a growth area for me personally. I view myself as a higher education researcher who has conducted a few evaluations. It is one thing to conduct evaluations; it is another thing to teach students how to do the same. After being hired, I was asked to revamp this course to focus more on evaluation. I am entering my third year as a tenure-track university professor and have no qualms about teaching; however, I know this specific course is going to really keep me on my toes. (McGowan, reflective journal entry, one week before the start of the semester)

As illustrated in the quotes above, we both had concerns about teaching our introduction to evaluation courses. These concerns also played out in our dialogic conversations. Notes from the week of September 13, 2015, include discussion around the following topics: classroom management, logic model exercise planning, and assignment grading. During this week, Boyce asked for advice about students spending time off task during classroom activities and for general grading rubric examples. Then, Boyce refreshed McGowan’s logic modeling skills and shared a few examples and exercises that he could use in class. Throughout the rest of the semester, many of our conversations began with “check-ins,” where we relayed our concerns and also shared some of our class-related triumphs and potential areas of improvement. Regardless of our individual angsts, we embraced these new challenges.

Objective and Content Variation Between Courses

There was a great deal of variation between course content and objectives across our courses. While we both sought to ensure that our students understood the foundational process of how to plan and implement an evaluation, we went about doing so differently. Our variation stemmed from differences in our fields and department, student needs, and our own expertise and interests. For example, the ERM MS and PhD program offers (a) introduction to evaluation, (b) practicum in evaluation, (c) advanced evaluation theory, (d) policy in evaluation, (e) collaborative approaches, and (f) culturally responsive approaches evaluation courses. The “introduction to evaluation” course is one of the six evaluation courses offered within the department. It is meant to serve as a foundational course with the expectation that over three fourths of the enrolled students, no matter the department, will likely also at least take the practicum in evaluation course where they spend the entire semester conducting an evaluation—from meeting their stakeholder, to developing a plan, collection of data, and reporting findings via a written and oral report. Within the SAAHE program, “Planning and Evaluation Programs in Higher Education” is the only evaluation course and must fill a number of objectives within one semester including program evaluation, program planning, and assessment. As such, we each had slightly different objectives and expectations for our courses. During our fourth dialogic conversation, we each shared our overarching goals for our courses. Our review of course syllabi and corresponding dialogic conversation notes led us to the following reflections:

I would like my students to be able to do the following by the end of my course: (1) compare and contrast major theories/approaches of program evaluation, (2) begin to determine appropriate approaches and questions for an evaluation based on purpose, context, and stakeholders, (3) be able to construct logic models and evaluation framework matrices, and link evaluation questions to appropriate data collection methods and (4) become sophisticated consumers of program evaluations reported by others. (Boyce, reconstruction of dialogic conversation based on notes and course syllabus)

I want my students to do the following: (1) identify and understand principles, models, and applications of program planning and program evaluation, (2) develop an understanding of program evaluation practice by designing and conducting an evaluation for a higher education and student affairs functional area, (3) develop an understanding of ethical, political, and cultural issues facing program evaluators, and (4) write an evaluation report to meet stakeholder information and decision making needs. (McGowan, reconstruction of dialogic conversation based on notes and course syllabus)

In terms of course content, Boyce’s course began with the infamous “evaluation of a cookie” exercise where students are given different types of cookies and asked to develop criteria on which to judge their quality (Preskill & Ruff-Eft, 2015). She then introduced program evaluation as a field with guiding principles (AEA, 2004), standards (Yarbrough, Shulha, Hopson, & Caruthers, 2011), and competencies (AEA, 2017). Then, over the course of a month, students were introduced to evaluation theory trees to illustrate the multiplicity of perspectives and the broad range of theoretical and historical influences that define the field (Alkin, 2013; Mertens & Wilson, 2012). The second half of the semester was spent with hands-on activities surrounding the technical aspects of evaluation planning such as logic modeling, stakeholder involvement, key question development, various data collection and analysis methods, and reporting. The final project was to create an evaluation plan in response to a fictional RFPs she created. McGowan’s course was more practical and hands-on throughout the entire semester. Given that the course was based on ACPA/NASPA’s competency areas focused on assessment, evaluation, and research, he began by emphasizing how this course was essential in their training as future higher education and student affairs practitioners (ACPA/NASPA, 2015). He then continued with a discussion about how to conduct an evaluation, the role of stakeholders, how to choose evaluation models, the varied types of data sources, the ways to analyze data, and ways to construct evaluation reports. After hearing about the “cookie exercise,” he also conducted this activity with his students midway through the semester. His students’ final project was to design and conduct a mini program evaluation for a higher education stakeholder. The differing course content highlights the diversity in priorities, expectations, and goals each of us had for our introduction to evaluation courses.

Semester Time Constraints

Pedagogical decision-making based on course content and student feedback was a recurring theme in both of our narratives throughout the semester. In reflecting on the nexus between time and content, we oftentimes compromised content more than we would have liked given time constraints. Embedded within our narratives was a constant reflection of having to make pedagogical decisions based on time constraints. Each of us reflected on how we felt restricted given the vast amount of content that needed to be covered in the following ways:

I am unsure that there is enough time in the semester to cover all of the content in this course in a meaningful and thorough way. The first half of the course is spent defining terms and concepts, and framing evaluation. Each of Alkin’s (2013) branches (methods, use, and, values) only get one class period with an additional week for the Mertens and Wilson’s (2012) social justice branch. I could really spend a whole semester examining, comparing, and contrasting major theories. The second half of the class is more about the technique of evaluation planning including stakeholder-evaluator relationships, question development, data collecting, and logic models. It’s really a great deal to cover in a short period of time. (Boyce, reflective journal entry, four weeks into the semester)

I just finished doing a mid-semester check in with students and it seems like they are really enjoying the evaluation project. While gathering informal feedback, they mentioned wanting more time in class to work with their group members. As a result, I am now going to build in time to work on their evaluation project groups. I struggled with this decision as I am now compromising other course content in a few places. I want to fit everything in and give them a chance to have some group time in class. Definitely a fine line for sure. (McGowan, reflective journal entry, nine weeks into the semester)

In any course or professional development workshop, the instructor must be conscious of the time line in conjunction with objectives, lecture, and activities. In addition to the reflections above, we spent time during dialogic conversations discussing ways to manage time while delivering the required course content. One conversation halfway into the semester resulted in Boyce modifying her syllabus because she felt it was important to have two guest speakers video conference into her course to speak with her students about data visualization and the social justice “branch” (Mertens & Wilson, 2012) of evaluation approaches. These changes forced her to eliminate one lecture and two class activities. Surprisingly, this theme remained consistent across both courses. While objectives and content differed, we both, along with our students, felt pressure to squeeze course content into a mere 15 weeks.

Significant Course Changes for the Second Iteration

At various points throughout the semester, we documented ways in which we could improve our respective courses. Various mentors had encouraged us to jot down notes about specific activities and how they went throughout the semester. For example, in Boyce’s course, it was evident that her students struggled with the concepts of ontology, epistemology, axiology, and methodology and their relationship to evaluation practice. After her initial lesson on these “ologies,” she felt that students needed additional time and discussion around this topic so ultimately she came back to those concepts in subsequent weeks. Her class on logic modeling went very well. Students were enthusiastic and engaged. It was evident that the technical skills needed to construct logic models had resonated with them. During a dialogic conversation, Boyce decided to adapt her logic model exercise and implement it in McGowan’s course. End of semester dialogic notes echoed Boyce’s concerns about the balance between evaluation theory and technical practice. Further, her final reflective journal entries captured the decision to address this issue in subsequent iterations of the course:

Having had some time to reflect on the course, I believe it makes sense to change it for the upcoming year. Six weeks is not enough time to do the evaluation theory and approaches content justice. We could also probably spend two whole class periods on logic models. So while I believe that evaluation is much more than a technocratic endeavor, I believe it is my responsibility to ensure that students are able to think well and become consumers of evaluation. Having 14 weeks dedicated to the technical aspects of evaluation should prove to be a valuable foundation for incoming evaluation MS and PhD students and enough about the technique of the practice of evaluation for students who are taking this course as an elective. There is a piece of me that is hesitant to let theory go, but I will spend some time thinking about how to talk about values, cultures, bias in knowledge production and different approaches throughout the course. I will also suggest the development of an advanced evaluation theory course for evaluation students in our PhD program. (Boyce, reflective journal, final entry)

McGowan ultimately decided that the higher education competency of program planning was not adequately covered in his course. Although program evaluation skills can transfer to program planning, he determined that additional explicit attention was needed in the next iteration. During his final reflective journal entry, he stated:

I plan to move this course in a different direction to align with the field of higher education’s professional competencies. I spent a significant amount of time on program evaluation and not enough on program planning this semester. Program planning and program evaluation could be two separate courses so I need to figure out a way to balance the content a little more. The competency area focuses on the ability to design, conduct, critique, and use various methodologies and the results obtained from them, to utilize processes and their results to inform practice. Although this current course accomplishes some of these tasks, I want to be more intentional about making these connections moving forward. (McGowan, reflective journal, final entry)

As mentioned above, we reflected on ways to best adapt our courses to prepare for the second iteration the following fall semester. Several weeks after the conclusion of our respective courses, both of our concerns about content were confirmed in student feedback from course evaluations. In Boyce’s course, it became evident that students wanted to spend less time on evaluation theory and more time on the practical aspects of evaluation. A few student comments included:

I am not too interested in theory, but I understand the class needed to cover it. (ERM 642 Student)

It may be helpful to spend more time on practical components of evaluation. Theory is important but it felt like the majority of class. (ERM 642 Student)

At the beginning it was a challenge to put pieces together. I didn’t know how to “use” theory or how it fit with evaluation practice. (ERM 642 Student)

Although McGowan’s students appreciated the small-scale evaluation project, they wanted more information on the program-planning process, hence the course title. Two students stated:

I would have liked to learn more about program planning. (HED 663 Student)

I do not feel like I learned the “program planning” piece of the title of the course. (HED 663 Student)

Ultimately, we both decided to make changes to our courses for the next iteration.

Necessity of Informal and Formal Support Structures

Given our varied experiences and background with conducting evaluations and teaching graduate-level courses, it was important to have both informal and formal support structures in place. Fortunately, we both were assigned formal mentors within our respective departments. These mentors offered advice on all aspects of our professional journey including teaching, research, and service. Further, we both were observed halfway into the semester and were given formal formative feedback about our teaching style, pedagogy, course activities, and student engagement. In addition to formal support, we also informally developed the professional relationship that lead to this article. Both types of support were important for our growth as novice evaluation educators. We offer examples of these support mechanisms in the reflections below:

Only two weeks left in the semester and I am ready to hesitantly cheer and sigh a breath of relief. The weekly meetings with McGowan have proved to be incredibly useful. He provided me with tips on classroom management and student engagement that have proved to be invaluable. I was also able to meet regularly with the course’s previous instructor and she provided me with suggestions for activities and potential discussion points. I am not sure what I would have done without them. I also had support from my department chair who met with me monthly to talk about teaching, scholarship, and service. Finally, the associate dean for Academic Affairs and Student Services observed my class, conducted a focus group with my students, and wrote a mid-term semester report on my teaching strengths and weaknesses. All of the previous have allowed me to develop and improve throughout the semester. (Boyce, reflective journal entry, 12 weeks into the semester)

This is my third year teaching and I have never had to ask for this much help. I do realize this is the first course that I have ever taught outside of my primary focus area. Boyce’s knowledge of evaluation proved beneficial particularly when brainstorming ways to deliver some of the course content. This research process served as a unique opportunity to improve my teaching, expand my knowledge of evaluation, and to bond with a new faculty colleague [Boyce] as we support each other through the promotion and tenure process. Going through this reflective process was extremely valuable and its impact cannot be underestimated. (McGowan, reflective journal entry, final entry)

Many of our reflective narratives and dialogic conversations notes described how we supported each other and received support from multiple others at various points during the semester. For example, notes from a dialogic conversation the week of August 31, 2015, included review of each other’s syllabus and conversation surrounding how to best utilize our mentors. We often chatted about how fortunate we were to be afforded with the levels of support offered at our institution.

Lessons Learned and Implications for Practice

Becker and Renger (2017) state “a significant contribution to the field are the lessons learned as they help move evaluation theory and practice forward” (p. 147). We reflected on two key lessons learned from engaging in this process: (1) balancing course content and (2) support structures.

Balancing Course Content

Our experiences with course development and implementation were directly impacted by the extent to which we were familiar with evaluation theory and practice and needs of our intended audiences. Initially, we both had conversations about what content and literature should be included within each of our courses. At the outset, Boyce was concerned that well-known evaluation theorists were not included within McGowan’s course. McGowan countered that it was necessary for him to teach the course in such a way that it was most useful to future higher education leaders (not professional evaluators), who may one day be called to conduct internal evaluations of their programs or who will have to read external evaluation reports. McGowan and many of Boyce’s students felt that less time within her course should have been spent on evaluation theory. Further, she ultimately realized that half a semester is not enough time to include meaningful engagement with evaluation theory while trying to teach foundational evaluation methodology. Additionally, McGowan’s students would have liked a little bit more information about different approaches to evaluation and program-planning techniques. Thus, for both courses, it ended up being a matter of balancing content with time constraints, student needs, and instructor interest and expertise.

Evaluation is conducted as a service to something else and is born of the need to rate quality, systematically analyze outcomes, and provide formative feedback (Shadish, Cook, & Leviton, 1991). As such, evaluation in and of itself is an interdisciplinary endeavor. While it is important that as a field we continue to professionalize and distinguish ourselves, there is much that can be learned from other fields about the training of practitioners. While evaluation is not solely a technocratic endeavor (Schwandt, 2009), we believe it is useful for evaluation educators to first begin with focusing on the development of sound practice, that is meaningfully informed by evaluation purpose, stakeholder voice, underlying values, and method choice instead of trying to squeeze in evaluation theory. Then, as students of evaluation continue to grow in expertise, they can more deeply engage with a variety of evaluation theories and approaches.

Support Structures

Research in other professional fields has found that most new faculty members are not adequately prepared or mentored (Austin, 2003; Siler & Kleiner, 2001). We found this to be true, however, our lack of expertise differed. Similar to findings from other fields, we primarily learned how to teach our introduction to evaluation courses by doing it (Knight, Tate, & Yorke, 2006). Our experience was unique in that we both had a number of formal and informal support structures available to us that we utilized as necessary. It would be ideal if all novice evaluation educators had practical support for pedagogy as well as content. Those who pursue a PhD in an evaluation-related field will likely have little teaching experience as those programs are primarily for graduate students. As such, there are few formal teaching opportunities. While Boyce was comfortable with evaluation theory and practice, she found support for her pedagogical skills through formal observation of her course and meetings with her department chair. Meetings with McGowan also assisted in the development of her class management skills, as he was able to give her feedback during their informal dialogic conversations.

Those instructors who are similar to McGowan, and we suspect that many faculty (e.g., tenure-seeking, adjunct, and clinical) within higher education, economics, psychology, or public health departments similar to him, have foundational pedagogical skills but may lack deep practical experience in program evaluation. McGowan’s pedagogical skills were stronger than his evaluation content knowledge. As such, Boyce offered support as he developed his syllabus and class activities throughout the semester. His course was also observed midway through the semester, and his assigned department mentor provided him with feedback. By partaking in this study, we were able to not only become better teachers of evaluation but also formed a colleagueship that will assist us in the navigation of the promotion and tenure process together. As faculty and instructors with diverse backgrounds are tasked with developing and teaching introduction to evaluation courses in a variety of programs and departments, adequate support structures that attend to their unique needs are essential. Given that not all introduction to evaluation instructors are members of AEA, the field will need to be innovative in strategies for how such professionals can be encouraged, understood, and supported.

Study Limitations

Becker and Renger (2017) have argued that reflection is key to articulating experiences that are often difficult to express. Further, Stake (1975) has shown that case study methodology, which often utilizes reflection about experiences, can be useful to extract tacit knowledge or “unexpressible associations which give rise to new meanings” (Stake, 1975, p. 6, as cited in Becker & Renger, 2017). We have employed reflection and narrative in this study to provide a first-person view of two novice evaluation educators as we developed, implemented, and evaluated our introduction to evaluation courses. This study is not generalizable in the traditional sense nor is it meant to be. We recognize that our two experiences are not all-encompassing or representative of all novice evaluation educators. Nevertheless, we have provided an initial synopsis of how educational background, perspectives on evaluation, and research interests have had an impact on our experiences. We hope that this article will be a catalyst for further research on evaluation educators, especially novices.

Future Research and Concluding Thoughts

As the field of program evaluation continues to grow, it is imperative to understand who and what perspectives are shaping the next generation of evaluators. Previous research has demonstrated that there is a breadth of diversity in the training new evaluators receive. Little is known about training that new evaluation educators receive, their perspectives, or opinions. Recent studies have begun to broach these topics with AEA members (Davies & MacKay, 2014). However, studies are needed that include participants similar to McGowan, who do not consider AEA their “professional-home” organization. Finding instructors of evaluation courses from university websites and contacting them directly could be an initial first step. Future studies should also examine the difference and similarities in perspectives on the purpose of evaluation and how evaluation should be taught from the viewpoint of evaluation instructors. While parallel research examines what should be taught, employing future research as outlined above will allow the field to better understand those who are developing and implementing such curricula and how to reach them so that we can continue to examine best practices for evaluator education and professional development.

Reflection is an honored tradition in evaluation and evaluators are encouraged to reflect upon our practice (Kallemeyn, Hall, Friche, & McReynolds, 2015). Novice faculty have found that reflection, included within larger narrative inquiry research, can assist with the venture of grooming scholars into faculty (Cole, McGowan, & Zerquera, 2017). We encourage novice and seasoned evaluation educators alike, to not only familiarize themselves with teaching evaluation literature but also to systematically reflect upon their experiences as they develop and implement evaluation courses.

Footnotes

Authors’ Note

Second Resubmission to the American Journal of Evaluation, Teaching and Learning in Evaluation Section on December 9, 2017.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

ACPA/NASPA. (2015). Professional competency areas for student affairs practitioners. Washington, DC: Author.

Alkin

M. C.

(2011). Evaluation essentials: From A to Z. New York, NY: Guilford.

Alkin

M. C.

(2013). Evaluation roots: A wider perspective of theorists’ views and influences (2nd ed.). Thousand Oaks, CA: Sage.

Alkin

M. C.

Christie

C. A.

(2002). The use of role-play in teaching evaluation. American Journal of Evaluation, 23, 209–218.

American Evaluation Association. (2004). Guiding principles. Retrieved from http://www.eval.org/p/cm/ld/fid=51

American Evaluation Association. (2017). Invitation to provide feedback on the AEA draft competencies. Retrieved from http://www.eval.org/p/cm/ld/fid=472

Austin

A. E.

(2003). Creating a bridge to the future: Preparing new faculty to face changing expectations in a shifting context. The Review of Higher Education, 26, 119–144.

Becker

K. L.

Renger

(2017). Suggested guidelines for writing reflective case narratives: Structure and indicators. American Journal of Evaluation, 38, 138–150.

Bogdan

Taylor

S. J.

(1975). Introduction to qualitative research methods. New York, NY: Wiley.

10.

Boulmetis

Dutwin

(2011). The ABCs of evaluation: Timeless techniques for program and project managers (3rd ed.). San Francisco, CA: Wiley.

11.

Boyce

A. S.

Chouinard

J. A.

(2017). Moving beyond the buzzword: A framework for teaching culturally responsive approaches to evaluation. Canadian Journal of Program Evaluation, 32, 266–279.

12.

Brandon

P. R.

(2014). Book review: Participatory evaluation up close: An integration of research-based knowledge. American Journal of Evaluation, 35, 291–297.

13.

Campbell

(2010). Comparative case study. In Mills

A. J.

Eurepos

Wiebc

(Eds.), Encyclopedia of case study research (pp. 174–175). Thousand Oaks, CA: Sage.

14.

Carnegie Classification of Institutions of Higher Education. (2016). Retrieved from http://carnegieclassifications.iu.edu/

15.

Chouinard

J. A.

(2013). The case for participatory evaluation in an era of accountability. American Journal of Evaluation, 34, 237–253.

16.

Chouinard

J. A.

Boyce

A. S.

(2017). The impact of practice on pedagogy: Reflections of novice evaluation teachers. Canadian Journal of Program Evaluation, 32, 280–287.

17.

Chouinard

J. A.

Boyce

A. S.

Hicks

Jones

Long

Pitts

Stockdale

(2017). Navigating theory and practice in evaluation fieldwork: Experiences of novice evaluation practitioners. American Journal of Evaluation, 38, 493–506.

18.

Christie

C. A.

(2003). What guides evaluation? A study of how evaluation practice maps onto evaluation theory. New Directions for Evaluation, 97, 7–36.

19.

Christie

C. A.

Quinones

Fierro

(2014). Informing the discussion on evaluator training: A look at evaluators’ course taking and professional practice. American Journal of Evaluation, 32, 274–290.

20.

Cole

E. R.

McGowan

B. L.

Zerquera

D. D.

(2017). First-year faculty of color: Narratives about entering the academy. Equity and Excellence in Education, 50, 1–12.

21.

Conner

R. F.

(2001). Evaluation training in the U.S. An overview of training options and an illustrative training course. Journal of International Cooperation in Education, 4, 39–51.

22.

Cousins

J. B.

Chouinard

J. A.

(2012). Participatory evaluation up close: A review and integration of research-based knowledge. Greenwich, CT: Information Age.

23.

Darabi

(2002). Teaching program evaluation: Using a systems approach. American Journal of Evaluation, 23, 219–228.

24.

Davies

MacKay

(2014). Evaluator training content and topic valuation in university evaluation courses. American Journal of Evaluation, 35, 419–429.

25.

Endrizzi

(2011). Savoir enseigner dans le supérieur: Un enjeu d’excellence pédagogique [Knowing how to teach in higher education: An issue of pedagogical excellence]. Dossier d’actualité veille et analyses, 64, 1–24.

26.

Everitt

(1996). Developing critical evaluation. Evaluation, 2, 173–188.

27.

Fay

(1987). Critical social science. Cambridge, MA: Polity.

28.

Galport

Azzam

(2016). Evaluator training needs and competencies: A gap analysis. American Journal of Evaluation, 38, 80–100. doi:10.1177/1098214016643183

29.

Glaser

B. G.

Strauss

(1967). The discovery of grounded theory: Strategies for qualitative research. New York, NY: Aldine.

30.

Gredler

M. E.

Johnson

R. L.

(2001). Lessons learned from the directed evaluation experience. American Journal of Evaluation, 22, 97–104.

31.

Greene

J. C.

Boyce

A. S.

Ahn

(2011). Values-engaged, educative evaluation guidebook. University of Illinois, Urbana-Champaign. Created and produced with funds from the National Science Foundation. AEA eLibrary. Retrieved from https://higherlogicdownload.s3-external-1.amazonaws.com/EVAL/Values%20Engaged%20Educative%20Guidebook.pdf?AWSAccessKeyId=AKIAJH5D4I4FWRALBOUA&Expires=1521478184&Signature=y0ROyP235TG0l%2Bvfm9ONAWIlI8g%3D

32.

Greene

J. C.

Caracelli

V. J.

Graham

W. F.

(1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11, 255–274.

33.

Hargreaves

W. A.

Attkisson

C. C.

Horowitz

M. J.

Sorensen

J. E.

(1978). The education of evaluators. In Attkisson

C. C.

Hargreaves

W. A.

Horowitz

M. J.

Sorensen

J. E.

(Eds.), Evaluation of human service programs (pp. 479–486). San Francisco, CA: Academic.

34.

Hurley

Renger

Brunk

(2005). Learning from a challenging fieldwork evaluation experience: Perspectives of a student and an instructor. American Journal of Evaluation, 26, 562–578.

35.

Imas

L. G.

(2010, May). The movement for global competencies for development evaluators. Annual meeting of the Canadian Evaluation Society, Victoria, BC, Canada.

36.

Kallemeyn

L. M.

Hall

Friche

McReynolds

(2015). Cross-continental reflections on evaluation practice: Methods, use, and valuing. American Journal of Evaluation, 36, 339–357.

37.

Knight

Tait

Yorke

(2006). The professional learning of teachers in higher education. Studies in Higher Education, 31, 319–339.

38.

LaVelle

J. M.

Donaldson

S. I.

(2010). University-based evaluation training programs in the United States 1980–2008: An empirical examination. American Journal of Evaluation, 31, 9–23.

39.

Lee

Wallace

T. L.

Alkin

(2007). Using problem-based learning to train evaluators. American Journal of Evaluation, 28, 536–545.

40.

Levin-Rozalis

Rosenstein

(2003). A mentoring approach to the one-year evaluation course. American Journal of Evaluation, 24, 245–259.

41.

Mark

M. M.

(2008). Building a better evidence base for evaluation theory. In Smith

N. L.

Brandon

P. R.

(Eds.), Fundamental issues in evaluation (pp. 111–134). New York, NY: Guilford.

42.

Merriam

S. B.

(1998). Qualitative research and case study applications in education. San Francisco, CA: Jossey-Bass.

43.

Mertens

D. M.

Wilsons

A. T.

(2012). Program evaluation theory and practice: A comprehensive guide. New York, NY: Guilford.

44.

Morris

(1994). The role of single evaluation courses in evaluation training. New Directions for Program Evaluation, 62, 51–59.

45.

National Academies Press. (2005). Rising above the gathering storm: Energizing and employing America for a brighter economic future. Retrieved from http://www.nap.edu/catalog.php?record_id=11463

46.

National Academies Press. (2010). Rising above the gathering storm, revisited: Rapidly approaching category 5. Retrieved from http://www.nap.edu/catalog.php?record_id=12999

47.

Oliver

D. E.

Casiraghi

A. M.

Henderson

J. L.

Brooks

A. M.

Mulsow

(2008). Teaching program evaluation: Three selected pillars of pedagogy. American Journal of Evaluation, 29, 330–339.

48.

Preskill

H. S.

Ruff-Eft

(2015). Building evaluation capacity: Activities for teaching and training (2nd ed.). Thousand Oaks, CA: Sage.

49.

Rugh

Segone

(Eds.). (2013). Voluntary organisations for professional evaluation (VOPEs): Learning from Africa, Americas, Asia, Australasia, Europe and Middle East. Geneva, Switzerland: UNICEF.

50.

Ryan

G. W.

Bernard

H. R.

(2003). Techniques to identify themes. Field Methods, 15, 85–109.

51.

Schwandt

T. A.

(2009). Globalizing influences in the Western evaluation imaginary. In Ryan

K. E.

Cousins

J. B.

(Eds.), The Sage handbook of educational evaluation (pp. 19–36). Thousand Oaks, CA: Sage.

52.

Shadish

W. R.

Cook

T. D.

Leviton

L. C.

(1991). Foundation of program evaluation theories and practice. Newbury Park, CA: Sage.

53.

Siler

B. B.

Kleiner

(2001). Novice faculty: Encountering expectations in academia. Journal of Nursing Education, 40, 397–403.

54.

Skolits

G. J.

Morrow

A. J.

Burr

E. M.

(2009). Reconceptualizing evaluator roles. American Journal of Evaluation, 30, 275–295.

55.

Stake

R. E.

(1995). The art of case study research. Thousand Oaks, CA: Sage.

56.

Stake

R. E.

(2006). Multiple case study analysis. New York, NY: Guilford.

57.

Stevahn

King

J. A.

Ghere

Minnema

(2005). Establishing essential competencies for program evaluators. American Journal of Evaluation, 26, 43–59.

58.

Thomas

V. G.

Madison

(2010). Integration of social justice into the teaching of evaluation. American Journal of Evaluation, 31, 570–583.

59.

Trevisan

M. S.

(2002). Enhancing practical evaluation training through long-term evaluation projects. American Journal of Evaluation, 23, 81–92.

60.

Trevisan

M. S.

(2004). Practical training in evaluation: A review of the literature. American Journal of Evaluation, 25, 255–272.

61.

University of North Carolina at Greensboro. (2017). About UNCG. Retrieved from https://admissions.uncg.edu/discover-about.php

62.

University of North Carolina at Greensboro School of Education. (2017). About UNCG SOE. Retrieved from http://soe.uncg.edu/about/mission-and-vision/

63.

Yarbrough

D. B.

Shulha

L. M.

Hopson

R. K.

Caruthers

F. A.

(2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage.