Abstract
The profession of evaluation continues to grow, generating more demand for evaluation services than can be fulfilled by the supply of well-trained evaluation practitioners. In this brief forum article, we discuss the current state of evaluator education in the United States and the ways in which university-based programs support the development of evaluation practitioners and scholars. We then discuss the opportunities and challenges university-based programs face, focusing on program design, the challenges of being an evaluation-focused faculty member, and the specific needs of evaluation graduate students. We conclude by providing a research agenda for evaluation scholars interested in advancing the knowledge base about university-based evaluator education.
Keywords
The United Nations’ (2014) Resolution A/RES/69/237 formally designated 2015 as the International Year of Evaluation, helping formalize evaluation as a mainstream activity across the world. A wide range of stakeholders require information about their programs, policies, and interventions to better understand and document their effects. Because these stakeholders look to well-prepared and qualified evaluators to help answer questions about their programs, the preservice education of evaluators is a critically important topic for the evaluation community. Yet, there are significant gaps in the published knowledge base about evaluator education and the ways in which evaluators are prepared for practice (Gullickson et al., 2019; King & Ayoo, 2020). We offer the following perspective, drawing from both existing scholarship and our experience in developing, implementing, and expanding evaluator education programs in U.S. university systems. We recognize that our expertise is limited to the U.S. context, though it is likely that the challenges and opportunities facing university-based evaluator education programs are similar across the world.
In this article, we will share a brief overview of evaluator education in universities, including discussions of graduate degrees, graduate certificates, and professional development opportunities. We will follow with a behind-the-scenes discussion of the opportunities and challenges in developing and implementing evaluator education programs, including the expectations and demands placed on faculty, adjunct instructors, and students. We conclude with specific research questions about evaluator education.
Evaluator Education
The preservice education and socialization of evaluators are critical issues facing universities and voluntary organizations for professional evaluation across the world. Evaluator education spans continents, domains of practice, and educational specializations, and the definitions of evaluator education program used by U.S. researchers (Altschuld et al., 1994; Engle, Altschuld, & Kim, 2006; LaVelle, 2019) may not apply well to universities across the world, in part because of naming conventions and/or collaborations (e.g., Canada’s Consortium of Universities for Evaluator Education [CUEE]) and different educational systems. A scan of published papers and publicly available directories suggests that a number of universities outside the United States offer graduate degrees with an emphasis on evaluation—primarily master’s degrees, with a few offering doctoral degrees. There are a number of master’s and doctoral programs in Canada (CUEE, 2018), for example, and mainly master’s degree programs in Europe (Beywl & Harich, 2007; Friedrich, 2016) and Africa (Basheka & Byamugisha, 2015).
There are three primary ways in which universities help individuals scaffold their evaluation skills and knowledge: university degrees, university certificates, and professional development. We recognize the value of single-course experiences in evaluator education (Morris, 1994), though stand-alone courses are not the focus of this article. We define an evaluator education program as a series of two or more courses with evaluation in the title and an explicit evaluation focus (LaVelle, 2019; LaVelle & Donaldson, 2010).
Undergraduate and Graduate Degrees
Research has suggested that a significant proportion of undergraduate students are interested in evaluation as a profession (LaVelle, 2011) and that evaluation can be taught at the undergraduate level (McShane et al., 2015). A recent analysis suggests that approximately 50% of public and private universities in the United States offer evaluation-related courses for undergraduate students (LaVelle et al., 2020). It does not appear, however, that any universities award an undergraduate degree with a specialization in evaluation.
At the graduate level, more universities across the world are offering evaluation-focused advanced degrees than at any other time in history (LaVelle, 2018, 2019), though there appears to be no consensus on what an evaluation curriculum is or should be. Different perspectives are reflected in course titles and descriptions of program goals and in the values and skills of program alumni. Broadly, there is little consensus about the degree to which evaluation should be taught as (1) a methodological subspecialization, (2) an applied consulting practice, (3) a profession drawing from and informing scholarly theory in other disciplines, (4) a stand-alone discipline with its own knowledge and research base, or (5) a combination of all the above.
Divergent views about curriculum are common in professions, as they work to solidify their knowledge base (Pavalko, 1988) and distinguish between what should be included in formal coursework and what can be learned on the job (Schiro, 2008). The 2018 Directory of Evaluator Education Programs in the United States (LaVelle, 2018) illustrates the range of options available, including online and distance education options, topical courses in evaluation, course offerings in methods and inquiry, and a broad range of options for culminating projects or papers. Master’s programs generally require 30–42 credits to complete, and doctoral programs require 72 credits. Most programs require 6–12 credit hours in courses with “evaluation” in the title, with the remainder distributed across inquiry/research methods courses and context-specific content courses.
Certificate Programs
Because graduate certificate degrees in evaluation have received less scholarly attention than master’s and doctoral programs, they are a bit of a mystery in the evaluator education community. Little historic data exist about their development, implementation, completion requirements, and influence on the field of evaluation. LaVelle (2014) found that certificate programs with an evaluation focus required a median of 13 credit hours to complete and typically required one course each in evaluation, inquiry methodology, and a content area such as education, leadership, social work, or health.
Professional Development
Some universities periodically offer in-service professional development workshops on evaluation-related or evaluation-specific topics. These take the form of lectures, half- or full-day workshops, or consecutive days of linked workshops (e.g., The Evaluator’s Institute, Minnesota Evaluation Studies Institute), can be in-person or online, and fall into broad categories of basics, advanced, special topics, methods, and evaluation in context. A sample of covered topics includes foundations of evaluation; using theory to improve evaluation practice; utilization-focused evaluation; developmental evaluation; theory-driven evaluation; budgeting; positionality in evaluation; experimental and quasi-experimental design; qualitative methods and analysis; introduction to survey methods; positive approaches to evaluation; culturally responsive evaluation; Latinx evaluation; lesbian, gay, bisexual, transgender, and queer–inclusive evaluation; cost–benefit analysis; needs assessment; educational evaluation; policy evaluation; and health program evaluation.
University-based evaluator education programs have much to offer would-be and current evaluation practitioners and can provide educational opportunities to meet a range of needs. They have evolved to offer more nuanced, specific, and adaptive courses addressing the needs of a global society interested in credible and actionable evidence. Ongoing work is needed to ensure that course offerings and curricular systems align with the larger needs of organizations and societies and that program participants are prepared to meet contemporary and future challenges.
Challenges and Opportunities in University-Based Evaluator Education Programs
Evaluator education programs are complex and exist in a wide range of interrelated educational, political, financial, and social ecosystems. Like those in other disciplines, evaluation programs are subject to institutional pressures, enrollment requirements, requirements for publication and external funding, faculty perspectives, disagreements, turnover, and professional pressures about accreditation and certification. While these challenges are endemic in university life and unlikely to change, there are several evaluation-specific challenges we must discuss openly. We focus here on challenges with the existing curriculum, revisiting “core” classes, development of between-university collaborations, the preparation of evaluation faculty, and challenges specific to graduate students.
Aligning Programs and Courses With Evaluator Competencies
In the formal professions (e.g., law, medicine, and clergy) and semiprofessional disciplines (e.g., teaching, social work, epidemiology, etc.; Etzioni, 1969; Pavalko, 1988), university-based education programs align with professional criteria and standards, clearly articulating what students need to be able to know, feel, think, and do upon graduation, and have designed their courses and other educational experiences accordingly (Schiro, 2008). The field of evaluation, however, has not yet aligned its curricula and courses with the competency frameworks and other foundational documents recognized by the professional evaluation associations (Stevahn et al., 2005a, 2005b).
A starting point in the United States is the American Evaluation Association’s (AEA) Evaluator Competencies. The AEA formally adopted this framework in 2018, providing a framework for evaluator education centered around five domains: 1.0. Professional Domain—what makes evaluation distinct as a profession. 2.0. Methodology Domain—technical aspects of inquiry, such as framing questions, designing studies, sampling, collection, analyzing data, interpreting results, and reporting findings. 3.0. Context Domain—understanding the unique circumstances and settings of evaluations and their users/stakeholders. 4.0. Management Domain—logistics, such as determining and monitoring work plans, time lines, resources, and other components needed to complete and deliver a study. 5.0. Interpersonal Domain—human relations and social interactions that ground evaluator effectiveness.
In addition to the competencies, the AEA (2018) provides a set of guiding principles for evaluators and a statement on cultural competence in evaluation and supports the third edition of the Program Evaluation Standards (Yarbrough et al., 2011). These provide an ethical and procedural foundation to help evaluator education programs develop curricula and learning experiences to produce practitioners who provide high-quality evaluation services to a wide variety of clients and consumers. While the degree to which evaluator education programs use these documents in their design and implementation is not immediately apparent, programs that link curricula to these types of documents are well-positioned to explain their program’s value to administrators, students, and other stakeholders (Bright & Richards, 2001). In addition, they will be better prepared than programs at competing institutions if a voluntary accreditation or certification system is ever implemented in the United States (Altschuld & Engle, 2015; McDavid & Huse, 2015).
Revisiting Core Program Courses
Each university course is a source of revenue, and revenue influences decisions by deans and chairs (Bright & Richards, 2001). Partially in response to this pressure, faculty continuously advocate for courses in their area of expertise. For what kinds of courses, should evaluation faculty advocate beyond the ubiquitous Evaluation 101? Trevisan (2004), for example, argued for practical experience in evaluation. We agree about its importance for both master’s and doctoral students and think faculty should incorporate it into every curriculum. Just as important are formal courses in evaluation theory—the core knowledge base of evaluation—and evaluation-specific research courses for doctoral students, so they can expand the knowledge base of evaluation (Mark, 2008). We see courses on teaching/training/building capacity as value added for any university-based program.
Evaluation theory
Shadish (1998) described evaluation theory as “who we are,” as it both informs the core identity of an evaluator and distinguishes the evaluator from a basic or applied research methodologist. Indeed, evaluation theory gives evaluators language to use in practice, forms the core of many evaluation-specific debates, supplies the field’s knowledge base, and provides the face evaluators share with the rest of the professional world. As well, it prescribes processes for evaluation practice and implicitly refers to the kinds of outcomes expected from an evaluation process. Evaluation theory, as conceptualized by Shadish et al. (1991) and by Christie and Alkin (2013), is not well understood in an educational setting and work by LaVelle (2018) suggests a paucity of theory-related courses. Evaluation theory has been critiqued as overly focused on North American evaluation (Alkin, 2013), leading to fruitful debates across the world. Indeed, recent work by Cloete (2016) on African-centric evaluation theory and discussions by Chilisa et al. (2016) on decolonizing evaluation have led to a broader perspective, and we are excited to see what the future holds.
Research on evaluation (RoE)
RoE seems uncommon in evaluator education programs, and there are little data on student training in basic research on the applied field of evaluation. Data from the 2018 Directory of Evaluator Education Programs in the United States suggest it is a rare topic, though RoE may be embedded in courses under different titles. Scholars have explored different topics in this realm, including an interest/needs analysis (Szanyi et al., 2012), a review of published articles (Coryn et al., 2017), and an entire volume of New Directions for Evaluation (Brandon, 2015), so there is no lack of ideas on what could be explored by graduate students and faculty. The lack of funding, however, is a challenge. It seems that if the purpose of a doctoral degree is to advance knowledge in a field (Rhodes, 2001), the hallmark of an evaluation-focused doctoral degree should be a dissertation on evaluation itself. We find Mark’s (2008) framework of RoE helpful as we guide doctoral students toward conducting empirical RoE on contexts; activities; outcomes; and professional issues using descriptive, predictive, classificatory, or values-based inquiry.
Building capacity and teaching
Evaluators in practice are often called to help build the evaluation capacity of their stakeholders (Baizerman et al., 2002; Preskill & Boyle, 2008), but it is not apparent whether university programs prepare students for this educative component. It is likely that evaluators working to build capacity rely on existing skills augmented with activities found in resource books such as Preskill and Russ-Eft’s (2016) sourcebook of training activities for evaluation. Graduate students may participate in programs like Preparing Future Faculty or in teaching-specific courses or may find a faculty mentor, but these alone seem insufficient to prepare students for the complex world of evaluator education.
Developing Interuniversity Collaborations
Interuniversity collaborations provide an interesting opportunity in evaluator education. In Canada, the CUEE (2018) is a university-level partnership in which members share resources and work to raise awareness of educational opportunities. CUEE architects designed the collaboration with an eye toward preparing graduates to attain the Canadian Evaluation Society’s Credentialed Evaluator [CE] designation. Members provide graduate-level evaluator education experiences that align with the CE’s academic requirements and capitalize on relationships with influential stakeholders.
It is not immediately apparent, however, whether CUEE is a collaboration at the student level, with courses from one institution truly equivalent to those offered elsewhere or with students able to take courses from a different institution and have them count toward a degree at their home university. Our sense is that this type of reciprocity is not likely—though, if implemented widely, such arrangements could be very beneficial for students, especially in the age of COVID-19. The challenge is this: Faculty work only for their home university and are limited in the amount of time for commitments outside of their teaching, research, and service. An offer to teach even a single course at another institution must be handled carefully, so that the faculty member does not accidentally go against their contractual obligations. For the university itself, there is no incentive to support building something that will split tuition revenue between institutions, even if such an arrangement would benefit a particular field or profession.
Being an Evaluation Faculty
Course and curriculum implementation is dependent on the instructors available and willing to lead students through learning processes and experiences. Instructors are generally either tenured/tenure-track faculty or adjunct instructors, the major difference being their role in the university. The primary role of tenured/tenure-track faculty is to conduct research, followed by teaching and service. Activities are weighted and valued differently across universities and departments, and it is critical for faculty to know the emphasis placed on each criterion, so they can use their time wisely. The primary role of adjunct faculty is something other than research or teaching at the university, with teaching often a way of supplementing existing income.
Tenured/tenure-track faculty
Compared with appointments in other fields, tenure-track appointments in evaluation are scarce. This is partly because few universities have departments of evaluation or give degrees specifically in evaluation. Most tenured or tenure-track faculty with evaluation interests and expertise must thus master a second area of expertise such as education, public health, psychology, public policy, organizational behavior, human resources, or public administration.
For tenured/tenure-track faculty, the primary indicator of productivity is the quality and quantity of peer-reviewed publications in valued journals. Monographs, books, and book chapters are typically valued less than articles, though the relative weight of each contribution depends on the university and the review committee. We recommend that tenure-track faculty focus as much as possible on peer-reviewed articles in high-impact journals, as this type of scholarship is valued most by promotion and tenure review committees (Marshall & Rothgeb, 2011; Park & Gordon, 1996). It is important to point out that many universities do not count evaluation reports, primarily because they are not peer-reviewed and often do not contribute to the general scholarly knowledge base. This disincentivizes evaluation faculty to conduct evaluations unless they bring a significant amount of money to the university (e.g., external grants and contracts), contribute to the college/department’s finances (e.g., indirect cost recovery), or directly support one or more graduate students.
Tenure-track faculty must show a record of research productivity, and failure usually means a faculty member’s contract will not be renewed. At many institutions, untenured faculty are encouraged to aim for at least two peer-reviewed publications per year, though this expectation can vary by department and college. This is a major undertaking in any field and substantially more difficult in evaluation because much of our scholarly work focuses on improving evaluation practice, which is of interest to only a limited number of journals.
A tenure award guarantees perpetual job security except in cases of university-level structural changes (e.g., a department closing) or in extreme cases of unethical faculty behavior. This means faculty can settle on a research agenda so long as they publish approximately two peer-reviewed works per year (Greene et al., 2008). This time can be invested in students, research, curricular development, service, or any number of other areas, including applications for external grants and funding. The rewards are great for both individual and the field when an evaluation scholar earns tenure, though untenured faculty often report that meeting the scholarship requirements of most universities is quite difficult because of these structural expectations.
Adjunct instructors
As the number of residential and online evaluation degrees and professional development programs has grown, so have the opportunities for faculty willing to work part-time on short-term contracts. Many evaluation practitioners now supplement their incomes by teaching single courses or workshops at universities, professional association meetings, and professional development institutes such as the Evaluators’ Institute and the International Program for Development Evaluation Training.
For experienced evaluators and evaluation educators, these nontenure-track appointments seem more accessible now than in the past (Rhoades, 1996) and often offer flexibility for faculty to maintain up to full-time employment beyond teaching their workshops and classes. The workshop compensation rate is high compared with the amount of work required to make the experience successful, though the compensation rate for university adjunct teaching is consistently low. These contract appointments, however, tend to be unstable, and faculty are often not guaranteed courses when enrollments dip. Contract faculty are also more likely to be terminated, especially for less than outstanding performance, than evaluation faculty who hold more stable tenure-track or tenured university appointments (Rhoades, 1996).
Flourishing as Evaluation Graduate Students
Students considering applying for graduate education should carefully consider their interests and goals (Donaldson & Christie, 2006; LaVelle, 2011) and recognize that programs aim to prepare students for a career primarily focused on evaluation practice, evaluation teaching and scholarship, or a hybrid career, which seeks a balance of practice, teaching, and scholarship. The path students desire to walk will help guide their choice of courses and professional experiences. Finding good mentorship is a challenge for many graduate students, but particularly so for evaluation students, as their interests are often in the practice of evaluation and not in research. Students must remember that the primary responsibility of tenured/tenure-track faculty is research, so faculty might be more helpful in preparing students for a research career than for applied evaluation practice. Likewise, faculty ought to be aware that not all of their students—or even many of them—want to become faculty and should tailor their mentorship appropriately.
There is good news for graduate students: Graduating master’s and doctoral students in evaluation typically have many employment opportunities. Ongoing efforts at the University of Minnesota have documented over 1,000 job descriptions for evaluators, including solicitations for full-time and part-time evaluators, for evaluation faculty, and for practitioners with evaluation skills in addition to another area of expertise (e.g., health, policy, education) as well as requests for proposals and requests for qualifications. This project suggests a wide range of pay scales, consistent with the range of contexts in which evaluators work (Dewey et al., 2008; Nielsen et al., 2018). Job descriptions often call for evaluators with expertise in inquiry design and methodology and sometimes include an interest in a specific theoretical perspective, such as theory-driven, utilization-focused, empowerment, systems thinking, or culturally responsive evaluation. A more recent development has been the call for evaluators with the ability to build evaluation capacity within teams and organizations (Preskill & Boyle, 2008), suggesting that graduate students would benefit from the ability to teach about evaluation and provide technical assistance around it.
Doctoral students interested in a career in evaluation practice should think about how their dissertation processes and products can help open doors for their future career. Similarly, students preparing for an academic career should think about how their RoE can help distinguish them from their colleagues, and how they will convert their dissertation studies into further scholarship. Building from the ideas of Krathwohl and Smith (2005), our perspective is this:
(1) The dissertation is a high-quality, research-based, original contribution to the knowledge base of a field or discipline. (2) Students working toward a doctoral degree with a specialization in evaluation should conduct an RoE dissertation. (3) Dissertations take a long time to finish and a great amount of effort and resources. (4) An intrepid doctoral student will find a way to leverage their dissertation topic and process into the next stage of their career. (5) A good advisor will keep numbers 1–4 at the forefront of the student’s mind.
New Directions for Research on Evaluator Education
We began this article by discussing how relatively little is known about evaluator education programs and systems (King & Ayoo, 2020). We conclude by suggesting that evaluator education systems and processes are a vibrant area of future research for those passionate about both evaluation and education. We foresee several paths forward for strong RoE agendas and offer the following prompts for collaboration and critique.
− To what degree are the existing evaluator education programs aligned with the competencies identified by the professional evaluation associations? − For programs that offer a PhD with a focus on evaluation, what kinds of topics are students exploring for their dissertation studies? Are the results of those studies getting published in peer-reviewed journals? − What kinds of mentoring and educational supports are necessary for students participating in online or distance evaluation courses? How are they similar or different from in-person students? − For faculty, to what degree are their applied evaluation projects valued in the traditional academic environment? − To what degree are evaluation faculty publishing in the mainstream evaluation journals compared with other topical journals? − Who is teaching evaluation in university programs? Are they tenured, tenure track, or adjunct? What are their affiliations and qualifications, what do their courses focus upon, and to what degree do they blend scholarship and practice? − What effects do evaluator education programs and courses have on students? − How can evaluator educators assess their impact on students? − Should a doctoral degree with an emphasis in evaluation be a practice-based degree or a research degree?
In this article, we discussed the current state of university-based evaluator education programs and then identified a number of areas filled with both promise and challenge—specifically, the curriculum, collaborations, faculty, and students. In sum, the profession of evaluation continues to grow and to offer practicing evaluators a wide range of career opportunities. Evaluation faculty positions for those willing to work on a short-term contract basis also appear to be plentiful. However, tenure-track and tenured faculty positions specifically in evaluation continue to be scarce, and evaluation scholars often need to develop at least one additional area of expertise to build the kind of scholarly record and have the citation impact needed to advance through the promotion systems at most research universities. Finally, evaluation students now have many options to choose from when they decide to pursue additional evaluation education. We hope that by openly discussing the structures and processes of university-based evaluator education and clarifying some of the clandestine elements of the system, future evaluator educators—and program designers—will be aware of the opportunities and challenges inherent in the task of preparing an enthusiastic population of practicing evaluators.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
