Abstract
Although research on evaluation instruction has increased in the past decade, little is known about evaluation education in Council on Education for Public Health (CEPH)–accredited schools of public health and master of public health (MPH) programs. In this study, instruction of competencies for program evaluators as enumerated by Stevahn, King, Ghere, and Minnema and pedagogies used by instructors to cover the competencies in the MPH degree within CEPH-accredited schools and programs are investigated. To understand these questions, a two-phase sequential mixed method design was used where the first was a review of 156 schools and programs and their 652 evaluation course offerings and the second consisted of administration of a survey to the 580 instructors who taught evaluation courses coded as either partially or fully focused on evaluation, of which 128 responded (22.07%). Results suggest evaluation courses in CEPH-accredited schools and programs cover many evaluator competencies, albeit to various degrees, predominately through lectures and reading materials.
Background and Introduction
The Council on Education for Public Health [CEPH] is an independent agency recognized by the U.S. Department of Education that accredits schools of public health and public health programs that educate students at the bachelor, master, and doctoral levels in the United States, Canada, Lebanon, Mexico, and the West Indies. 1 CEPH (2017a, 2017b)-accredited schools of public health and master of public health (MPH) programs are intended to prepare students for careers in public health practice and research, including evaluation of health and public health initiatives. To accomplish these goals, programs “…are free to have one degree offering or many,” while schools “…typically offer a broader range of different degree options” (CEPH, 2018b, n.p.).
Accredited schools and programs educate and provide skill development for graduate students in foundational public health knowledge, foundational competencies, and concentration competencies (CEPH, 2016). By graduation, all MPH and doctor of public health (DrPH) graduate students should be grounded in the profession and science of public health and factors related to human health and the foundational public health knowledge. According to CEPH, all MPH graduates should demonstrate competency in evidence-based approaches to public health, public health and health-care systems, planning and management to promote health, policy in public health, leadership, communication, interprofessional practice, and systems thinking. CEPH further specifies competencies unique to students graduating with a DrPH which include data collection and analysis; leadership, management, and governance; policy and programs, and education and workforce development (CEPH, 2016). All graduate students must also attain competence specific to their concentration area (e.g., epidemiology, health education, and biostatistics)—these competencies are not only unique to concentration areas but also to each school and program. While the extent to which evaluation is emphasized varies widely within concentrations, MPH students in CEPH-accredited schools of public health and MPH programs are required to receive instruction related to evaluation of health and public health initiatives prior to graduating (CEPH, 2016).
Recognizing the importance of evaluation in public health, the Institute of Medicine [IOM] reported in The Future of Public Health (1988) that evaluation is an essential function of public health practice. Likewise, the Centers for Disease Control and Prevention (CDC, 2013), in its “10 Essential Public Health Services,” indicated that evaluating the “…effectiveness, accessibility, and quality of personal and population-based health services” (n.p.) is an indispensable public health activity that all communities should undertake. Furthermore, in 1999, the CDC further emphasized the importance of evaluation as a central function of public health in publishing A Framework for Program Evaluation in Public Health (1999). In alignment with the recognition of the importance of evaluation in practice, competency sets for new or established public health professionals include evaluation as a competency (e.g., Applied Epidemiology Competencies, Certified in Public Health, Certified Health Education Specialist, and Master Certified Health Education Specialist). In addition, through the MPH Core Competency Model (2006), the Association for Schools and Programs in Public Health (ASPPH) devised and recommended the use of evaluation-related competencies within curriculum standards to equip MPH students with the ability to manage, design, conduct, report, and use evaluations when planning and implementing public health interventions.
Unlike public health, the American Evaluation Association [AEA] only recently formally endorsed professional competencies. In summer 2018, the AEA Board officially approved a set of competencies for program evaluators. These competencies were developed and vetted through a multistep member engagement and revision process. Several decades ago, the Guiding Principles for Evaluators (Newman, Scheirer, Shadish, & Wye, 1995) and the Program Evaluation Standards (Joint Committee on Standards for Educational Evaluation, 1981) were published as an effort to professionalize the field. These guiding principles and standards form the basis of several efforts to develop evaluation competencies (notably, Stevahn, King, Ghere, & Minnema’s [2005a, 2005b] Essential Competencies for Program Evaluators, and Stufflebeam and Wingate’s [2005] Self-Assessment of Program Evaluation Expertise). Although they were published 13 years ago, the revised Essential Competencies for Program Evaluators (Stevahn et al., 2005a, 2005b) and Self-Assessment of Program Evaluation Expertise (Stufflebeam & Wingate, 2005) are the most recently published peer-reviewed set of evaluation competencies that evaluators and instructors had the opportunity to leverage for their activities prior to summer 2018.
Both Stevahn, King, Ghere, and Minnema (2005a, 2005b) and Stufflebeam and Wingate (2005) classified evaluation competencies into domains based on evaluation practice. Stevahn et al. (2005a, 2005b) grouped 71 competencies into six evaluation practice domains, which serve as the foundation for this research: (a) professional practice (6 competencies), (b) systematic inquiry (20 competencies), (c) situational analysis (12 competencies), (d) project management (12 competencies), (e) reflective practice (5 competencies), and (f) interpersonal competence (6 competencies). Stufflebeam and Wingate (2005) organized 77 competencies into eight domains: (a) standards/metaevaluation (6 competencies), (b) evaluation approaches and models (8 competencies), (c) evaluation of particular areas (12 competencies), (d) designing evaluations (9 competencies), (e) evaluation methods and techniques (23 competencies), (f) providing evaluation training (6 competencies), (g) professional development (7 competencies), and (h) developing a view of evaluation (8 competencies).
Despite the importance of evaluation competence in public health practice and research, the quality, consistency, and content of evaluation education in public health are nearly undocumented. One exception, however, is Fierro and Christie’s (2011) study in which they examined the prevalence and general content of evaluation-related coursework in CEPH-accredited schools and programs for MPH students concentrating in health education or epidemiology. Given the current lack of knowledge regarding instruction about evaluation in public health coursework, it is unclear whether the IOM vision of the “future of public health” including a substantive evaluation function can be realistically fulfilled. Systematically derived knowledge about evaluation instruction in CEPH-accredited schools of public health and MPH programs would contribute to calls for research on evaluation training and education that have been made for more than a decade (Coryn et al., 2016, 2017; Henry & Mark, 2003; LaVelle & Donaldson, 2010; Mark, 2007; Stufflebeam & Coryn, 2014) and inform the ongoing dialogue regarding evaluator competencies (AEA, 2015). 2
The purpose of the current study was to investigate instruction of evaluation competencies in the MPH degree within CEPH-accredited schools of public health and MPH programs. Specifically, the research questions addressed were: What competencies for program evaluators as enumerated by Stevahn et al. (2005a, 2005b) are taught in the MPH degree within CEPH-accredited schools’ and programs’ evaluation courses? What instructional strategies (i.e., pedagogies) are used by instructors in the MPH degree at CEPH-accredited schools and programs to cover the evaluator competencies?
Method
A sequential mixed method design, in which a qualitative phase preceded and informed a quantitative phase, was used to address the research questions. The qualitative phase consisted of a document analysis and the quantitative phase consisted of an Internet survey.
Phase I: Document Analysis
Sample
All evaluation courses taught at the MPH level, within CEPH-accredited schools and programs served as the population of interest. Courses could be required, recommended, or offered as electives. At the time of the investigation (2016–2017), the CEPH website indicated 156 CEPH-accredited schools (n = 49) and programs (n = 106) in the United States (n = 151), Canada (n = 3), Lebanon (n = 1), and West Indies (n = 1). One school and one program in Puerto Rico and one school in Mexico were excluded because their websites were not in English and five schools and programs were excluded because they are based outside the United States. A total of 148 academic institutions located in the United States were therefore included in the study population.
A review of university registrar websites; graduate catalogues; CEPH-accredited schools of public health; and MPH programs’ websites, course websites, and course syllabi led to the identification of 652 courses with evaluation in the title or course description. Thirty courses were not focused on program or intervention evaluation and were therefore excluded from the course sampling frame. An additional 42 courses were excluded from inclusion since contacting course instructors was an essential element of Phase II. Of these 42 courses, e-mail address could not be located for all instructors (n = 25), no course instructor was listed online (n = 16), and an instructor passed away (n = 1). Accounting for these exclusions, 580 courses were included in the population sampling frame.
Instrumentation
A piloted, standardized data extraction form was used to record and code information about evaluation courses from CEPH-accredited schools of public health and MPH programs meeting the inclusion criteria. At the school and program level, this included the name of the university or college, the name of CEPH-accredited school of public health or MPH program, and whether it was a school or program of public health. For each evaluation course, the data abstraction form was used to record various instructor and course characteristics (e.g., course description/course objectives, level of evaluation focus [i.e., full focus, partial focus, or no focus], textbooks used). When available, course syllabi also were obtained.
Procedures
Over a 10-month period, two reviewers used the data extraction form to record and code information from university registrar websites; graduate catalogues; CEPH-accredited schools of public health; and MPH program websites, course abstracts, course websites, and course syllabi, and they coded the course as full focus, partial focus, and no focus on evaluation as well not program or intervention evaluation, insufficient information to make a determination, and no description provided. 3 Prior to coding, in order to familiarize themselves with the coding procedures and “…identify and reduce areas of ambiguity” (Coryn, Noakes, Westine, & Schröter, 2011, p. 210), the two reviewers engaged in a calibration procedure wherein both coded evaluation course information for two schools and two MPH programs accredited by CEPH. Based on the results of the calibration procedure, the data extraction form was modified.
Once the data extraction form was finalized, the two reviewers extracted data from documents and coded each of the courses independently. Interrater agreement for the independent coding procedure for exact agreement across the level of evaluation focus (i.e., full, partial, and none) was po = .88, and accounting for the probability of chance agreement was κ = .73. Following the independent coding procedure, the two reviewers engaged in a consensus-seeking procedure wherein they deliberated incongruities and arrived at mutually agreed upon codes for each course.
Analysis
An Excel database was used to record, store, and retrieve extracted data. Content analysis techniques were applied in order to efficiently and effectively describe information extracted from the coded documents and descriptive statistics were used to determine frequencies, averages, and ranges of responses.
Phase II: Internet Survey
Sample
All courses (N = 580) identified for use in the document analysis served as the population of interest in Phase II. Serving as proxies for courses, instructors who most recently taught each course served as the sampling frame.
Instrumentation
An Internet survey was developed to gather information pertaining to course content (specifically evaluator competencies) and instructional pedagogy. Prior to implementation of the survey, the primary investigator conducted a four-phase pilot test: (1) Two advisors and one content expert provided feedback on the measurement, clarity, response options, and interpretation of each question; (2) 10 individuals with expertise in either evaluation or assessment pilot tested the survey; (3) primary investigator reviewed respondents’ answers and feedback; and (4) the primary investigator modified the survey questions accordingly. These questions included, but were not limited to, the instructor’s definition of evaluation, content related specifically to Stevahn et al.’s (2005a) evaluator competencies taxonomy, instructional pedagogy, and instructor demographic characteristics. Minor wording changes (predominately in verb tense) were made to Stevahn et al.’s (2005a) specific competencies within each of the six competency domains in order to facilitate respondent understanding. 4
For the items related to specific evaluator competencies, instructors were asked to indicate whether they covered the competency in their evaluation course. When instructors indicated they taught a particular competency, they then were asked to indicate the method or methodologies used to teach the competency. Regarding the later, instructors could indicate that they taught the competency through one or more of the following instructional strategies: lecture(s), in-class exercise(s), out-of-class exercise(s), course reading(s), paper(s), group project(s) other than field evaluations, field evaluation(s) in a group, field evaluation(s) individually, and/or journaling.
Procedures
Using the Qualtrics survey platform, and closely adhering to Dillman, Smyth, and Christian’s (2014) tailored design method, administration of the Internet survey occurred over a 4-week period. All instructors in the sampling frame received a presurvey notification announcing the survey, its purposes, and time frame for completion 1 week prior to the actual survey. The following week, an e-mail message providing access to the Internet survey was sent. Once per week thereafter, over a 2-week period, reminder e-mails were sent to individuals who had not yet completed the survey.
Analysis
Survey responses were downloaded from the Qualtrics system as a tab-delimited file. Closed-response items (i.e., quantitative) were imported into and analyzed using IBM SPSS Statistics Version 24. Descriptive statistics were predominately used to calculate frequencies. Due to differential item response, the denominators used to calculate percentages varied from item to item. Free-response items (i.e., qualitative) were imported into and thematically analyzed using MaxQDA Version 12.
Institutional Review
The study was reviewed and approved by the Western Michigan University Human Subjects Institutional Review Board. Prior to participating in the study, participants in the Internet survey were asked to read an electronic informed consent (provided in the survey access e-mail).
Results
Of the 580 unique evaluation courses identified through the document analysis and included in the study, slightly more than one fourth (25.86%, n = 150) were fully focused on evaluation and nearly three fourths (74.14%, n = 430) were partially focused on evaluation. Three-hundred and eleven (53.62%) were located in a school of public health and 269 (46.38%) were in a program of public health. Of the fully focused evaluation courses, 80 (53.33%) were located within a school of public health and 70 (46.67%) within a program of public health. Of the partially focused evaluation courses, 231 (53.72%) were located within a school of public health and 199 (45.58%) within a program of public health.
Of the 580 instructors invited to participate in the survey, 128 provided responses, yielding a response rate of 22%. No statistically significant differences between the characteristics (e.g., school, program, level of evaluation focus) of the responding instructors and the population sampling frame were identified through tests of equality of proportions.
On average, survey respondents had taught evaluation for 9 years (Min = 0; Max = 25; SD = 7.11). More than two thirds (69.97%, n = 87) taught their evaluation course once per year. Nearly all instructors reported that they have managed (90.70%, n = 86), designed (95.45%, n = 88), conducted (96.59%, n = 88), used (94.25%, n = 87), or reported (96.59%, n = 88) evaluations. Of the 88 respondents who indicated their highest degree, the highest degree attained by the majority of respondents included doctor of philosophy (PhD; 68.18%), DrPH (17.05%), or MPH (10.23%). Of the 82 respondents who indicated their major for their highest degree, only 3.66% majored in evaluation for their highest degree. More commonly, the major for the highest degree attained was in public health (50.00%), psychology (9.76%), or demography (7.32%).
Coverage of Evaluator Competencies
According to the survey respondents, all of Stevahn et al.’s (2005a, 2005b) evaluator competency domains are taught, albeit to varying degrees, in evaluation courses offered by CEPH-accredited schools of public health and MPH programs. As illustrated in Figure 1, professional practice and systematic inquiry received the greatest coverage, followed by situational analysis and interpersonal competence. Project management and reflective practice received only cursory coverage.Including both fully and partially focused courses, Stevahn et al.’s (2005a) professional practice domain (i.e., “…fundamental norms and values underlying evaluation practice,” p. 52), on average, received the most frequent coverage (75.02%; see Figure 1). This high average result stems from the majority of these courses covering all but one (“applying professional evaluation standards,” 22.63%) of the six competencies under this domain (see Figure 2).
Competency domain coverage. Percentages reflect averages of averages (e.g., if there were three competencies [A, B, and C] within a domain and instructors, on average, reported that 20% taught Competency A, 60% taught Competency B, and 100% taught Competency C, the final average is 60% for that domain). The lower and upper limits represent the low and high coverage of competencies within domains, not confidence intervals. IC = interpersonal competence; PM = project management; PP = professional practice; RP = reflective practice; SA = situational analysis; SI = systematic inquiry.
Professional practice domain coverage (n ranges from 103 to 108). Numbering of specific competencies corresponds to Stevahn, King, Ghere, and Minnema’s (2005a) numbering (table 1, pp. 49–51).
Considering both fully and partially focused courses, the systematic inquiry domain (i.e., “…technical aspects of evaluation practice”; Stevahn et al., 2005a, p. 52) received, on average, coverage in slightly more than two thirds (69.30%; see Figure 1) of the evaluation courses reported on by instructors. Although the majority of the domain competencies were taught in more than half of the courses, the “conducting metaevaluations” competency was infrequently covered (10.64%; see Figure 3) and received the least coverage across all of the 35 evaluator competencies.
Systematic inquiry domain coverage (n ranges from 92 to 104). Numbering of specific competencies corresponds to Stevahn, King, Ghere, and Minnema’s (2005a) numbering (table 1, pp. 49–51).
Stevahn et al.’s (2005a) situational analysis domain (i.e., “…analyzing and attending to the unique interests, issues, and contextual circumstances pertaining to any given evaluation,” p. 52), on average, for both fully and partially focused courses combined, received modest coverage (60.20%; see Figure 1) in evaluation courses. Although classified as “modest coverage,” as illustrated in Figure 4, most of the competencies under this domain (10 of 12) were addressed in over 50% of these courses. Exceptions included “attending to issues of organizational change” (34.41%) and “addressing conflicts” (46.74%). The lower overall average (in comparison with the professional practice and systematic inquiry domains) results from several of the competencies in this domain (5 of 12) being covered in approximately half of the courses.
Situational analysis domain coverage (n ranges from 90 to 94). Numbering of specific competencies corresponds to Stevahn, King, Ghere, and Minnema’s (2005a) numbering (table 1, pp. 49–51).
On average, including both fully and partially focused courses, the interpersonal competence domain (i.e., “…people skills used in conducting evaluation studies, such as communication, negotiation, conflict, collaboration, and cross-cultural skills”; Stevahn et al., 2005a, p. 52) was covered in slightly more than half (56.30%; see Figure 1) of the evaluation courses. Even so, some competencies within this domain were frequently covered through these courses such as “using written communication skills” (85.23%) and “using verbal/listening communication skills” (71.59%). These competencies were emphasized far more frequently than others in this domain, including “using negotiation skills” (28.41%) and “using conflict resolution skills” (27.59%; see Figure 5).
Interpersonal competence domain coverage (n ranges from 87 to 88). Numbering of specific competencies corresponds to Stevahn, King, Ghere, and Minnema’s (2005a) numbering (table 1, pp. 49–51).
On average, the project management domain (i.e., “…nuts and bolts of conducting an evaluation, such as budgeting, coordinating resources, and supervising procedures”; Stevahn et al., 2005a, p. 52) competencies are taught in less than one third (29.98%; see Figure 1) of evaluation courses for both fully and partially focused courses combined. Of the 12 competencies subsumed in the domain, none are taught in even half of the evaluation courses reported on (see Figure 6) and 3 (“training others involved in conducting the evaluation” [17.78%], “writing formal agreements” [16.85%], and “supervising others involved in the evaluation” [13.48%]) are covered in less than one in five of these courses.
Project management domain coverage (n ranges from 88 to 91). Numbering of specific competencies corresponds to Stevahn, King, Ghere, and Minnema’s (2005a) numbering (table 1, pp. 49–51).
Overall, considering both fully and partially focused courses, the competencies within the reflective practice domain (i.e., “…one’s awareness of evaluation expertise and needs for growth, including knowing oneself as an evaluator, assessing personal needs for enhanced practice, and engaging in professional development toward that goal”; Stevahn et al., 2005a, p. 52) are infrequently covered in the evaluation courses reported on by instructors (23.60%; see Figure 1). As shown in Figure 7, only two of the five competencies associated with reflective practice are taught in more than one fourth of these courses (i.e., “forming an awareness of self as an evaluator” [38.20%] and “reflecting on personal evaluation practice” [26.97%], respectively).
Reflective practice domain coverage (n = 89). Numbering of specific competencies corresponds to Stevahn, King, Ghere, and Minnema’s (2005a) numbering (table 1, pp. 49–51).
Instructional Strategies Used to Teach Evaluator Competencies
Students enrolled in evaluation courses at CEPH-accredited schools of public health and MPH programs learn about, and are expected to acquire, evaluation competencies via a combination of instructional strategies. These strategies most frequently included attending lectures (90.36%) and reading course materials (68.44%). Other instructional and learning strategies, as shown in Table 1, were used far less frequently. In descending order, the least frequently used strategies consisted of in-class exercises (35.26%), paper writing (30.46%), out-of-class exercises (24.25%), group projects other than field evaluations (20.23%), field evaluations in a group (19.30%), field evaluations individually (9.87%), and journaling (1.09%).
Instructional Strategies by Competency Domain.a
aPercentages reflect averages of averages.
Discussion
The public health literature has established the importance of designing, conducting, reporting, and using evaluations in public health practice and research (Association of Schools of Public Health, 2006; CEPH, 2016; IOM, 1988; Public Health Functions Steering Committee, 2008). This literature, as well as existing research on evaluation, provides few insights into the content, quality, and consistency of evaluation training in CEPH-accredited schools of public health and MPH programs. As such, this study explored the content of evaluation training from the perspective of faculty who teach these courses in CEPH-accredited schools of public health and MPH programs.
The findings from this study indicate that CEPH-accredited schools of public health and MPH programs do cover several of the evaluator competencies proposed by Stevahn et al. (2005a, 2005b). However, the extent to which coursework with a full or partial focus on evaluation addresses each of the evaluation domains and competencies varies greatly. Survey responses regarding 128 evaluation courses indicated that competencies within the professional practice and systematic inquiry domains were most often covered, followed by situational analysis and interpersonal competence. Fewer than half of the evaluation courses covered competencies under the project management and reflective practice domains. Although a single course in evaluation, particularly one that has a partial emphasis on evaluation, could (or should) cover all of the competencies proposed by Stevahn et al. (2005a, 2005b), the overall patterns in competencies covered across the evaluation coursework do shed light on opportunities for enhancing existing coursework to prepare MPH graduates for professional positions in which they are asked to lead or engage in evaluations.
It is likely that coursework that is part of an MPH degree but not classified as a full or partial evaluation course in this study covers competencies within several of the domains, most specifically systematic inquiry, project management, and interpersonal competence. For example, only 49% of evaluation courses addressed the systematic inquiry competency “analyzing data.” However, all MPH students complete a biostatistics course as part of their curriculum and gain exposure to data analysis through these courses prior to graduation. Project management and interpersonal competencies such as “responding to requests for proposals,” “presenting work in a timely manner,” and “pursuing professional development in relevant content areas” (which are not necessarily “unique” to the practice of evaluation) may be covered through a combination of required courses for an MPH degree or applied practice experience which is required of all MPH students.
What is perhaps most enlightening is the absence of some competencies that are uniquely evaluation-specific from courses that were classified as fully or partially focused on evaluation. For example, competencies that reside at the heart of the professional practice of evaluation such as “determining program evaluability,” “applying professional evaluation standards,” and “conducting metaevaluation” were covered infrequently. Perhaps the absence of these competencies, particularly “applying professional evaluation standards,” should not be surprising since only 19% (n = 17) of respondents indicated they were members of AEA.
Implications
Findings from this study offer evidence that there is room to increase the coverage of evaluation domains and competencies in CEPH-accredited schools of public health and MPH programs’ evaluation courses. Nevertheless, the results suggest there is a need for evaluation course instructors to reconsider or add to their course content in light of widely accepted evaluation competencies. Perhaps one step forward in this process may include evaluation course instructors conducting a gap analysis of their course content with Stevahn et al.’s (2005a, 2005b) Essential Competencies for Program Evaluators and then inclusion of the evaluation competency gaps in their courses. However, such a step seems to promote “silos” that may be resulting in some of the incongruence between what the professional evaluation community sees as important competencies and what gets included in public health evaluation coursework. In light of this, it seems that a prudent path forward could, and perhaps should, be to create a stronger connection between the professional public health community and the professional evaluation community. Findings from this study, and from Fierro and Christie (2011), indicate that individuals developing coursework and supporting materials (e.g., textbooks) for evaluation in public health may be “going it alone.” Improved collaboration between public health and evaluation could help to improve the alignment between existing coursework in public health academe with what is expected from professional evaluators.
Given the central importance of evaluation in public health practice, evaluation and public health experts should collaborate on identifying the essential evaluator competencies for public health professionals and subsequently identify means for integrating such competencies into existing training programs. For instance, representatives from AEA could collaborate with representatives from the American Public Health Association (APHA), ASPPH, and CEPH to determine which evaluation competencies are most relevant to CEPH-accredited schools of public health and MPH programs’ evaluation courses and incorporate evaluation-specific competencies into the CEPH-MPH foundational competencies. These representatives could then collaboratively work with CEPH-accredited schools of public health and MPH programs to develop curriculum for evaluation courses that have a full or partial focus on evaluation (i.e., competencies to cover, recommended readings, and teaching methods to instruct on competencies). Such collaborative efforts require both an understanding of the current state of evaluation as well as a strong understanding of public health practice and training. As such, the development and implementation of essential evaluation competencies for public health should involve experts from both public health and evaluation.
Limitations
Lack of coverage of some competencies, in particular, many of those associated with systematic inquiry, project management, and interpersonal competence, are likely taught in “nonevaluation” courses, not examined in this study. In addition, to identify evaluation courses, the two coders relied on university, school and program, and course websites, which might have had outdated information, provided partial information, and/or painted the perfect representation of a course. Furthermore, since the document analysis data collection phase took 10 months to complete, the data that were collected related to the course demographics could have changed over the time period. Also, the Internet survey was administered to only course instructors at institutions with websites in English and when contact information was available. Relatedly, generalizability is compromised given that the survey response rate was 22% and missing responses were not missing at random, preventing imputation and suggesting systematic bias in missingness.
Future Research
Given the limited number of research studies performed in this area, there remain wide gaps in knowledge about the instruction of evaluation within public health academe. Future research may repeat this study with the soon to be published AEA-approved evaluator competencies and the newly published, in 2016, evaluation-related competencies of the MPH foundational competencies, for example, “Assess population needs, assets and capacities that affect communities’ health,” “Select methods to evaluate public health programs,” and “Evaluate policies for their impact on public health and health equity” (CEPH, 2016, pp. 17–18). The MPH foundational competencies are new competencies since the initiation of this study and thus the researchers plan to conduct additional research related to this initial study to determine if more is being done to train public health professionals in evaluation. Future research may also investigate the most essential and valued evaluation competencies for public health professionals. In “Evaluator Competencies in University-Based Evaluation Training Programs,” Stevahn et al. (2005b) noted: …specialty-oriented programs would strive to offer courses more narrowly focused on only those competencies most needed in a given area or context, such as business, education, government, health, or social service. Similarly, courses may be offered to help students master the methods that specific types of practice predominately use—such as quantitative methods in large-scale survey studies or qualitative methods in small-scale case studies—or master particular approaches that certain types of practice predominately use such as objectives-oriented, utilization-focused, participatory, or transformative evaluation, to name a few. (p. 110)
In this study, the evaluation content of existing courses was examined; future studies should include exploration of nonevaluation content in evaluation courses and the quality of instruction and consistency to which evaluation should be implemented across all MPH concentrations. Studies that compare what is taught in CEPH-accredited schools of public health and MPH programs with what is sought among employers who hire MPH graduates of CEPH-accredited schools and programs have the potential to elucidate the alignment of the knowledge, skills, and abilities between recent graduates and the workplace (see also Dewey, Montrosse, Schröter, Sullins, & Mattox, 2008). To successfully determine the gaps, one needs to first explore the extent to which evaluation competencies are taught in courses not defined as evaluation, including the required applied practice experience in MPH curriculum.
Such workplace–academia comparison studies also hold the potential to help better understand which MPH concentration areas may need to emphasize evaluation coursework more than others. A substantial question that remains in this line of inquiry is who among public health professionals actually leads and actively contributes to the planning and implementation of evaluations in the workplace. Anecdotal information suggests that epidemiologists often fulfill this role, yet previous research (Fierro & Christie, 2011) suggests that epidemiologists trained at an MPH level are rarely required to complete evaluation coursework prior to graduation. Increasing understanding of which public health professionals are most in need of evaluation training to perform their jobs can also aid in the future conversations envisioned between AEA, APHA, ASPPH, and CEPH.
Footnotes
Acknowledgments
The authors would like to acknowledge and thank Dr. Robin L. Miller for initial conceptualization contributions and Dr. Sabrina E. Holley for assistance with data collection and coding.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by Western Michigan University.