Instruction of Evaluation Competencies in Council on Education for Public Health (CEPH)-Accredited Master of Public Health (MPH) Degree Programs

Phase I: Document Analysis

Phase II: Internet Survey

Institutional Review

The study was reviewed and approved by the Western Michigan University Human Subjects Institutional Review Board. Prior to participating in the study, participants in the Internet survey were asked to read an electronic informed consent (provided in the survey access e-mail).

Coverage of Evaluator Competencies

According to the survey respondents, all of Stevahn et al.’s (2005a, 2005b) evaluator competency domains are taught, albeit to varying degrees, in evaluation courses offered by CEPH-accredited schools of public health and MPH programs. As illustrated in Figure 1, professional practice and systematic inquiry received the greatest coverage, followed by situational analysis and interpersonal competence. Project management and reflective practice received only cursory coverage.Including both fully and partially focused courses, Stevahn et al.’s (2005a) professional practice domain (i.e., “…fundamental norms and values underlying evaluation practice,” p. 52), on average, received the most frequent coverage (75.02%; see Figure 1). This high average result stems from the majority of these courses covering all but one (“applying professional evaluation standards,” 22.63%) of the six competencies under this domain (see Figure 2).

OPEN IN VIEWER

Figure 1.

Competency domain coverage. Percentages reflect averages of averages (e.g., if there were three competencies [A, B, and C] within a domain and instructors, on average, reported that 20% taught Competency A, 60% taught Competency B, and 100% taught Competency C, the final average is 60% for that domain). The lower and upper limits represent the low and high coverage of competencies within domains, not confidence intervals. IC = interpersonal competence; PM = project management; PP = professional practice; RP = reflective practice; SA = situational analysis; SI = systematic inquiry.

OPEN IN VIEWER

Figure 2.

Professional practice domain coverage (n ranges from 103 to 108). Numbering of specific competencies corresponds to Stevahn, King, Ghere, and Minnema’s (2005a) numbering (table 1, pp. 49–51).

Considering both fully and partially focused courses, the systematic inquiry domain (i.e., “…technical aspects of evaluation practice”; Stevahn et al., 2005a, p. 52) received, on average, coverage in slightly more than two thirds (69.30%; see Figure 1) of the evaluation courses reported on by instructors. Although the majority of the domain competencies were taught in more than half of the courses, the “conducting metaevaluations” competency was infrequently covered (10.64%; see Figure 3) and received the least coverage across all of the 35 evaluator competencies.

OPEN IN VIEWER

Figure 3.

Systematic inquiry domain coverage (n ranges from 92 to 104). Numbering of specific competencies corresponds to Stevahn, King, Ghere, and Minnema’s (2005a) numbering (table 1, pp. 49–51).

Stevahn et al.’s (2005a) situational analysis domain (i.e., “…analyzing and attending to the unique interests, issues, and contextual circumstances pertaining to any given evaluation,” p. 52), on average, for both fully and partially focused courses combined, received modest coverage (60.20%; see Figure 1) in evaluation courses. Although classified as “modest coverage,” as illustrated in Figure 4, most of the competencies under this domain (10 of 12) were addressed in over 50% of these courses. Exceptions included “attending to issues of organizational change” (34.41%) and “addressing conflicts” (46.74%). The lower overall average (in comparison with the professional practice and systematic inquiry domains) results from several of the competencies in this domain (5 of 12) being covered in approximately half of the courses.

OPEN IN VIEWER

Figure 4.

Situational analysis domain coverage (n ranges from 90 to 94). Numbering of specific competencies corresponds to Stevahn, King, Ghere, and Minnema’s (2005a) numbering (table 1, pp. 49–51).

On average, including both fully and partially focused courses, the interpersonal competence domain (i.e., “…people skills used in conducting evaluation studies, such as communication, negotiation, conflict, collaboration, and cross-cultural skills”; Stevahn et al., 2005a, p. 52) was covered in slightly more than half (56.30%; see Figure 1) of the evaluation courses. Even so, some competencies within this domain were frequently covered through these courses such as “using written communication skills” (85.23%) and “using verbal/listening communication skills” (71.59%). These competencies were emphasized far more frequently than others in this domain, including “using negotiation skills” (28.41%) and “using conflict resolution skills” (27.59%; see Figure 5).

OPEN IN VIEWER

Figure 5.

Interpersonal competence domain coverage (n ranges from 87 to 88). Numbering of specific competencies corresponds to Stevahn, King, Ghere, and Minnema’s (2005a) numbering (table 1, pp. 49–51).

On average, the project management domain (i.e., “…nuts and bolts of conducting an evaluation, such as budgeting, coordinating resources, and supervising procedures”; Stevahn et al., 2005a, p. 52) competencies are taught in less than one third (29.98%; see Figure 1) of evaluation courses for both fully and partially focused courses combined. Of the 12 competencies subsumed in the domain, none are taught in even half of the evaluation courses reported on (see Figure 6) and 3 (“training others involved in conducting the evaluation” [17.78%], “writing formal agreements” [16.85%], and “supervising others involved in the evaluation” [13.48%]) are covered in less than one in five of these courses.

OPEN IN VIEWER

Figure 6.

Project management domain coverage (n ranges from 88 to 91). Numbering of specific competencies corresponds to Stevahn, King, Ghere, and Minnema’s (2005a) numbering (table 1, pp. 49–51).

Overall, considering both fully and partially focused courses, the competencies within the reflective practice domain (i.e., “…one’s awareness of evaluation expertise and needs for growth, including knowing oneself as an evaluator, assessing personal needs for enhanced practice, and engaging in professional development toward that goal”; Stevahn et al., 2005a, p. 52) are infrequently covered in the evaluation courses reported on by instructors (23.60%; see Figure 1). As shown in Figure 7, only two of the five competencies associated with reflective practice are taught in more than one fourth of these courses (i.e., “forming an awareness of self as an evaluator” [38.20%] and “reflecting on personal evaluation practice” [26.97%], respectively).

OPEN IN VIEWER

Figure 7.

Reflective practice domain coverage (n = 89). Numbering of specific competencies corresponds to Stevahn, King, Ghere, and Minnema’s (2005a) numbering (table 1, pp. 49–51).

Instructional Strategies Used to Teach Evaluator Competencies

Students enrolled in evaluation courses at CEPH-accredited schools of public health and MPH programs learn about, and are expected to acquire, evaluation competencies via a combination of instructional strategies. These strategies most frequently included attending lectures (90.36%) and reading course materials (68.44%). Other instructional and learning strategies, as shown in Table 1, were used far less frequently. In descending order, the least frequently used strategies consisted of in-class exercises (35.26%), paper writing (30.46%), out-of-class exercises (24.25%), group projects other than field evaluations (20.23%), field evaluations in a group (19.30%), field evaluations individually (9.87%), and journaling (1.09%).

OPEN IN VIEWER

Table 1.

Instructional Strategies by Competency Domain.^a

Domain		Attend Lectures (%)	Read Course Materials (%)	In-Class Exercises (%)	Paper Writing (%)	Out-of-Class Exercises (%)	Group Projects Other Than Field Evaluations (%)	Group Field Evaluations (%)	Individual Field Evaluations (%)	Journaling (%)
5.0	Reflective practice	97.20	48.37	29.60	16.90	15.39	1.74	11.93	0.00	1.74
3.0	Situational analysis	94.88	67.45	31.49	35.69	23.34	24.45	15.67	5.05	0.17
4.0	Project management	94.18	46.96	19.51	15.56	9.49	17.89	27.87	9.76	0.24
2.0	Systematic inquiry	92.13	70.62	38.37	32.09	29.34	19.80	23.46	14.03	1.25
1.0	Professional practice	88.64	75.59	37.85	25.68	24.40	16.45	17.36	11.51	2.17
6.0	Interpersonal competence	77.53	49.68	48.12	29.79	18.77	36.33	19.38	5.45	0.24

^aPercentages reflect averages of averages.

Implications

Findings from this study offer evidence that there is room to increase the coverage of evaluation domains and competencies in CEPH-accredited schools of public health and MPH programs’ evaluation courses. Nevertheless, the results suggest there is a need for evaluation course instructors to reconsider or add to their course content in light of widely accepted evaluation competencies. Perhaps one step forward in this process may include evaluation course instructors conducting a gap analysis of their course content with Stevahn et al.’s (2005a, 2005b) Essential Competencies for Program Evaluators and then inclusion of the evaluation competency gaps in their courses. However, such a step seems to promote “silos” that may be resulting in some of the incongruence between what the professional evaluation community sees as important competencies and what gets included in public health evaluation coursework. In light of this, it seems that a prudent path forward could, and perhaps should, be to create a stronger connection between the professional public health community and the professional evaluation community. Findings from this study, and from Fierro and Christie (2011), indicate that individuals developing coursework and supporting materials (e.g., textbooks) for evaluation in public health may be “going it alone.” Improved collaboration between public health and evaluation could help to improve the alignment between existing coursework in public health academe with what is expected from professional evaluators.

Given the central importance of evaluation in public health practice, evaluation and public health experts should collaborate on identifying the essential evaluator competencies for public health professionals and subsequently identify means for integrating such competencies into existing training programs. For instance, representatives from AEA could collaborate with representatives from the American Public Health Association (APHA), ASPPH, and CEPH to determine which evaluation competencies are most relevant to CEPH-accredited schools of public health and MPH programs’ evaluation courses and incorporate evaluation-specific competencies into the CEPH-MPH foundational competencies. These representatives could then collaboratively work with CEPH-accredited schools of public health and MPH programs to develop curriculum for evaluation courses that have a full or partial focus on evaluation (i.e., competencies to cover, recommended readings, and teaching methods to instruct on competencies). Such collaborative efforts require both an understanding of the current state of evaluation as well as a strong understanding of public health practice and training. As such, the development and implementation of essential evaluation competencies for public health should involve experts from both public health and evaluation.

Limitations

Lack of coverage of some competencies, in particular, many of those associated with systematic inquiry, project management, and interpersonal competence, are likely taught in “nonevaluation” courses, not examined in this study. In addition, to identify evaluation courses, the two coders relied on university, school and program, and course websites, which might have had outdated information, provided partial information, and/or painted the perfect representation of a course. Furthermore, since the document analysis data collection phase took 10 months to complete, the data that were collected related to the course demographics could have changed over the time period. Also, the Internet survey was administered to only course instructors at institutions with websites in English and when contact information was available. Relatedly, generalizability is compromised given that the survey response rate was 22% and missing responses were not missing at random, preventing imputation and suggesting systematic bias in missingness.

Future Research

Given the limited number of research studies performed in this area, there remain wide gaps in knowledge about the instruction of evaluation within public health academe. Future research may repeat this study with the soon to be published AEA-approved evaluator competencies and the newly published, in 2016, evaluation-related competencies of the MPH foundational competencies, for example, “Assess population needs, assets and capacities that affect communities’ health,” “Select methods to evaluate public health programs,” and “Evaluate policies for their impact on public health and health equity” (CEPH, 2016, pp. 17–18). The MPH foundational competencies are new competencies since the initiation of this study and thus the researchers plan to conduct additional research related to this initial study to determine if more is being done to train public health professionals in evaluation. Future research may also investigate the most essential and valued evaluation competencies for public health professionals. In “Evaluator Competencies in University-Based Evaluation Training Programs,” Stevahn et al. (2005b) noted:

…specialty-oriented programs would strive to offer courses more narrowly focused on only those competencies most needed in a given area or context, such as business, education, government, health, or social service. Similarly, courses may be offered to help students master the methods that specific types of practice predominately use—such as quantitative methods in large-scale survey studies or qualitative methods in small-scale case studies—or master particular approaches that certain types of practice predominately use such as objectives-oriented, utilization-focused, participatory, or transformative evaluation, to name a few. (p. 110)

In this study, the evaluation content of existing courses was examined; future studies should include exploration of nonevaluation content in evaluation courses and the quality of instruction and consistency to which evaluation should be implemented across all MPH concentrations. Studies that compare what is taught in CEPH-accredited schools of public health and MPH programs with what is sought among employers who hire MPH graduates of CEPH-accredited schools and programs have the potential to elucidate the alignment of the knowledge, skills, and abilities between recent graduates and the workplace (see also Dewey, Montrosse, Schröter, Sullins, & Mattox, 2008). To successfully determine the gaps, one needs to first explore the extent to which evaluation competencies are taught in courses not defined as evaluation, including the required applied practice experience in MPH curriculum.

Such workplace–academia comparison studies also hold the potential to help better understand which MPH concentration areas may need to emphasize evaluation coursework more than others. A substantial question that remains in this line of inquiry is who among public health professionals actually leads and actively contributes to the planning and implementation of evaluations in the workplace. Anecdotal information suggests that epidemiologists often fulfill this role, yet previous research (Fierro & Christie, 2011) suggests that epidemiologists trained at an MPH level are rarely required to complete evaluation coursework prior to graduation. Increasing understanding of which public health professionals are most in need of evaluation training to perform their jobs can also aid in the future conversations envisioned between AEA, APHA, ASPPH, and CEPH.