An Assessment of Evaluation Instruction Related to the Health Education Specialist Practice Analysis Competencies in Health Education–Related Professional Preparation Programs

Abstract

Background. Conducting evaluation and research is one of the seven areas of responsibility for health education specialists. However, there is little evidence to suggest that professional preparation programs use the competencies as identified by the 2015 Health Education Specialist Practice Analysis (HESPA) as a basis for instruction. Aims. To determine the degree to which departments with undergraduate health education–related professional preparation programs provide instruction in the HESPA 2015 Model evaluation competencies. Method. This was a descriptive, cross-sectional, nonexperimental study design. Data were collected using an online survey from programs that prepare students to sit for the CHES exam. Results. Faculty provide instruction about all evaluation subcompetencies regardless of entry- or advanced-level designation. Developing an evaluation plan, monitoring implementation, and selecting, adapting, and creating instruments to collect data represent the competencies taught most frequently. On average, two to three class periods are spent on each competency. Discussion. Advanced-level competencies, more commonly associated with graduate-level training and degrees, are commonly taught at the undergraduate level. This may be due to a lack of awareness of changes to the designation of competencies and subcompetencies as entry- or advanced-level in the HESPA 2015 Model framework, a lack of awareness or appreciation for the framework in general, competing values in the selection of curriculum content, or lack of professional consensus related to terms and definitions. Conclusion. Faculty should consider emphasizing entry-level competencies and subcompetencies consistent with process and formative evaluation. The health education profession should also standardize evaluation and research terms and definitions.

Keywords

competencies curriculum evaluation health education health education specialists professional preparation

Evaluation in health education settings has been defined as a process of determining the value or worth of a program, or any of its components, based on predetermined standards of success identified by stakeholders (McKenzie, Neiger, & Thackeray, 2017). Evidence-based health education and promotion practice calls for a systematic and scientific approach to evaluation as the basis for program decision making (Joint Committee on Health Education and Promotion Terminology, 2012). This requires a standard set of evaluation competencies informed and validated by the health education profession, then translated to practice by health education specialists.

The health education profession has a long-documented history of validating key competencies, including evaluation, that serve as the basis for its professional credentials (i.e., Certified Health Education Specialists [CHES] or Master Certified Health Education Specialist [MCHES]) and the theoretical bases for both professional preparation and professional development (National Commission for Health Education Credentialing [NCHEC], 2017a). Most recently, the Health Education Specialist Practice Analysis (HESPA) was conducted during 2013-2014 and published in 2015 (NCHEC & Society for Public Health Education (SOPHE), 2015). The findings from the HESPA study validated seven areas of responsibility, 36 competencies (“a broadly defined skill or ability”) and 258 subcompetencies (“a cluster of simpler but essential related skills or abilities”; NCHEC & SOPHE, 2015, pp. 89-90). Of the subcompetencies, 55% were classified as entry-level practice and the remainder were classified as either advanced 1– or advanced 2–levels of practice (NCHEC, 2017b). Entry level is classified as, “the practice level of a health education specialist with a minimum of a baccalaureate degree with professional preparation in the field of health education” (NCHEC & SOPHE, 2015, p. 89). The advanced 1–level is “the practice level of a health education specialist with a minimum of a baccalaureate degree with professional preparation in the field of health education plus various combinations of degree (baccalaureate or master’s) and years of experience,” while the advanced 2–level is “the practice level of a health education specialist with a minimum of a doctoral degree in the field of health education, irrespective of years of experience” (NCHEC & SOPHE, 2015, p. 89).

Conducting evaluation and research related to health education was one of the seven areas of responsibility (Area IV—conduct evaluation and research related to health education/promotion) validated in the HESPA, with six related competencies (i.e., develop evaluation plan for health education/promotion; develop a research plan for health education/promotion; select, adapt and/or create instruments to collect data; collect and manage data; analyze data; interpret results; apply findings) and 57 related subcompetencies. Area of Responsibility III, Competency 3.4 “monitor implementation of health education/promotion,” has an additional eight subcompetencies that closely relate to process evaluation (NCHEC & SOPHE, 2015). Of these collective 65 evaluation-related subcompetencies, only 17, or 26%, are identified as entry level.

How professional preparation programs use these competencies to guide curricular decisions varies. Programs seeking accreditation from the Council on Education for Public Health (CEPH) that also prepare students for the CHES exam, must ensure that the curriculum includes instruction related to CHES competencies (CEPH, 2016). However, outside this, there is no evidence that health education–related professional preparation programs are required to use, or use, the HESPA 2015 Model competencies as a basis for curriculum development or their instruction, or that graduates are necessarily prepared for questions related to evaluation on the NCHEC’s CHES examination. Performance on the CHES examination from 2012 to 2016, indicates that the average score for “evaluate programs” was the lowest of all the seven areas of responsibility (66.1%). By comparison, the average percent score across all years and areas of responsibility, excluding evaluation, was 71% (L. Lysoby, personal communication, June 30, 2017). While it is not clear why scores are lowest for evaluation, these data may indicate that professional preparation programs are not adequately preparing students to answer questions related to evaluation on the CHES examination, and by association, perform entry-level evaluation in professional practice.

Previous research has examined the content of professional preparation courses for program planning (Linnan et al., 2005), patient education (Heitzer, McKenzie, Amschler, & Bock, 2009), school health methods (Fisher, Price, Telljohann, & Dake, 2015), and school health content and tools (Davidson, Telljohann, Dake, & Price, 2010). However, to our knowledge, there has never been an assessment of program evaluation courses. Therefore, the purpose of this study was to determine the degree to which departments within colleges and universities in the United States that offer undergraduate programs in health education, health promotion, or community health, provide training in HESPA 2015 Model evaluation competencies and subcompetencies.

Method

Sample

This was a descriptive, cross-sectional, nonexperimental study design. There is currently no published directory of college or university health education and related programs. To select schools, we obtained from the NCHEC, a list of 205 colleges and universities that prepared students to sit for the CHES exam and had at least one student who took the exam in 2015. We delimited the sample to departments that self-identified as health education, health promotion, community health or similarly related areas and that offered related undergraduate degrees. In order for a student to sit for the CHES exam, the individual must have a degree in one of those disciplines, or an official transcript that reflects a minimum of 25 semester or 37 quarter hours of coursework in the seven areas of responsibility and competency for health education specialists.

A research assistant contacted the department chair or administrative assistant at each school via telephone and e-mail to confirm contact information for the person who taught the program evaluation course, or the course in which evaluation concepts were taught. At least 8 to 10 attempts were made to reach each school.

Of the 205 departments identified, the following were excluded: 18 schools did not respond to the inquiry for information, 8 offered graduate programs only, 5 reported the related degree program was not currently offered or was being phased-out, 9 reported not offering a course(s) that covered topics related to program evaluation, 1 was Spanish only, 1 had multiple campuses, 1 was a psychology department, and for 2 we were unable to confirm faculty names.

The Brigham Young University institutional review board formally reviewed and approved the study. Completion of the survey implied the respondent’s consent to participate. The Dillman total design method was used to administer an online survey (Millar & Dillman, 2011). A US$20 gift card to Amazon.com was offered as an incentive for completing the survey.

Measures

An 83-item survey was used to assess the extent to which each of the HESPA 2015 Model competencies and subcompetencies associated with Area of Responsibility IV and Competency 3.4 of Area of Responsibility III (process evaluation) were addressed in the course in which evaluation concepts were taught, as well as how much time was spent teaching those competencies. Questions also asked about the academic department demographics. The survey instrument was based on a similar assessment of school health education methods courses (Fisher et al., 2015). Face validity was established by inviting eight current or former health education or health promotion professors to participate in cognitive pretesting of the survey; five people responded. Modifications to the survey were made based on their feedback.

To assess the extent to which subcompetencies and competencies were taught, respondents were asked if in either or both of the last two times they taught the undergraduate evaluation course, or the course in which evaluation principles were taught, they covered topics identified as evaluation subcompetencies (yes or no). Responses were coded as 1 if the subcompetency was taught and 0 if the subcompetency was not taught. The overall score for each competency was the proportion of subcompetencies taught ranging from 0.0 if no subcompetencies were covered to 1.0 if all subcompetencies were covered.

If respondents answered “yes” to teaching at least one subcompetency within a competency, they were asked about the collective amount of time spent teaching that competency. Response options ranged from 1 to 5 (1 = part of 1 class period; 2 = 1 class period; 3 = 2 class periods; 4 = 3 class periods; 5 = 4 or more class periods). One class period was defined as approximately 1 hour. The mean score was then calculated.

To assess specific skills taught related to data collection, analysis, and reporting, additional survey questions asked about statistical software (e.g., SAS), survey administration software (e.g., Qualtrics), and data visualization. Demographic questions included program enrollment, course enrollment, instruction format, assessment measures, credit hours, whether the course was required or elective, if it was a quarter or semester length course, and the type of course materials used.

Statistical Analysis

Descriptive statistics included frequencies and means. To assess differences in the proportion of the subcompetencies taught within each competency we used analysis of variance (ANOVA). ANOVA was also used to test for differences between amount of time spent teaching each competency, with Tukey’s honest significant difference for multiple comparisons. Though the data were not normally distributed, results from parametric and nonparametric statistical tests were qualitatively the same so we chose to use the parametric statistics which are fairly robust for departures from normality (Glass & Hopkins, 1996).

Results

Ninety-eight respondents started the survey; 11 of the surveys were incomplete for a final response rate of 54.3% (N = 87). Demographics of the sample programs are presented in Table 1.

Table 1.

Demographics of Programs Represented.

Variable	% (n)
Course required or elective
Course is required of all students in department	36.8 (32)
Course is required of students in specific concentrations/emphases	60.9 (53)
Course is not required of any student (i.e., it is an elective)	2.3 (2)
Course type
Stand-alone course	26.4 (23)
Same course as program planning	48.3 (42)
Principles are taught in an introductory health education, health promotion, community health, or related course	2.3 (2)
Principles are taught in two or more courses	14.9 (13)
Other	8 (7)
Teaching format
In-person/face-to-face	79.3 (69)
Distance learning/online	6.9 (6)
Hybrid/blending of in-person or face-to-face and distance/online learning	9.2 (8)
Varies by semester or quarter	1.1 (1)
Other	3.4 (3)
Average enrollment
≤10 students	9.2 (8)
11-19 students	20.7 (18)
20-29 students	40.2 (35)
30-39 students	13.8 (12)
≥40 students	16.1 (14)
Teach on quarter or semester system
Quarter	4.6 (4)
Semester	95.4 (83)
Credit hours
2	1.1 (1)
3	86.2 (75)
4	9.2 (8)
≥5	3.4 (3)
Last time course taught
Within the last year (since February 2016)	86.2 (75)
1-2 years	10.3 (9)
5 or more years	3.4 (3)
Number of undergraduate students enrolled in program^a
<50	14.9 (13)
50-99	17.2 (15)
100-199	25.3 (22)
200-299	16.1 (14)
300-399	3.4 (3)
400-499	6.9 (6)
≥500	14.9 (13)

Total n may not equal 100% due to missing data.

All Evaluation-Related Competencies

In general, evaluation courses are required, are assigned three credits, and average 20 to 29 students. Respondents reported teaching about all the evaluation and research subcompetencies at some level regardless of whether the subcompetencies are defined by HESPA as entry level or advanced (see Table 2).

Table 2.

Percent of Programs Teaching Evaluation-Related Subcompetencies.

Entry or advanced level	Subcompetency number	Subcompetency name	% (n)
Entry	3.4.2	Assess progress in achieving objectives	89.7 (78)
Entry	3.4.4	Modify plan when needed	86.2 (75)
Entry	4.7.1	Communicate findings to priority populations, partners, and stakeholders	86.2 (75)
Entry	3.4.1	Monitor progress in accordance with timeline	82.8 (72)
Entry	4.3.4	Identify useable items from existing instruments	81.6 (71)
Entry	3.4.8	Monitor adherence to ethical principles in the implementation of health education/promotion	80.5 (70)
Entry	3.4.3	Ensure plan is implemented consistently	79.3 (69)
Entry	4.4.5	Comply with laws and regulations when collecting, storing, and protecting participant data	79.3 (69)
Entry	3.4.5	Monitor use of resources	77.0 (67)
Entry	4.4.4	Use available technology to collect, monitor, and manage data	74.7 (65)
Entry	4.7.4	Incorporate findings into program improvement and refinement	74.6 (65)
Entry	4.3.5	Adapt/modify existing items	72.4 (63)
Entry	4.7.2	Solicit feedback from priority populations, partners, and stakeholders	71.3 (62)
Entry	3.4.7	Ensure compliance with legal standards	70.1 (61)
Entry	4.4.3	Monitor and manage data collection	69.0 (60)
Entry	3.4.6^a	Evaluate sustainability of implementation	58.1 (50)
Entry	4.7.3	Evaluate feasibility of implementing recommendations	49.4 (43)
Advanced 2	4.3.1	Identify existing data collection instruments	95.4 (83)
Advanced 2	4.2.3	Conduct search for related literature	83.9 (73)
Advanced 2	4.3.10	Ensure fairness of data collection instruments (for example, reduce bias, use language appropriate to priority population)	83.9 (73)
Advanced 2	4.6.8^a	Develop recommendations based on findings	82.6 (71)
Advanced 2	4.3.6	Create new items to be used in data collection	81.6 (71)
Advanced 2	4.6.7^a	Draw conclusions based on findings	80.2 (69)
Advanced 2	4.2.4	Analyze and synthesize information found in the literature	79.3 (69)
Advanced 2	4.4.2	Collect data based on the evaluation or research plan	79.3 (69)
Advanced 2	4.7.5	Disseminate findings using a variety of methods	79.3 (69)
Advanced 2	4.3.3	Create new data collection instruments	78.2 (68)
Advanced 2	4.3.2	Adapt/modify existing data collection instruments	75.9 (66)
Advanced 2	4.2.1	Create statement of purpose	74.7 (65)
Advanced 2	4.2.14	Apply ethical principles to the research process	72.4 (63)
Advanced 2	4.6.5^a	Identify limitations of findings	72.1 (62)
Advanced 2	4.1.9	Develop data analysis plan for evaluation	71.3 (62)
Advanced 2	4.6.4^a	Propose possible explanations of findings	7.3. (53)
Advanced 2	4.2.9	Identify research participants	69.0 (60)
Advanced 2	4.5.3	Analyze data using descriptive statistical methods	69.0 (60)
Advanced 2	4.6.2	Explain how the results address the questions and/or hypotheses	69.0 (60)
Advanced 2	4.2.6	Assess the merits and limitations of qualitative and quantitative data collection	67.8 (59)
Advanced 2	4.2.8	Determine suitability of existing data collection instruments	66.7 (58)
Advanced 2	4.3.9	Ensure that data collection instruments generate reliable data	65.5 (57)
Advanced 2	4.5.1	Prepare data for analysis	64.4 (56)
Advanced 2	4.2.7	Select research design to address the research questions	63.2 (55)
Advanced 2	4.6.1	Synthesize the analyzed data	63.2 (55)
Advanced 2	4.2.11	Develop data collection procedures for research	62.1 (54)
Advanced 2	4.3.7	Pilot test data collection instrument	62.1 (54)
Advanced 2	4.2.5	Develop research questions and/or hypotheses	60.9 (53)
Advanced 2	4.2.10^a	Develop sampling plan to select participants	59.3 (51)
Advanced 2	4.4.1	Train data collectors involved in evaluation and/or research	58.6 (51)
Advanced 2	4.5.5	Use technology to analyze data	54.0 (47)
Advanced 2	4.3.8	Establish validity of data collection instruments	52.9 (46)
Advanced 2	4.2.2	Assess feasibility of conducting research	50.6 (44)
Advanced 2	4.6.3^a	Compare findings to results from other studies or evaluations	50.0 (43)
Advanced 2	4.5.4	Analyze data using inferential statistical methods	48.3 (42)
Advanced 2	4.2.12	Develop data analysis plan for research	47.1 (41)
Advanced 2	4.6.6^a	Address delimitations as they relate to findings	41.9 (36)
Advanced 2	4.2.13	Develop a plan for non-respondent follow-up	28.7 (25)
Advanced 1	4.1.2	Develop questions to be answered by the evaluation	94.3 (82)
Advanced 1	4.1.8	Develop data collection procedures for evaluation	93.1 (81)
Advanced 1	4.1.5	Assess needed and available resources to conduct evaluation	88.5 (77)
Advanced 1	4.1.10	Apply ethical principles to the evaluation process	87.4 (76)
Advanced 1	4.1.3	Create a logic model to guide the evaluation process	75.9 (66)
Advanced 1	4.1.7	Select a model for evaluation	75.9 (66)
Advanced 1	4.5.2	Analyze data using qualitative methods	64.4 (56)
Advanced 1	4.1.4	Adapt/modify a logic model to guide the evaluation process	60.9 (53)
Advanced 1	4.1.1	Determine the purpose and goals of evaluation	100 (87)
Advanced 1	4.1.6	Determine the types of data (for example, qualitative, quantitative) to be collected	100 (87)

n = 86.

As noted, a mean score for the proportion of a competency that was taught was calculated based on how many subcompetencies were taught. When comparing between competencies, overall, there was a statistically significant difference in the proportion of content taught within the competency (F = 6.20; p < .001). Faculty members taught the most content related to three of the eight evaluation-related competencies: developing an evaluation plan (Competency 4.1; M = .85), monitoring implementation (Competency 3.4; M = .78), and selecting, adapting, and creating instruments to collect data (Competency 4.3; M = .75). They taught less about Competency 4.2: developing a research plan (M = .63), Competency 4.4: collect and manage data (M = .72), Competency 4.5: analyze data (M = .60), Competency 4.6: interpret results (M = .67), and Competency 4.7: apply findings (M = .72). There was a statistically significant difference in the proportion of content taught within the competency (F = 6.20; p < .001).

There was a statistically significant difference for time spent teaching for all eight of the competencies (F = 8.56; p < .001). Faculty spend, on average, between two and three class periods on each competency (see Table 3). More class periods are spent on Competencies 3.4, 4.1, and 4.3.

Table 3.

Mean Number of Class Periods Spent Teaching Research and Evaluation-Related Competencies.

Competency	n	Mean^a	95% CI
4.1: Develop evaluation plan for health education/promotion	87	3.57	[3.31, 3.84]
4.3: Select, adapt and/or create instruments to collect data	86	3.52	[3.28, 3.76]
3.4: Monitor implementation of health education/promotion	85	3.39	[3.12, 3.66]
4.5: Analyze data	69	3.04	[2.74, 3.34]
4.2: Develop a research plan for health education/promotion	81	2.99	[2.70, 3.27]
4.6: Interpret results	73	2.78	[2.53, 3.03]
4.4: Collect and manage data	82	2.70	[2.43, 2.96]
4.7: Apply findings	73	2.58	[2.34, 2.83]

Note. CI = confidence interval.

Response options were from 1 to 5 (1 = part of 1 class period; 2 = 1 class period; 3 = 2 class periods; 4 = 3 class periods; 5 = 4 or more class periods).

Entry-Level Subcompetencies

A competency can be composed of all entry-level subcompetencies, all advanced-level subcompetencies, or a combination of both. There were no statistically significant differences in the proportion of entry-level subcompetencies taught within the associated competency (M = .7820 to .7040; p = .407). That is to say, faculty were not teaching more entry-level subcompetencies content in one competency than in the other competencies. Comparing between competencies, faculty members reported teaching the most subcompetencies related to Competency 3.4 and the least subcompetencies from Competency 4.7 (see Table 4).

Table 4.

Mean Proportion of Entry-Level Competencies Taught in Undergraduate Evaluation-Related Courses.

Competency	N	Mean	95% CI
3.4: Monitor implementation of health education/promotion	86	.78	[.72, .84]
Includes 8 subcompetencies: 3.4.1 to 3.4.8
4.3: Select, adapt, and/or create instruments to collect data	87	.77	[.70, .84]
Includes 2 subcompetencies: 4.3.4 and 4.3.5
4.4: Collect and manage data	87	.74	[.67, .82]
Includes 3 subcompetencies: 4.4.3, 4.4.4, 4.4.5
4.7: Apply findings	87	.70	[.63, .78]
Includes 4 subcompetencies: 4.7.1 to 4.7.4

Note. CI = confidence interval.

Advanced-Level Subcompetencies

There was a statistically significant difference in the proportion of subcompetencies content taught within the competency (p < .001). Faculty reported spending more time teaching subcompetencies related to Competencies 4.1, 4.3, and 4.7 (see Table 5) than any of the other competencies. It is noted that for the advanced level, Competency 4.7 has only one associated subcompetency.

Table 5.

Mean Proportion of Advanced-Level Competencies Taught in Undergraduate Evaluation-Related Courses.

Competency	N	Mean	95% CI
4.1: Develop evaluation plan for health education/promotion	87	.84	[.81, .88]
Includes 10 subcompetencies: 4.1.1 to 4.1.10
4.3: Select, adapt, and/or create instruments to collect data	87	.74	[.57, .70]
Includes 8 subcompetencies: 4.3.1; 4.3.2; 4.3.3; 4.3.6; 4.3.7; 4.3.8; 4.3.9; 4.3.10
4.4: Collect and manage data	87	.69	[.69, .80]
Includes 2 subcompetencies: 4.4.1 and 4.4.2
4.6: Interpret results	86	.67	[.61, .77]
Includes 8 subcompetencies: 4.6.1 to 4.6.8
4.2: Develop a research plan for health education/promotion	86	.63	[.52, .68]
Includes 14 subcompetencies: 4.2.1 to 4.2.14
4.5: Analyze data	87	.60	[.59, .74]
Includes 5 subcompetencies: 4.5.1 to 4.5.5
4.7: Apply findings	87	.79	[.71, .88]
Includes 1 subcompetency: 4.7.5

Note. CI = confidence interval.

Three quarters of respondents (74.7%) indicated that they taught content related to using technology to collect, monitor, and manage data (Subcompetency 4.4.4). For statistical packages, the most commonly used was SPSS (n = 24; 27.6%). Additionally 14.9% (n = 13) reported using other statistical packages, including Excel (n = 9), ArcGis, G*Power, Google Sheets, and Mini-tab (n = 1 each). Data collection software was split between Survey Monkey (n = 35; 40.2%), Qualtrics (n = 28; 32.3%), and Google Forms (n = 23; 26.4%).

The majority of respondents (86.2%; n = 75) reported teaching about communicating findings to stakeholders and priority populations (Subcompetency 4.7.1). The mean score for time spent teaching was 2.17, or just more than one class period. Creating charts was noted as a key element of data visualization to communicate evaluation results. How to create charts and tables was a commonly taught topic (67.8%; n = 59 and 63.2%; n = 55, respectively).

Discussion

The present study assessed the degree to which departments that offer undergraduate professional preparation programs in health education–related disciplines provide instruction in the HESPA 2015 Model evaluation competencies and subcompetencies. Competencies are composed of a set of subcompetencies that are either all entry-level (i.e., 3.4), all advanced-level (e.g., 4.5), or a combination (e.g., 4.4). Therefore, a discussion about the results will refer to both the competencies and associated subcompetencies.

To some degree, respondents teach all the competencies in their courses though they tend to spend the most time on competencies related to developing an evaluation plan, monitoring implementation (i.e., formative and process evaluation), and data collection instruments. As per HESPA definitions, only one of these three competencies where faculty spend the most time teaching has subcompetencies that are completely entry level (3.4: monitoring implementation). Developing an evaluation plan (4.1) is entirely advanced-level subcompetencies. For data collection instruments (Competency 4.3), only two subcompetencies (4.34 and 4.35) are considered entry level. In fact, the other eight data collection–related subcompetencies are advanced 2–level. Overall, of all 65 evaluation-related subcompetencies, the 5 that programs teach most often are all advanced level.

These data reveal a discrepancy between what is identified as entry-level competency in the HESPA framework and what is actually taught in professional preparation programs. One contributing factor to this discrepancy may be the absence of an overarching paradigm that helps connect the development of evaluation curriculum with the competencies and subcompetencies in the HESPA framework. Systematic use of formative and summative evaluation, a long-standing methodology in health education practice, is one such paradigm that might be useful. For example, the eight subcompetencies related to evaluation in Area of Responsibility III, all enumerated in Competency 3.4 (monitoring implementation), are closely associated with formative and process evaluation, which respectively involve quality improvement during program implementation and evaluation of implementation after program conclusion (McKenzie et al., 2017). Of the 57 evaluation subcompetencies in Area of Responsibility IV, only 9 are entry level, and these 9 also relate either directly or indirectly to formative or process evaluation.

All remaining evaluation competencies and subcompetencies in Area of Responsibility IV are designated as either advanced 1–level (n = 10) or advanced 2–level (n = 38), which may be more appropriately taught in graduate programs that generally involve more sophisticated elements of evaluation and research such as research design, data collection, and analysis and interpretation of results. These competencies tend to be more closely associated with summative (i.e., impact and outcome) evaluation (McKenzie et al., 2017). While formative and process evaluation may be as programmatically valuable as summative evaluation, they tend to be associated with less complex methodology and are more consistent with entry-level evaluation practice. An increased focus on formative and process evaluation at the undergraduate level with more attention on summative evaluation at the graduate level may better align evaluation curriculum in professional preparation programs to the HESPA 2015 Model competencies and subcompetencies.

The discrepancy between entry-level competencies listed in the HESPA 2015 Model framework and the content faculty select to include in undergraduate evaluation courses may be due to respondents either failing to distinguish between entry-level and advanced-level competencies and subcompetencies (i.e., they are not aware of or may not understand the HESPA framework), or intentionally blending competencies based on personal values or competing professional standards. Not all those teaching evaluation courses in professional programs are CHES-eligible. These instructors may be less likely to refer to the HESPA 2015 Model competencies when designing their evaluation courses.

The data in this study were collected 1 year following the release of the 2015 HESPA framework, which reclassified some of the competencies and subcompetencies as entry or advanced level. A lack of awareness of these changes is one explanation for the noted discrepancy.

Other factors, including personal values, competing professional standards, or the way HESPA 2015 Model competencies and subcompetencies are interpreted, may also influence the blending of entry- and advanced-level instruction in professional preparation. For example, in the earlier stages of credentialing, Luebke and Bohnenblust (1994) suggested that while the certification of health educators was based on a core set of responsibilities and competencies, many of those teaching in professional preparation programs were more inclined to structure content based on their perceptions of professional practice within specific settings. As described by Demers and Mamary (2008), “educational competencies used in formal academic settings may not perfectly relate to workforce competencies, which describe the skills, knowledge, and abilities of practitioners in their functional roles” (p. 2).

Faculty members may also tailor their instruction to the needs of local or regional employers or to a different set of professional standards (e.g., National Public Health Performance Standards, public health accreditation, and so forth) that may have different or at least nuanced standards for evaluation and research. For example, the American Evaluation Association (AEA) is currently drafting revised “evaluator competencies” for professional evaluators that are different than those listed in the HESPA framework (AEA, 2017).

The ways in which faculty members interpret or define HESPA 2015 Model competencies and subcompetencies may also contribute to the blending phenomenon identified in this study. While the profession has attempted to define key terminology in its Report of the 2011 Joint Committee on Health Education and Promotion Terminology (2012), only six terms in the report relate generally to evaluation and research and only four of those six terms relate explicitly. In a broad sense, the health education profession has not adequately defined and standardized terms related to evaluation and research competencies and subcompetencies. Thus, it is left to individual interpretation about what these terms mean and how they should be taught. The challenge related to interpretation may be further exasperated if those designing questions for the CHES examination are testing mastery of competencies based on definitions different from those used by faculty members in professional preparation programs.

Although the HESPA study produced a revised list of core knowledge items associated with the seven areas of responsibility, and the competencies and subcompetencies are published in an organized and easily readable format, there is no related glossary or notes section in which terms are adequately defined. While in several cases, terms and even complete descriptions of competencies are self-explanatory, others are not. For example, within Competency 3.4, it is not clear what is meant by “ensure that a plan is implemented consistently (3.4.3).” Does this mean the plan should be implemented consistently across all sites receiving a program or activity, or does it pertain to consistency compared with a timeline or logic model, or does it relate to sequential implementation (i.e., there is consistency between the first time the plan is implemented compared with subsequent implementation)? This is only one example of a potentially larger problem associated with inconsistencies in terms and definitions related to evaluation and research. Thus, there is a continuing need for professional preparation, professional practice, and credentialing and accreditation bodies to work toward consensus by standardizing definitions of terms related to evaluation and research.

It is encouraging that respondents in this study tended to teach principles related to evaluation more than research per se at the undergraduate level. For example, the three competencies with the highest proportion of advanced 2–level subcompetencies typically considered “research” in nature are taught less than the other four competencies.

This study demonstrates that evaluation is generally being taught at the undergraduate level within health education–related professional preparation programs. It also demonstrates that those who teach undergraduate evaluation courses blend and inconsistently apply the validated entry-level and advanced-level competencies identified in the HESPA framework. Additional research may help determine if this blending effect is unintentional or intentional as well as the underlying factors that influence course content selection.

Limitations and Strengths

What faculty members actually teach about evaluation in undergraduate programs may be somewhat different from what is reported in this study due to two factors. First, these data are self-report. Second, though multiple attempts were made to reach potential respondents, and an incentive was offered, the response rate was 54%, lower than the target of 70%.

This is the first assessment of its kind that we are aware of that measures what health education–related professional preparation programs teach about evaluation. The basis for the questionnaire was the competencies and subcompetencies validated in the HESPA, an established framework within the health education profession.

Implications for Practice

Given that scores related to evaluation are consistently among the lowest on the CHES exam, these findings are particularly relevant and timely. There are several implications for practice. While brief exposure to all evaluation and research concepts may hold some value to increased understanding of what comprises evaluation, we recommend that undergraduate programs consider emphasizing formative and process evaluation (i.e., the 17 entry-level subcompetencies as identified by the HESPA 2015 Model), thereby creating more depth and more clearly defining what it means to obtain entry-level competency in evaluation. Faculty in professional preparation may also consider teaching a stand-alone evaluation course instead of merging it with program planning. Including planning, implementation, and evaluation in one course may result in limited time for evaluation as it is often left to the end, when time may be short. Faculty in professional preparation programs may also consider including evaluation as part of a culminating experience through a capstone course or internship. We suspect the blending phenomenon identified in this study and the inconsistent application or teaching of competencies and subcompetencies, is partly due to a lack of standardized definitions related to evaluation. We recommend that the NCHEC and the SOPHE create a glossary of terms related to its own evaluation competencies and subcompetencies. Such a glossary could lead to increased standardization of evaluation terminology and further delineate between entry- and advanced-level standards. It could also guide course content selection, which may ensure a better prepared workforce.

It has been reported that evaluation is applied inconsistently in health promotion programs, that health promotion has failed to contribute to its own quality improvement, and that its practitioners lack knowledge, skills, and familiarity with evaluation methods (Kozica, Lombard, Hider, Harrison, & Teede, 2015). Furthermore, Lobo, Petrich, and Burns (2014) reported that health promotion practitioners view evaluation as a compliance process compared with a coherent and logical framework that can be used for program development. Drawing clear distinctions between undergraduate and graduate training related to evaluation, increasing the number of stand-alone courses in evaluation, including evaluation as part of a culminating experience, and standardizing evaluation and research terms and definitions within the profession may all help address these challenges.

Footnotes

Acknowledgements

We acknowledge Linda Lysoby and the National Commission for Health Education Credentialing for their technical assistance with this research study.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Rosemary Thackeray

References

American Evaluation Association. (2017). AEA evaluator competencies. Retrieved from http://www.eval.org/p/cm/ld/fid=51

Council on Education for Public Health. (2016). Accreditation criteria. Retrieved from https://ceph.org/criteria-revision/

Davidson

Telljohann

S. K.

Dake

J. A.

Price

J. H.

(2010). Institutions of higher education pre-service school health education practices. American Journal of Health Education, 41, 329-336. doi:10.1080/19325037.2010.10599161

Demers

A. R.

Mamary

E. M.

(2008). Assessing the professional development needs of public health educators in light of changing competencies. Preventing Chronic Disease: Public Health Research, Practice and Policy, 5(4), A129. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2578793/

Fisher

C. M.

Price

J. H.

Telljohann

S. K.

Dake

J. A.

(2015). A national assessment of colleges and university school health education methods courses. Journal of School Health, 85, 223-230. doi:10.1111/josh.12241

Glass

G. V.

Hopkins

K. D.

(1996). Statistical method in education and psychology (3rd ed.). Needham, MA: Allyn & Bacon.

Heitzer

J. G.-H.

McKenzie

J. F.

Amschler

D. H.

Bock

(2009). A descriptive analysis of patient education courses in undergraduate and graduate health education programs. Health Promotion Practice, 10, 244-253. doi:10.1177/1524839907309046

Joint Committee on Health Education and Promotion Terminology. (2012). Report of the 2011 Joint Committee on Health Education and Promotion Terminology. American Journal of Health Education, 43(Suppl. 2), 1-19. doi:10.1080/19325037.2012.11008225

Kozica

S. L.

Lombard

C. B.

Hider

Harrison

C. L.

Teede

J. H.

(2015). Developing comprehensive health promotion evaluations: A methodological review. Journal of Public Health, 2(1), 00007. doi:10.15406/mojph.2015.02.00007

10.

Linnan

L. A.

Sterba

K. R.

Lee

A. M.

Bontempi

J. B.

Yang

Crump

(2005). Planning and the professional preparation of health educators: Implications for teaching, research, and practice. Health Promotion Practice, 6, 308-319. doi:10.1177/1524839903260946

11.

Lobo

Petrich

Burns

S. K.

(2014). Supporting health promotion practitioners to undertake evaluation for program development. BMC Public Health. doi:10.1186/1471-2458-14-1315

12.

Luebke

J. K.

Bohnenblust

S. E.

(1994). Responsibilities and competencies: Implications for health education professional preparation programs. Journal of Health Education, 25, 227-229. doi:10.1080/10556699.1994.10608570

13.

McKenzie

J. F.

Neiger

B. L.

Thackeray

(2017). Planning, implementing and evaluating health promotion programs: A primer (7th ed.). Upper Saddle River, NJ: Pearson Education.

14.

Millar

M. M.

Dillman

D. A.

(2011). Improving response to web and mixed-mode surveys. Public Opinion Quarterly, 75, 249-269. doi:10.1093/poq/nfr003

15.

National Commission for Health Education Credentialing. (2017a). Health education job analysis projects. Retrieved from http://www.nchec.org/health-educator-job-analysis-projects

16.

National Commission for Health Education Credentialing. (2017b). Health education specialist practice analysis (HESPA) study: Executive summary. Retrieved from https://www.nchec.org/assets/2251/executive_summary.pdf

17.

National Commission for Health Education Credentialing & Society for Public Health Education. (2015). A competency-based framework for health education specialists–2015. Whitehall, PA: Author.