Monitoring the implementation of a program being evaluated can improve the interpretability of data collected and help evaluators to avoid committing a Type III error: evaluating a program that has not been adquately implemented. This article describes an evaluation that analyzed the implementation of a school health education curriculum, assessed cognitive learning outcomes attributable to the curriculum, and examined the relationship between classroom implementation and changes in students' knowledge. Five fifth-grade classes (n = 101) participated in the curriculum, and five classes (n = 84) served as a comparison group. Data collection procedures involved a pretest and posttest of all students' health-related knowledge, daily monitoring of classroom implementation by the five teachers participating, and questionnaires completed by principals and teachers. Analysis methods included descriptive statistics, parametric and non- parametric tests of significance, and qualitative assessment procedures. Results indicated that the curriculum had a positive effect on learning in students; curriculum implementation varied considerably among the five classes participating; teaching/learning activities that were most and least likely to be implemented could be identified and described; both teachers and principals perceived the program favorably: some health instruction was occurring in the comparison classes, so it was not appropriate to consider them as pure controls; and no statistically significant relationship between curriculum implementation and cognitive outcomes was observed. This study provides evidence of the need for and value of measuring implementation of programs being evaluated. Implications for developing implementation measures and the role of formative evaluation in health education practice are considered.
Get full access to this article
View all access options for this article.
References
1.
Cook TDCampbell DT: Quasi-Experimentation: Design and Analysis Issues for Field Settings. Chicago, Rand McNally, 1979.
2.
Dobson D., Cook TJ: Avoiding type III errors in program evaluation: Results from a field experiment . Eval Prog Plann3:269-276, 1980.
3.
Rezmovic EL: Program implementation and evaluation results: A reexamination of Type III error in a field experiment. Eval Prog Plann5:111-118. 1982.
4.
Scanlon JW, Horst P., Nay JN et al: Evaluability assessment: Avoiding type III and IV errors, in Gilbert GR, Conklin PJ (eds): Evaluation Management: A Source Book of Readings. Charlottesville, VA. US Civil Service Commission, 1977.
5.
Green LW: Evaluation and measurement: Some dilemmas for health education. Am J Publ Health67:151-156. 1977.
6.
Kreuter MW, Green LW: Evaluation of school health education: Identifying purpose, keeping perspective . J Sch Health48:228-235. 1978.
7.
Kolbe LJ: Evaluating effectiveness: The problems of behavioral criteria. Health Educ10:12-13, 1979.
8.
Green LW, Heit P., Iverson DC, et al: The school health curriculum project: Its theory, practice, and measurement experience. Health Educ Q7:14-34, 1980.
9.
Baric L.: Evaluation: Obstacles and potentialities. Int J Health Educ23:142-149, 1980.
10.
Hochbaum GM : Certain problems in evaluating health educaton . Health Values6:14-20. 1982.
11.
Levine S., Kolbe LJ: Making a difference: Formative evaluation of health education. Focal Points, September, 1983.
12.
Campbell DT, Stanley JC: Experimental and Quasi-Experimental Designs for Research. Chicago, Rand McNally. 1963.
13.
Guttentag M.Struening EL: The handbook: Its purpose and organization, in Guttentag M.Struening EL (eds): Handbook of Evaluation Research (vol 2). Beverly Hills, Sage, 1975.
14.
Boruch R., Gomez H.: Sensitivity, bias, and theory in impact evaluation. Prof Psychol8:411-434, 1977.
15.
Morris L., Fitz-Gibbon C.: How to Measure Program Implementation. Beverly Hills, Sage, 1978.
16.
Patton MQ: Evaluation of program implementation, in Sechrest L.West SG, Phillips M. (eds): Evaluation Studies: Review Annual (vol 4). Beverly Hills , Sage, 1979.
17.
Williams W.Elmore RF: Social Program Implementation. New York, Academic Press, 1976.
18.
Elmore RF: Backward mapping: Implementation research and policy decisions, in Williams W (ed): Studying Implementation: Methodological and Administrative Issues. Chatham, Chatham House. 1982.
19.
Charters WWJones JE: On the risks of appraising non-events in program evaluation. Educ Res2:5-7, 1973.
20.
Hall GE, Loucks SF: A developmental model for determining whether treatment is actually implemented . Am Educ Res J3:263-276, 1977.
Hall GE, Loucks SF: Bridging the gap: Policy research rooted in practice, in Lieberman A, McLaughlin MW (eds): Policy Making in Education: Eighty-First Yearbook of the Society for the Study of Education. Chicago. Society tor the Study of Education, 1982.
23.
Joint Commission on Standards for Educational Evaluation: Standards for Evaluation of Educational Programs, Projects, and Materials. New York. McGraw Hill, 1981.
24.
Basch CE: Research on disseminating and implementing school health education. J Sch Health54:57-66, 1984.
25.
Leithwood KA , Montgomery DJ: Evaluating program implementation. Eval Rev4:193-214, 1980.
26.
Williams W.: Implementation analysis and assessment, in Cook TD.Delrosario ML.Hennigan KM, et al (eds): Evaluation Studies: Review Annuals (vol 3). Beverly Hills, Sage, 1978.
27.
Fullen M.Pomfret A.: Research on curriculum and instruction implementation. Rev Educ Res47:335-397, 1977.
28.
Goodlad JI: Curriculum Inquiry: The Study of Curriculum Practice. New York, McGraw Hill. 1981.
29.
Freeman HE: The present status of evaluation research, in Guttentag M (ed): Evaluation Studies: Review Annual (vol 2). Beverly Hills, Sage, 1977.
30.
Patton MQ: Utilization Focused Evaluation. Beverly Hills, Sage, 1978.
31.
Rossi PH: Issues in evaluation of human services delivery. Eval Q2:573-599, 1978.
32.
Sechrest L.. West SGPhillips M., et al: Some neglected aspects in evaluation research, in Sechrest L, West SG, Phillips M, et al (eds): Evaluation Studies: Review Annual (vol 4). Beverly Hills, Sage. 1979.
33.
Boruch RF, Cordray DS, Pion G., et al: A mandated appraisal of evaluation practices: Digest of recommendations to Congress and to the Department of Education. Educ Res10:10-13, 1981.
34.
James RK, Hall G.: A study of the concerns of science teachers regarding an implementation of ISCS. J Res Sci Teach18:479-487, 1981.
35.
Tomatzky LG, Johnson EC: Research on implementation: Implications for evaluation practice and evaluation policy. Eval Prog Plann5:193-198, 1982.
36.
Gross N., Giacquinto JB, Bernstein M.: Implementing Organizational Innovations: A Sociological Analysis of Planned Social Change. New York, Basic Books , 1971.
37.
Shipman MD: The impact of a curriculum project. J Curric Stud.5:47-54, 1973.
Leinhardt G.: Program evaluation: An empirical study of individualized instruction. Am Educ Res J14:277-293, 1977.
40.
Leinhardt G.: Applying a classroom process model to instructional evaluation. Curr Inquiry8:155-176, 1978.
41.
Gibney T.Karns E.: Mathematics education—1955-1975: A summary of findings. Educ Lead36:357-359. 1979.
42.
Yager RE, Stodghill R.: Social science in an age of reason. Educ Lead36:439-445, 1979.
43.
Ponder G.: The more things change...: The status of social studies. Educ Lead6:515-518, 1979.
44.
Leinhardt G. : Modeling and measuring educational treatment in evaluation. Rev Educ Res50:393-420, 1980.
45.
Cooley WW, Leinhardt G.: The instructional dimensions study. Educ Eval Pol Anal2:7-25. 1980.
46.
Yin RK: Changing urban bureaucracies: How new practices becomes routinized. Lexington, Lexington Books, 1979 .
47.
St Pierre RG, Cook TD, Shaw R.: An Evaluation of the Nutrition Education Training Program: Nebraska's Experience Nutrition Curriculum. Cambridge, ABT Associates, 1981.
48.
Gersten RM, Carnine DW, Williams PB: Measuring implementation of a structured educational model in an urban school district: An observational approach. Educ Eval Pol Anal4:67-80, 1982.
49.
Basch CE: Evaluation of a School Health Educational Curriculum: Implementation Analysis and Outcome Assessment. Unpublished doctoral dissertation, Southern Illinois University, Carbondale, IL, 1982.
50.
Conrad KJ, Eash MJ: Measuring implementation and multiple outcomes in a child parent center compensatory education center program. Am Educ Res J20:221-236, 1983.
51.
Scheirer MA , Rezmovic EL: Measuring the degree of implementation. Eval Rev7:599-634, 1983.
52.
Kreuter MW, Christenson GM, Davis R.: School health education research: Future issues and challenges. J Sch Health54:27-32, 1984.
53.
Basch CE, Sliepcevich EM: Innovators, innovations and implementation: A framework for curricular research in school health education. Health Educ14:20-24, 1983.
54.
Foster RL: The fifth graders are excited about how their lungs work. J Natl Tuberc Respir Dis Assoc12:4-9. 1971.
55.
Foster RL: A search for inclusion. Health Educ Monogr36:56-60, 1973.
56.
Davis RL: Making health education relevant and exciting. Healh Ser Rep88:100-105, 1973.
57.
Edson L.: Schools attack the smoking problem. Am Educ9:10-14, 1973.
58.
Albino J., Davis RL: A health education program that works. Phi Delta Kappa54:256-259, 1975.
59.
Bruess CE: School health education: General gains from specific tasks. J Sch Health49:93-95, 1979.
60.
School Health Curriculum Project. Atlanta: Bureau of Health Education, DHHS Pub. No. (CDC) 80-8359, 1977.
61.
Evans NL, Middleton KH: Two programs exemplify nationwide involvement in school health education. Health Educ10:42-45, 1979.
62.
Olsen LK, Redican KJ, Krus PH: The school health curriculum project: A review of research studies. Health Educ11: 16-21, 1980.
63.
Popham WJ: Appropriate measuring instruments for health education. Health Educ13:23-26, 1982.
64.
Iverson DC: Issues related to designing and conducting school health education research . J Sch Health54:50-56, 1984.
65.
Heit P.: The Berkeley model. Healh Educ8:2-3, 1977.
66.
Yin RD: Studying the implementation of public programs, in Williams W (ed): Studying implementation: Methodological and administrative issues. Chatham, NC, Chatham Housc, 1982.
67.
Mullen PD, Iverson DC: Qualitative methods for evaluation research in health education program. Health Educ13:11-18. 1982.
