Promoting Quality of Program Delivery via an Internet Message Delivery System

Abstract

This article presents results from a study that evaluated an online message system designed to improve the delivery of prevention programs. We conducted a quasi-experimental study with 32 agencies and schools that implemented substance use prevention programs and examined differences between the comparison and intervention groups. We also examined the impact of dosage of the message system by comparing results among three groups of teachers: non-users, low users, and high users. Results for norm setting were marginally significant, such that teachers within the agencies assigned to the intervention condition scored higher on their understanding of norm setting at posttest compared to teachers within comparison agencies, after controlling for pretest knowledge scores and demographic items. In the model examining impact of dosage, high users of the intervention scored significantly higher on self-reported understanding of their program, quality of delivery, and program effectiveness compared to non-users. Low users of the intervention reported significantly higher quality of delivery compared to non-users.

Get full access to this article

View all access options for this article.

References

Abbott

R. D.

O'Donnell

Hawkins

J. D.

Hill

K. G.

Kosterman

Catalano

R. F.

(1998). Changing teaching practices to promote achievement and bonding to school. American Journal of Orthopsychiatry, 68, 542–552.

Aber

J. L.

Brown

J. L.

Jones

S. M.

(2003). Developmental trajectories toward violence in middle childhood: Course, demographic differences, and response to school-based intervention. Developmental Psychology, 39(2), 324–348.

August

G. L.

Bloomquist

M. L.

Lee

S. S.

Realmuto

G. M.

Hekter

J. M.

(2006). Can evidence-based prevention programs be sustained in community practice settings? The early risers advanced-stage effectiveness trial. Prevention Science, 7, 151–165.

Battistich

Schaps

Watson

Solomon

(1996). Prevention effects of the child development project: Early findings from an ongoing multisite demonstration trial. Special Issue: Preventing adolescent substance abuse. Journal of Adolescent Research, 11, 12–35.

Berman

McLaughlin

M. W.

(1976). Implementation of educational innovation. The Educational Forum, 40(3), 345–370.

Botvin

G. J.

Baker

Dusenbury

Tortu

Botvin

E. M.

(1990). Preventing adolescent drug abuse through a multimodal cognitive-behavioral approach: Results of a 3-year study. Journal of Consulting and Clinical Psychology, 58, 437–446.

Botvin

G. J.

Baker

Dusenbury

Botvin

E. M.

Diaz

(1995). Long-term follow-up results of a randomized drug abuse prevention trial in a white middle-class population. Journal of the American Medical Association, 273, 1106–1112.

Burke

R. V.

Oats

R. G.

Ringle

J. L.

Fichtner

L. O. N.

DelGaudio

M. B.

(2011). Implementation of a classroom management program with urban elementary schools in low-income neighborhoods: Does program fidelity affect student behavior and academic outcomes? Journal of Education for Students Placed at Risk (JESPAR), 16(3), 201–218.

Collaborative for Academic, Social, and Emotional Learning. (2003). Safe and sound: An educational leaders' guide to social, and emotional learning. Chicago, IL: CASEL.

10.

Dane

A. V.

Schneider

B. H.

(1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18(1), 23–45.

11.

Roux

A. V. Diez

(2002). A glossary of multilevel analysis. Journal of Epidemiology and Community Health, 56, 588–594.

12.

Domitrovich

C. E.

Greenberg

M. T.

(2000). The study of implementation: Current findings from effective programs that prevent mental disorders in school-aged children. Journal of Educational and Psychological Consultation, 11(2), 293–221.

13.

Domitrovich

C. E.

Bradshaw

Poduska

Becker

Ialongo

(2011). Enhancing the fidelity of a classroom-based, social emotional intervention through coaching. Paper presented at the Society for Research in Child Development, Montreal, Canada.

14.

Drug Strategies. (1999). Making the grade: A guide to school drug prevention programs. Washington, DC: Drug Strategies.

15.

Durlak

J. A.

Weissberg

R. P.

Dymnicki

A. B.

Taylor

R. D.

Schellinger

K. B.

(2011). The impact of enhancing students' social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82(1), 405–532.

16.

Durlak

DuPre

(2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3), 327–350.

17.

Dusenbury

Brannigan

Falco

Hansen

W. B.

(2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research, 18, 237–256.

18.

Dusenbury

Hansen

W. B.

Giles

(2003). Teacher training in norm setting approaches to drug education: A pilot study comparing standard and video-enhanced methods. Journal of Drug Education, 33(3), 325–336.

19.

Dusenbury

Brannigan

Falco

Lake

(2004). An exploration of fidelity of implementation in drug abuse prevention among five professional groups. Journal of Alcohol and Drug Education, 47(3), 4–18.

20.

Dusenbury

Brannigan

Hansen

W. B.

Walsh

Falco

(2005). Quality of implementation: Developing measures crucial to understanding the diffusion of preventive interventions. Health Education Research, 20(3), 308–313.

21.

Gottfredson

D. C.

Gottfredson

(2002). Quality of school-based prevention programs: Results for a national survey. Journal of Research in Crime and Delinquency, 39(1), 3–35.

22.

Hansen

W. B.

Dusenbury

Bishop

Derzon

J. H.

(2007). Substance abuse prevention program content: Systematizing the classification of what programs target for change. Health Education Research, 22, 351–360.

23.

Hansen

W. B.

McNeal

R. B.

(1999). Drug education practice: Results of an observational study. Health Education Research, 14(1), 85–97.

24.

Hansen

W. B.

Pankratz

M. M.

Dusenbury

Giles

S. M.

Bishop

D. C.

Albritton

(2013). Styles of adaptation: The impact of frequency and valence of adaptation on preventing substance abuse. Health Education, 113(4), 345–363. ISSN 0965-4283.

25.

Jones

S. M.

Brown

J. L.

Aber

(2011). Two-year impacts of a universal school-based social-emotional and literacy intervention: An experiment in translational developmental research. Child Development, 82(2), 533–554.

26.

Kalafat

Illback

R. J.

Sanders

Jr. (2007). The relationship between implementation fidelity and educational outcomes in a school-based family support program: Development of a model for evaluating multidimensional full-service programs. Evaluation and Program Planning, 30(2), 136–148.

27.

Kallestad

J. H.

Olweus

(2003). Predicting teachers' and schools' implementation of the Olweus bullying prevention program: A multilevel study. Prevention and Treatment, 6(1), article 21. doi 10.1037/1522-3736.6.1.621a

28.

Payne

A. A.

Gottfredson

D. C.

Gottfredson

G. D.

(2006). School predictors of the intensity of implementation of school-based prevention programs: Results from a national study. Prevention Science, 7(2), 225–237.

29.

Raudenbush

Byrk

(2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks: Sage Publications.

30.

Resnicow

Cross

Wynder

(1993). The know your body program: A review of evaluation studies. Bulletin of the New York Academy of Medicine, 70, 188–207.

31.

Ringwalt

C. L.

Ennett

Vincus

Thorne

Rohrbach

L. A.

Simons-Rudolph

(2002). The prevalence of effective substance use prevention curricula in U.S. middle schools. Prevention Science, 3, 257–265.

32.

Ringwalt

C. L.

Pankratz

M. M.

Jackson-Newsom

Gottfredson

N. C.

Hansen

W. B.

Giles

S. M.

(2010). Three-year trajectory of teachers' fidelity to a drug prevention curriculum. Prevention Science, 11(1), 67–76.

33.

Ringwalt

C. L.

Vincus

Hanley

Ennett

Bowling

Haws

(2011). The prevalence of evidence-based drug use prevention curricula in U.S. middle schools in 2008. Prevention Science, 13, 63–69.

34.

Ringwalt

C. L.

Vincus

Hanley

Ennett

Bowling

Rohrbach

L. A.

(2009). The prevalence of evidence-based drug use prevention curricula in U.S. middle schools in 2005. Prevention Science, 10, 33–40.

35.

Rohrbach

L. A.

Graham

J. W.

Hansen

W. B.

(1993). Diffusion of a school-based substance abuse prevention program: Predictors of program implementation. Preventive Medicine, 22, 237–260.

36.

Rohrbach

L. A.

Gunning

Sun

Sussman

(2010). The Project Towards No Drug Abuse (TND) dissemination trial: Implementation fidelity and immediate outcomes. Prevention Science, 11(1), 77–88.

37.

Ross

J. G.

Luepker

R. V.

Nelson

G. D.

Saavedra

Hubbard

B. M.

(1991). Teenage health teaching modules: Impact of teacher training on implementation and student outcomes. Journal of School Health, 61(1), 31–34.

38.

Smith

D. W.

McCormick

L. K.

Steckler

A. B.

McLeroy

K. R.

(1993). Teachers' use of health curricula: Implementation of growing healthy, project SMART, and the teenage health teaching modules. Journal of School Health, 63(8), 349–354.

39.

Tappe

M. K.

Galer-Unti

R. A.

Bailey

K. C.

(1995). Long-term implementation of the Teenage Health Teaching Modules by trained teachers: A case study. Journal of School Health, 65(10), 411–414.