Barriers and Facilitators to Program Evaluation and Dissemination: A Qualitative Study to Inform Implementation Strategies

Abstract

Implementation strategies support the adoption, implementation, and maintenance of evidence—based interventions in community settings. However, strategies often conclude when funding ends, leaving implementers without assistance to maintain interventions. Implementation strategies focused on program evaluation and dissemination—factors that lead to program maintenance—are needed. The goal of this study was to understand barriers and facilitators to evaluation and dissemination in community-based organizations to inform relevant implementation strategies. Semi-structured interviews were conducted using an interview guide based on Consolidated Framework for Implementation Research (CFIR) constructs hypothesized to influence evaluation and dissemination: complexity, self-efficacy, culture, available resources, leadership engagement, organizational incentives and rewards, external policies and incentives, reflecting and evaluating, and champions. The guide also queried implementation strategy preferences. A rapid deductive approach was used to analyze the data through a template aligned with the interview guide. Nine interviewees shared barriers around reflecting and evaluating (e.g., data collection challenges), available resources (e.g., organizational capacity), and organizational incentives and rewards (e.g., lack of recognition). Interviewees favored ongoing consultation, tools or plans, problem-solving support, and peer sharing. Based on these results, a technical assistance (TA) protocol was developed to include monthly consultation meetings and quarterly learning collaboratives to build capacity in project partner (UnitedHealthcare) grantees. Future work should evaluate the TA protocol for effectiveness in alleviating the identified barriers and improving program implementers’ evaluation and dissemination capacity. If successful, this model could be used in other community settings experiencing similar barriers, ultimately leading to improved maintenance of evidence-based interventions.

Keywords

implementation strategies evaluation dissemination community-based organizations food access technical assistance

Get full access to this article

View all access options for this article.

References

Aarons

G. A.

Hurlburt

Horwitz

S. M.

(2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 4–23. https://doi.org/10.1007/s10488-010-0327-7

Aarons

G. A.

Sklar

Mustanski

Benbow

Hendricks Brown

(2017). “Scaling-out” evidence-based interventions to new populations or new health care delivery systems. Implementation Science, 12, 111. https://doi.org/10.1186/s13012-017-0640-6

Bach-Mortensen

A. M.

Lange

B. C. L.

Montgomery

(2018). Barriers and facilitators to implementing evidence-based interventions among third sector organisations: A systematic review. Implementation Science, 13, 103. https://doi.org/10.1186/s13012-018-0789-7

Balis

L. E.

Harden

S. M.

(2021). “Replanning” a statewide walking program through the iterative use of the reach, effectiveness, adoption, implementation, and maintenance framework. Journal of Physical Activity & Health, 18(10), 1310–1317. https://doi.org/10.1123/jpah.2021-0034

Balis

L. E.

Houghtaling

Clausen

Lane

Wende

M. E.

Pereira

McLoughlin

G. M.

Harden

S. M.

(2024). Advancing implementation science in community settings: The implementation strategies applied in communities (ISAC) compilation. International Journal of Behavioral Nutrition and Physical Activity, 21(1), 132. https://doi.org/10.1186/s12966-024-01685-5

Balis

L. E.

Houghtaling

Harden

S. M.

(2022). Using implementation strategies in community settings: An introduction to the Expert Recommendations for Implementing Change (ERIC) compilation and future directions. Translational Behavioral Medicine, 12(10), 965–978. https://doi.org/10.1093/tbm/ibac061

Balis

L. E.

John

Harden

(2019). Beyond evaluation: Using the RE-AIM framework for program planning in extension. Journal of Extension, 57(2), 2T0T1.

Balis

Houghtaling

(2023) Matching barriers and facilitators to implementation strategies: Recommendations for community settings. Implementation Science Communications, 4, 144.

Balis

L. E.

Strayer

Ramalingam

Wilson

Harden

(2019). Open-access physical activity programs for older adults: A pragmatic and systematic review. The Gerontologist, 59(4), e268–e278. https://doi.org/10.1093/geront/gnx195

10.

Birken

S. A.

Haines

E. R.

Hwang

Chambers

D. A.

Bunger

A. C.

Nilsen

(2020). Advancing understanding and identifying strategies for sustaining evidence-based practices: A review of reviews. Implementation Science, 15(1), 88. https://doi.org/10.1186/s13012-020-01040-9

11.

Brand

Pischke

C. R.

Steenbock

Schoenbach

Poettgen

Samkange-Zeeb

Zeeb

(2014). What works in community-based interventions promoting physical activity and healthy eating? A review of reviews. International Journal of Environmental Research and Public Health, 11(6), 5866–5888. https://doi.org/10.3390/ijerph110605866

12.

Brownson

R. C.

Eyler

A. A.

Harris

J. K.

Moore

J. B.

Tabak

R. G.

(2018). Getting the word out: New approaches for disseminating public health science. Journal of Public Health Management and Practice, 24(2), 102–111. https://doi.org/10.1097/PHH.0000000000000673

13.

Brownson

R. C.

Jacobs

J. A.

Tabak

R. G.

Hoehner

C. M.

Stamatakis

K. A.

(2012). Designing for dissemination among public health researchers: Findings from a National Survey in the United States. American Journal of Public Health, 103(9), 1693–1699. https://doi.org/10.2105/AJPH.2012.301165

14.

Centers for Disease Control and Prevention. (2019). High obesity program. https://www.cdc.gov/nccdphp/dnpao/state-local-programs/hop-1809/high-obesity-program-1809.html

15.

Chambers

D. A.

Glasgow

R. E.

Stange

K. C.

(2013). The dynamic sustainability framework: Addressing the paradox of sustainment amid ongoing change. Implementation Science, 8, 117. https://doi.org/10.1186/1748-5908-8-117

16.

Chen

A. M. H.

Draime

J. A.

Berman

Gardner

Krauss

Martinez

(2022). Food as medicine? Exploring the impact of providing healthy foods on adherence and clinical and economic outcomes. Exploratory Research in Clinical and Social Pharmacy, 5, 100129. https://doi.org/10.1016/j.rcsop.2022.100129

17.

Consolidated Framework for Implementation Research Team. (2025). Consolidated framework for implementation research. Center for Clinical Management Research. http://www.cfirguide.org/

18.

Damschroder

L. J.

Aron

Keith

Kirsh

Alexander

Lowery

(2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4, 50. https://doi.org/10.1186/1748-5908-4-50

19.

Damschroder

L. J.

Reardon

C. M.

Widerquist

M. A. O.

Lowery

(2022). The updated consolidated framework for implementation research based on user feedback. Implementation Science, 17(1), 75. https://doi.org/10.1186/s13012-022-01245-0

20.

Domlyn

A. M.

Scott

Livet

Lamont

Watson

Kenworthy

Talford

Yannayon

Wandersman

(2021). R = MC2 readiness building process: A practical approach to support implementation in local, state, and national settings. Journal of Community Psychology, 49(5), 1228–1248. https://doi.org/10.1002/jcop.22531

21.

Downer

Berkowitz

S. A.

Harlan

T. S.

Olstad

D. L.

Mozaffarian

(2020). Food is medicine: Actions to integrate food and nutrition into healthcare. BMJ, 369, m2482. https://doi.org/10.1136/bmj.m2482

22.

Finley

Hamilton

Kowalski

Midboe

Nevedal

Young

(2022). Rapid qualitative methods in implementation science: Techniques and considerations [Conference session]. 15th Annual Conference on the Science of Dissemination and Implementation in Health, Washington, DC, United States.

23.

Gale

Erhardt

Bounthavong

Reardon

Damschroder

Midboe

(2019). Comparison of rapid vs in-depth qualitative analytic methods from a process evaluation of academic detailing in the Veterans Health Administration. Implementation Science, 14, 11. https://doi.org/10.1186/s13012-019-0853-y

24.

Glasgow

Harden

Gaglio

Rabin

Smith

Porter

Ory

Estabrooks

(2019). RE-AIM planning and evaluation framework: Adapting to new science and practice with a 20-year review. Frontiers in Public Health, 7, 64. https://doi.org/10.3389/fpubh.2019.00064

25.

Glasgow

Klesges

Dzewaltowski

Bull

Estabrooks

(2004). The future of health behavior change research: What is needed to improve translation of research into health promotion practice? Annals of Behavioral Medicine, 27(1), 3–12.

26.

Glasgow

Vogt

Boles

(1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 9, 1322–1327.

27.

Green

(2008). Making research relevant: If it is an evidence-based practice, where’s the practice-based evidence? Family Practice, 25(1), i20–i24. https://doi.org/10.1093/fampra/cmn055

28.

Guest

Namey

Taylor

Eley

McKenna

(2017). Comparing focus groups and individual interviews: Findings from a randomized study. International Journal of Social Research Methodology, 20(6), 693–708. https://doi.org/10.1080/13645579.2017.1281601

29.

Gus Schumacher Nutrition Incentive Program Nutrition Incentive Program Training Technical Assistance Evaluation Information Center. (n.d.). Nutrition Incentive Hub. https://www.nutritionincentivehub.org/

30.

Hamilton

(2013). Qualitative methods in rapid turn-around health services research. VA HSR&D Cyberseminar Spotlight on Women’s Health.

31.

Harden

S. M.

Smith

Smith-Ray

Ory

Estabrooks

Glasgow

(2018). RE-AIM in clinical, community, and corporate settings: Perspectives, strategies, and recommendations to enhance public health impact. Frontiers in Public Health, 6, 71. https://doi.org/10.3389/fpubh.2018.00071

32.

Harden

S. M.

Steketee

Glasgow

R. E.

Estabrooks

P. A.

(2021). Suggestions for advancing pragmatic solutions for dissemination: Potential updates to evidence-based repositories. American Journal of Health Promotion, 35(2), 289–294. https://doi.org/10.1177/0890117120934619

33.

Helfrich

C. D.

Y.-F.

Sharp

N. D.

Sales

A. E.

(2009). Organizational readiness to change assessment (ORCA): Development of an instrument based on the Promoting Action on Research in Health Services (PARIHS) framework. Implementation Science, 4(1), 38. https://doi.org/10.1186/1748-5908-4-38

34.

Hilts

K. E.

Yeager

V. A.

Gibson

P. J.

Halverson

P. K.

Blackburn

Menachemi

(2021). Hospital partnerships for population health: A systematic review of the literature. Journal of Healthcare Management / American College of Healthcare Executives, 66(3), 170–198. https://doi.org/10.1097/JHM-D-20-00172

35.

Houghtaling

Balis

Pradhananga

Cater

Holston

(2023). Healthy eating and active living policy, systems, and environmental changes in rural Louisiana: A contextual inquiry to inform implementation strategies. International Journal of Behavioral Nutrition and Physical Activity, 20(1), 132. https://doi.org/10.1186/s12966-023-01527-w

36.

Kasprzak

C. M.

Lally

A. E.

Schoonover

J. J.

Gallicchio

Haynes-Maslow

Vermont

L. N.

Ammerman

A. S.

Raja

Tumiel-Berhalter

Tirabassi

J. N.

Leone

L. A.

(2022). Operational challenges that may affect implementation of evidence-based mobile market interventions. BMC Public Health, 22(1), 776. https://doi.org/10.1186/s12889-022-13207-8

37.

King

Hawe

Wise

(1998). Making dissemination a two-way process. Health Promotion International, 13(3), 237–244. https://doi.org/10.1093/heapro/13.3.237

38.

Kononowech

Hagedorn

Hall

Helfrich

C. D.

Lambert-Kerzner

A. C.

Miller

S. C.

Sales

A. E.

Damschroder

(2021). Mapping the organizational readiness to change assessment to the Consolidated Framework for Implementation Research. Implementation Science Communications, 2(1), 19. https://doi.org/10.1186/s43058-021-00121-0

39.

Kowalski

C. P.

Nevedal

A. L.

Finley

E. P.

Young

J. P.

Lewinski

A. A.

Midboe

A. M.

Hamilton

A. B.

(2024). Planning for and Assessing Rigor in Rapid Qualitative Analysis (PARRQA): A consensus-based framework for designing, conducting, and reporting. Implementation Science, 19(1), 71. https://doi.org/10.1186/s13012-024-01397-1

40.

Leeman

Birken

S. A.

Powell

B. J.

Rohweder

Shea

C. M.

(2017). Beyond “implementation strategies”: Classifying the full range of strategies used in implementation science and practice. Implementation Science, 12(1), 125. https://doi.org/10.1186/s13012-017-0657-x

41.

Luke

Calhoun

Robichaux

Elliot

Moreland-Russell

(2014). The program sustainability assessment tool: A new instrument for public health programs. Preventing Chronic Disease, 11, 130184.

42.

Mazzucca

Arredondo

E. M.

Hoelscher

D. M.

Haire-Joshu

Tabak

R. G.

Kumanyika

S. K.

Brownson

R. C.

(2021). Expanding implementation research to prevent chronic diseases in community settings. Annual Review of Public Health, 42(1), 135–158. https://doi.org/10.1146/annurev-publhealth-090419-102547

43.

McNett

Tucker

Zadvinskis

Tolles

Thomas

Gorsuch

Gallagher-Ford

(2022). A qualitative force field analysis of facilitators and barriers to evidence-based practice in healthcare using an implementation framework. Global Implementation Research and Applications, 2(3), 195–208. https://doi.org/10.1007/s43477-022-00051-6

44.

Middel

C. N. H.

Schuitmaker-Warnaar

T. J.

Mackenbach

J. D.

Broerse

J. E. W.

(2019). Systematic review: A systems innovation perspective on barriers and facilitators for the implementation of healthy food-store interventions. International Journal of Behavioral Nutrition and Physical Activity, 16(1), 108. https://doi.org/10.1186/s12966-019-0867-5

45.

Moullin

J. C.

Dickson

K. S.

Stadnick

N. A.

Rabin

Aarons

G. A.

(2019). Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Implementation Science, 14(1), 1. https://doi.org/10.1186/s13012-018-0842-6

46.

Murriel

A. L.

Kahin

Pejavara

O’Toole

(2020). The high obesity program: Overview of the centers for disease control and prevention and cooperative extension services efforts to address obesity. Preventing Chronic Disease, 17, 190235. https://doi.org/10.5888/pcd17.190235

47.

Nathan

Powell

B. J.

Shelton

R. C.

Laur

C. V.

Wolfenden

Hailemariam

Yoong

S. L.

Sutherland

Kingsland

Waltz

T. J.

Hall

(2022). Do the Expert Recommendations for Implementing Change (ERIC) strategies adequately address sustainment? Frontiers in Health Services, 2, 905909. https://doi.org/10.3389/frhs.2022.905909

48.

Nevedal

A. L.

Reardon

C. M.

Opra Widerquist

M. A.

Jackson

G. L.

Cutrona

S. L.

White

B. S.

Damschroder

L. J.

(2021). Rapid versus traditional qualitative analysis using the Consolidated Framework for Implementation Research (CFIR). Implementation Science, 16(1), 67. https://doi.org/10.1186/s13012-021-01111-5

49.

Newcomer

K. E.

Hatry

H. P.

Wholey

J. S.

(2015). Planning and designing useful evaluations. In Newcomer

K. E.

Hatry

H. P.

Wholey

J. S.

(Eds.), Handbook of practical program evaluation (pp. 7–35). Wiley. https://doi.org/10.1002/9781119171386.ch1

50.

Ohio Department of Health. (n.d.). Creating healthy communities. https://odh.ohio.gov/know-our-programs/health-promotion/local-communities/local-communities

51.

Palinkas

L. A.

Aarons

G. A.

Horwitz

Chamberlain

Hurlburt

Landsverk

(2011). Mixed method designs in implementation research. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 44–53. https://doi.org/10.1007/s10488-010-0314-z

52.

Powell

B. J.

Waltz

T. J.

Chinman

M. J.

Damschroder

L. J.

Smith

J. L.

Matthieu

M. M.

Proctor

E. K.

Kirchner

J. E.

(2015). A refined compilation of implementation strategies: Results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science, 10, 21. https://doi.org/10.1186/s13012-015-0209-1

53.

Proctor

E. K.

Bunger

A. C.

Lengnick-Hall

Gerke

D. R.

Martin

J. K.

Phillips

R. J.

Swanson

J. C.

(2023). Ten years of implementation outcomes research: A scoping review. Implementation Science, 18(1), 31. https://doi.org/10.1186/s13012-023-01286-z

54.

Rabin

Brownson

Haire-Joshu

Kreuter

Weaver

(2008). A glossary for dissemination and implementation research in health. Journal of Public Health Management and Practice, 14(2), 117–123. https://doi.org/10.1097/01.PHH.0000311888.06252.bb

55.

Scaccia

J. P.

Cook

B. S.

Lamont

Wandersman

Castellow

Katz

Beidas

R. S.

(2015). A practical implementation science heuristic for organizational readiness: R = MC. Journal of Community Psychology, 43(4), 484–501. https://doi.org/10.1002/jcop.21698

56.

Stotz

S. A.

Budd Nugent

Ridberg

Byker Shanks

Her

Yaroch

A. L.

Seligman

(2022). Produce prescription projects: Challenges, solutions, and emerging best practices—Perspectives from health care providers. Preventive Medicine Reports, 29, 101951. https://doi.org/10.1016/j.pmedr.2022.101951

57.

Tabak

R. G.

Reis

R. S.

Wilson

Brownson

R. C.

(2015). Dissemination of health-related research among scientists in three countries: Access to resources and current practices. BioMed Research International, 2015, e179156. https://doi.org/10.1155/2015/179156

58.

Tabak

R. G.

Stamatakis

K. A.

Jacobs

J. A.

Brownson

R. C.

(2014). What predicts dissemination efforts among public health researchers in the United States? Public Health Reports, 129(4), 361–368. https://doi.org/10.1177/003335491412900411

59.

Taher

Muramatsu

Odoms-Young

Peacock

Michael

C. F.

Courtney

K. S

. (2022). An embedded multiple case study: Using CFIR to map clinical food security screening constructs for the development of primary care practice guidelines. BMC Public Health, 22(1), 97. https://doi.org/10.1186/s12889-021-12407-y

60.

Trapl

E. S.

Smith

Joshi

Osborne

Benko

Matos

A. T.

Bolen

(2018). Dietary impact of produce prescriptions for patients with hypertension. Preventing Chronic Disease, 15, E138. https://doi.org/10.5888/pcd15.180301

61.

UnitedHealthcare. (n.d.). Our vision for empowering health. https://www.uhc.com/about-us/community-impact/empowering-health

62.

U.S. Department of Agriculture. (2019). SNAP-Ed toolkit: Obesity prevention interventions and evaluation framework. https://snapedtoolkit.org/

63.

Wolfenden

Reilly

Kingsland

Grady

Williams

C. M.

Nathan

Sutherland

Wiggers

Jones

Hodder

Finch

McFadyen

Bauman

Rissel

Milat

Swindle

Yoong

S. L.

(2019). Identifying opportunities to develop the science of implementation for community-based non-communicable disease prevention: A review of implementation trials. Preventive Medicine, 118, 279–285. https://doi.org/10.1016/j.ypmed.2018.11.014

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB