The Data-to-Action Framework

Abstract

Although health education programs may benefit from quality improvement methods, scant resources exist to help practitioners apply these methods for program improvement. The purpose of this article is to describe the Data-to-Action framework, a process that guides practitioners through rapid-feedback cycles in order to generate actionable data to improve implementation of ongoing programs. The framework was designed while implementing DELTA PREP, a 3-year project aimed at building the primary prevention capacities of statewide domestic violence coalitions. The authors describe the framework’s main steps and provide a case example of a rapid-feedback cycle and several examples of rapid-feedback memos produced during the project period. The authors also discuss implications for health education evaluation and practice.

Keywords

developmental evaluation evaluation process evaluation quality improvement utilization evaluation violent behavior and prevention

While translation of research to health education practice has been widely discussed (Glasgow, Lichtenstein, & Marcus, 2002; Thigpen, Puddy, Singer, & Hall, 2013; Wandersman et al., 2008; Wilson, Brady, & Lesesne, 2011), health education still lacks a process model that explains how to use data generated from practice to improve practice. Hence, even highly motivated practitioners who embrace program improvement and collect evaluation data may be stymied when trying to use data in order to make ongoing adjustments during program implementation. Quality improvement holds promise for developing a process model that explains how public health programs can use implementation data for ongoing improvement and documenting progress toward program goals. Health care organizations and public health agencies rely heavily on quality improvement methods for delivering effective clinical services (Institute of Medicine, 2001; Bialek, Duffy, & Moran, 2009; Livingood et al., 2013; Dilley, Bekemeier, & Harris, 2012). Some efforts have been made to institute quality improvement processes in public health systems (Beaudry, Bialek, & Moran, 2014; Davis et al., 2012; McLees et al., 2014; Verma & Moran, 2014), but there is little evidence that quality improvement is a standard part of health education program implementation (Riley et al., 2010). Although health education programs may benefit from quality improvement methods (Woodhouse et al., 2013), scant resources exist to guide practitioners on applying these methods for program improvement.

The purpose of this article is to describe the Data-to-Action framework, a process to guide evaluators and practitioners in using rapid-feedback cycles to generate actionable data for improving implementation of ongoing multicomponent programs. The framework was designed while implementing DELTA PREP (Preparing and Raising Expectations for Prevention), a 3-year project aimed at building the primary prevention capacities of statewide domestic violence (DV) coalitions. The project’s summative and process evaluations are reported elsewhere (Freire, Zakocs, Le, Hill, Brown, & Wheaton, 2015; Schober & Fawcett, 2015; Zakocs & Freire, 2015).

We first provide background on DELTA PREP and then describe key elements of the Data-to-Action framework. Next, we describe how the framework was implemented within DELTA PREP and resulting actions. Last, we share insights learned about using the framework for improving implementation of DELTA PREP.

Background on DELTA PREP

DELTA PREP was a 4-year initiative intended to improve the organizational capacity of statewide DV coalitions in order to prevent intimate partner violence (IPV) before it begins (i.e., primary prevention). When the project started in 2008, many of the 19 state DV coalitions that participated in the project were mostly focused on responding to IPV, specifically through survivor services and advocacy. DELTA PREP was designed to facilitate coalitions’ integration of prevention within their existing organizational structures and functions, and combining prevention activities with regular coalition operations when possible. The project offered eight supports (Table 1) intended to accelerate coalitions’ building their prevention capacities and, in turn, facilitate them serving as catalysts for IPV prevention in their states. Project staff and consultants offered a mix of traditional training events, technical assistance (TA), and peer-to-peer coaching, as well as tools for capacity assessment, action planning, and documenting organizational changes and prevention activities. Figure 1 displays the project’s theory of change.

Table 1.

DELTA PREP Supports Offered to Coalitions.

Support element	Delivery
Training events	10 on-site training events were held with a total 440 participants attending that included 269 DELTA PREP representatives. Five webinars were held with a total of 60 participating.
Technical assistance	Over 320 proactive, technical assistance events were delivered via national conference calls, individual telephone calls to grantee coalitions, and written feedback on grantee documents.
Coaching hubs	All five coaching hubs were formed and continued to meet throughout the project period. During Year 1, almost all coaching hubs met monthly as requested by the Centers for Disease Control and Prevention (91%) with a total of 50 meetings held across the five hubs. On average, eight DELTA PREP and DELTA representatives attended monthly meetings per hub. A majority (72%) of meeting logs were posted on the workstation. In Year 3, three national coaching telephone conferences were held.
Action planning tools	All 19 coalitions received a workbook and guidance (via trainings and TA) for developing actions plans. All 19 coalitions drafted action plans that included organizational and prevention actions.
Coalition Prevention Capacity Assessment	All 19 coalitions completed the Coalition Prevention Capacity Assessment in Years 1 and 3. All 19 received customized reports for both rounds of assessment administration; reports were reviewed at training events.
Online workstation	158 DELTA PREP representatives, DELTA coaches, project staff, and consultants were registered to the workstation. Approximately 430 visits (on average) were made to the workstation monthly. On average, 27 DELTA PREP representatives or DELTA coaches visited this web site monthly, which equated to about 18% of registered DELTA PREP and DELTA coaches. Most posts to the workstation, such as announcements (93%) and shared documents (82%), were made by project staff rather than by coalition representatives.
Online documentation system	984 entries were made to the online documentation system during the 3-year project period with an average of 52 entries per coalition.

Figure 1.

DELTA PREP theory of change.

The project period included a 1-year planning phase, after which 19 state DV coalitions received funding to participate in DELTA PREP’s 3-year implementation phase. DELTA PREP was implemented by the Division of Violence Prevention at the Centers for Disease Control and Prevention (CDC) in partnership with the CDC Foundation and the Robert Wood Johnson Foundation.

Data-to-Action Overview

The Data-to-Action Framework was developed and implemented by members of the DELTA PREP project team who cowrote this article: project directors (KF and PB), project coordinator (JAH), project officer (JW), and evaluation consultant (RZ). Although developed for DELTA PREP, the framework outlines a generic process to guide practitioners in using rapid-feedback cycles to generate actionable data for improving implementation of ongoing programs.

Three improvement-oriented evaluation approaches informed the framework’s development: quality improvement (Bauer, Duffy, & Westcott, 2006), utilization-focused evaluation (Patton, 2008), and developmental evaluation (Patton, 2011). Key elements of the framework include the following: a team-based approach, ongoing adaptations for more nascent programs, generation of actionable data, and use of rapid-feedback cycles (Hargreaves, 2014; McNall, Welch, Ruh, Mildner, & Soto, 2004).

Team-Based Approach

Data-to-Action was designed around a collaborative, team-based approach where both programmatic staff and evaluators work together as partners (O’Sullivan, 2004). In this approach, the evaluator is considered a member of the project team who is helping improve the program (Patton, 2011) and may play several roles including a facilitator and reporter. As a facilitator, the evaluator helps programmatic staff to: identify questions they want answered and data that will be credible to them, engage in reflective “reality testing” by comparing how they believe the intervention is being implemented to what is actually happening on the ground, and generate possible actions that may be taken to improve implementation. As a reporter, the evaluator may be charged with collecting, synthesizing, and reporting on information about how the intervention is being implemented.

Ongoing Adjustment for Programs in Earlier Stages of Development

Programs typically follow a life cycle, spanning from earlier to later stages of development (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005). From an evaluation lens, more nascent programs are typically subjected to formative evaluation, while more mature programs are subjected to summative evaluation to render judgments of merit and worth and designated as evidence-based models for those deemed effective. Data-to-Action was designed for programs earlier in their stages of development so that adjustments can be made while the program is still being developed, refined, and pilot-tested. It may be viewed as a formative evaluation tool for uncovering an intervention’s strengths and weakness, assessing how well participants are receiving the programs, mapping progress toward desired outcomes, identifying implementation barriers, or describing how environmental conditions are affecting the intervention.

Actionable Data for Intended Users

We define actionable data as information that helps intended users make programmatic decisions. Intended users may be more likely to act upon data when (1) data answer the questions to which they want answers; (2) data are credible to them; (3) data are reported in a concise, understandable manner; (4) data are delivered before decisions need to be made; and (5) stakeholders make time to reflect on findings, implications, and possible actions (Patton, 2008).

Rapid-Feedback Cycles

Rapid-feedback cycles comprise the core of Data-to-Action, where data are continually collected, analyzed and used to inform action within a short time period, usually a few weeks to a few months. These cycles were informed by the Plan-Do-Check-Act cycles, a well-known quality improvement process (Tague, 2005). Rapid-feedback cycles continually produce information about how the intervention is being implemented over time, so that program staff are better able to take action while the intervention is being implemented. Five steps were used to carry out each rapid-feedback cycle: (1) clarify intent, (2) collect “good enough” data, (3) produce a brief memo, (4) facilitate a reflective debrief, and (e) make decisions (see Table 2 for description of each step).

Table 2.

Five Steps for Implementing a Rapid-Feedback Cycle.

Steps	Considerations
1. Clarify intent	Draft a 1- to 2-page protocol that explains purpose; identifies intended users; lists questions to which intended users want answers; describes low-cost, data collection strategies; and defines who will do what in a timely manner.
2. Collect “good enough” data	Collect and analyze data quickly. Results should be certain enough for intended users to make decisions about adjustments to the project. Data collected through existing programmatic channels are preferred, rather than orchestrating new venues for evaluation purposes.
3. Produce brief memo	Draft a concise memo that highlights the major findings only (leave out the details) and visually appealing (color, pictures, and graphics). Distribute the memo soon after data have been collected.
4. Engage in reflective debrief	Engage in a reflective discussion with the project team about findings from the memo. Center discussion around three questions:
	• What are we learning? (What?)
	• What are the implications for the project? (So what?)
	• What actions are required? (What now?)
5. Makes decisions	Informed by discussion, make a decision to
	• gather more information before taking actions,
	• take no action, or
	• take action: make adjustments to intervention.

Data-to-Action Implementation

During the 3-year DELTA PREP period, the project team conducted 20 rapid-feedback cycles. Most cycles focused on supports offered to coalitions to assess coalitions’ satisfaction with project supports and how they used supports to advance their prevention capacity building. Approximately half (55%) of the memo’s topics were identified prior to project implementation, while the other half emerged in response to challenges and opportunities during the project implementation. The evaluation consultant (RZ) drafted 50% of the memos, while project staff and other consultants drafted the others. Almost all (90%) memos were discussed during project team meetings.

Table 3 shows five examples of rapid-feedback cycles the project team implemented during DELTA PREP. As an illustrative example, we highlight a 6-month rapid-feedback cycle focused on the coaching hubs (Example 1 in Table 3). The project team established five, region-based coaching hubs where three to four DELTA PREP coalitions (project grantees) were paired with staff from DELTA coalitions who had received CDC funding to build prevention capacity for the previous 6 years. At the project’s onset, the DELTA coalitions were assumed to be experienced in integrating primary prevention into their coalitions and, thus, they were requested to serve as “coaches” to their sister DELTA PREP coalitions. Coaching hubs connected monthly via conference call. Coaches were expected to facilitate the calls, offer resources, and post monthly meeting notes to the online workstation. The project team engaged in the Data-to-Action five steps to assess and improve the coaching hubs.

Step 1. Clarify intent: The project team and DELTA coaches were identified as the primary users for coaching hub rapid-feedback cycle. Three questions were posed: What is the coaching process being used? What are the benefits and challenges to DELTA coaches and DELTA PREP coalitions when engaging with the coaching hubs? What are participants’ suggestions for improvement? A protocol clarifying intended users, questions, data collection strategies, and time line was drafted in April 2010.

Step 2. Collect “good enough” data: The team collected and analyzed data from three sources over a 2-month period (May-June 2010). One team member conducted a content analysis of 50 coaching logs posted on the online workstation that captured the essence of monthly coaching meetings. Next, the evaluation consultant held two focus groups with 13 DELTA coaches from each of the five hubs during an annual project meeting. Last, the project added questions about the coaching hubs to an existing posttraining survey completed by 35 representatives from 19 DELTA PREP coalitions.

Step 3. Produce a brief memo: The evaluation consultant analyzed data across the three data sources described in Step 2 to provide succinct answers to questions posted by the project team in Step 1. The findings were reported in a seven-page memo written for the project team. The memo was distributed to the project team 2 weeks after data collection was completed (June 2010). The memo included tables, diagrams, and quotes to describe how the coaching process was being implemented; benefits and challenges experienced by the coaches (DELTA coalitions) and mentees (DELTA PREP coalitions); and recommended changes by coaches and mentees. In sum, the team learned the coaching hub model was not being implemented as originally envisioned.

Step 4. Facilitate reflective debrief: Within a week after the memo was distributed (June 2010), the project team reflected on the findings, implications, and potential actions. The project team decided to draft two tailored memos and facilitate two separate debriefings with the coaches (DELTA coalitions) and mentees (DELTA PREP coalitions) to share what was learned about coaching hubs and to collectively generate solutions for problem areas. The team believed that separate briefings would allow each group to discuss coaching form the perspective of their defined roles within the coaching hubs.

Step 5. Make decisions: As a result of the debriefings, the project team decided to make several adjustments to the coaching hubs (September 2010). DELTA PREP Project staff would participate in monthly DELTA calls to improve communication between the DELTA PREP and DELTA projects. Second, DELTA coaches would plan and facilitate quarterly “national calls” on topics generated by DELTA PREP grantees and DELTA coaches in lieu of the monthly coaching calls. Third, executive directors and other coalition leaders would not be required to participate in monthly calls but would be encouraged to find other ways to connect with each other. Lastly, DELTA PREP staff would be available to join coaching calls as needed to clarify expectations.

Table 3.

Examples of Rapid-Feedback Cycle Memos by DELTA PREP’s Theory of Change Components.

Theory of change components	Rapid-feedback cycle steps
	1. Clarify Intent	2. Collect “good enough” data	3. Draft memo	4. Engage in reflective debrief^a	5. Make decisions
DELTA PREP supports
Coaching hubs	Assess coalitions participation in coaching activities and solicit feedback on benefits and challenges	Reviewed 50 coaching logs posted on the online workstation, held 5 focus groups with coaches for each of the hubs, added questions to a posttraining online survey to DELTA PREP coalition members	A 7-page memo was shared with project staff, and then a 2-page memo was developed for coalitions.	Participants reported some benefits with coaching such as learning from each other, having a sounding board, and knowing someone is there to help.	Staff decided to maintain coaching calls and add sharing and networking opportunities during onsite training.
				More often participants reported early challenges with coaching calls, such as lack of agenda, unclear coaching roles, and difficulty with phone format.
Online workstation	Examine the extent to which project participants use the workstation and factors that influence use	Reviewed workstation usage reports and counted number of posted documents over 3 months and examined use on days before and after staff sent messages to promote use, interviewed users about challenges and uses, repeated data collection 6 months later to assess changes in use	Findings for first data collection were included in a memo on multiple project components; second memo focused only on workstation.	Participants were not using the workstation discussion function or to share documents as intended; however, they were using the calendar and project update functions. Coalitions used other online sites and communication channels to connect with peers.	Project staff continued to post project updates and events on the workstation; staff ended weekly promotional efforts and used coalitions’ preferred communication channels (e.g., listserve) to promote sharing.
Online documentation support system (ODSS)	Assess coalitions’ use of the ODSS and perceived benefits and challenges	Interviewed staff from 9 of 19 coalitions during an onsite meeting; reviewed documented changes	5-page memo	All coalitions had documented changes in the ODSS. Some participants reported the greatest benefit was reviewing other coalitions’ organizational changes to get ideas, and many viewed the main use was for the funder not coalitions.	Project staff determined that current technical assistance and guidance had resulted in expected documentation. Staff discussed that organizational change data may not be as useful to coalitions and their partners as community-level data, but no specific actions were taken to increase data use.
Coalition change process
Proposed organizational changes in action plans	Examine coalitions’ proposed organizational changes and identify challenges with action panning process for future technical assistance	Reviewed and coded coalitions’ action plans, which included questions about coalitions’ experiences with the planning process	Developed 2-page memo and a table of organizational changes that listed coalitions for each proposed change	All coalitions (N = 19) submitted plans. About half of proposed organizational changes met the project definition. Some proposed changes were included in 10 or more plans.	Staff provided additional guidance on organizational change criteria for coalitions’ documentation of changes and posted table of proposed changes to the workstation to connect coalitions doing similar work They posted table of proposed changes on the workstation and encouraged coalitions to identify other coalitions working on similar changes during a national call.
				Coalitions proposed 56 types of organizational changes within six defined coalition areas (e.g., leadership, staffing, resources). Most changes (60%) were listed in the inventory provided to coalitions during the action planning training.
Prevention capacity goals
Prevention integration	Identify coalitions that have started to integrate prevention through organizational changes and identify factors that have contributed to coalitions integrating prevention	Interviewed participants from coalitions and reviewed their action plans	Drafted 2-page memo that synthesized eight key factors that contributed to coalitions’ prevention integration	Factors included characteristics of staff and leaders who championed prevention, as well as processes to engage all staff in prevention activities.	Findings were used to identify five case study coalitions

Presented findings are examples and are not necessarily complete findings from rapid-feedback memos.

Data-to-Action Findings

As intended, implementing rapid-feedback cycles led to specific project adjustments to improve the delivery of project supports and increase coalitions’ use of supports to integrate IPV prevention in their organizational structures and processes. The project team also used rapid-feedback cycles to track progress toward project goals and to communicate with project stakeholders about project implementation and outcomes.

Making Adjustments

Across all rapid-feedback cycles, three types of decisions were made: (1) take no action, (2) delay decision (i.e., gather more information before making a decision), or (3) make an adjustment to a project support to improve its use and usability. As expected, most adjustments were made to supports offered to grantee coalitions, such as restructuring in-person training events, altering the format of TA calls, offering additional supports not originally envisioned, clarifying expectations by drafting or revising existing guidance documents, reallocating resources, and revamping elements of the coaching hub model.

Almost all adjustments to supports were no- or low-cost changes that did not require additional project funds but did require additional staff time and flexibility in the project workload and priorities. In fact, project team time spent on implementing rapid-feedback cycles and making adjustments was the largest project “cost,” and a major factor that influenced whether or not the project team could make specific adjustment. For example, in one rapid-feedback cycle on TA provided by project staff, we found that DELTA PREP coalitions and project staff had different views of the purpose and focus of TA calls. Coalitions often perceived that TA calls were exclusively focused on technical monitoring for the grant and documentation of organizational changes listed in their action plans. In contrast, project team members believed the TA calls were mostly focused on consulting with coalitions on their action plan implementation and problem solving about challenges coalitions faced, with a little time spent on grant monitoring. Project staff determined that their discussions with coalitions about their prevention work was overshadowed by agenda items on documentation and progress reports. Therefore, staff added a quarterly TA call with each coalition to discuss how coalitions applied prevention concepts to their work and specific actions to integrate prevention within their organizations, without any discussion of grant monitoring.

In rarer circumstances, the project team learned that their time spent on certain activities did not result in increased coalition participation or satisfaction, and the team decided to reduce the amount of time spent on these activities. For example, one rapid-feedback cycle on coalitions’ online workstation use found that coalitions really liked the creative e-mails and activities they received from the project coordinator every Wednesday to drive participants to the workstation. The coordinator designed various puzzles and games that sometimes included clues to items on the workstation. However, workstation use did not actually increase in the days after staff sent e-mails. The project team decided that the 3 to 6 hours the project coordinator spent each week on “Workstation Wednesdays” should be shifted to other priorities.

Communicating With Stakeholders

Besides making programmatic adjustments, information generated from the rapid feedback memos were used to communicate with stakeholders. Findings from memos were used to inform grantee coalitions, CDC division leadership, and the funder throughout the project. Insights from the memos were summarized in meetings, or memos were directly shared. Approximately, 50% of memos were distributed to stakeholders beyond the project team. Memos were used to shape conversations with grantee coalitions, quickly respond to concerns raised, encourage cross-sharing and networking, and explain programmatic decisions. Memos also were used to answer specific requests from CDC division leadership, justify additional requests for the program, and make a case for changes to similar projects. Last, the project team used memos to complete funder progress reports and justify requests for budget adjustments and cost extensions.

Tracking Progress Toward Goals

Information generated from rapid feedback memos was also used to inform DELTA PREP’s summative and process evaluations (Freire et al., 2015; Zakocs & Freire, 2015). Some examples included revising evaluation questions to reflect new program priorities, gaining conceptual clarity about the desired capacity outcomes, informing the case study protocol, and designing the 6-month postproject interview guides. In some cases, data included in memos were used as proxy indicators for progress. One memo summarized data entered by coalition grantees into an online documentation system, which demonstrated that at the end of the first program year, all grantees had completed one or more organizational changes to integrate prevention. This information was used as an indicator that the project was moving toward achieving its goal.

Implications for Practice

The Data-to-Action framework was designed to provide actionable data for the project team to make better informed decisions for improving DELTA PREP implementation. We developed the framework to guide our efforts to improve DELTA PREP as it was being implemented over a 3-year period. The framework described here resulted from our initial rapid-feedback cycle efforts, refinements to those efforts over time, and reflection on how we could have improved the process by laying a more solid foundation before we began. Throughout our process, we discovered additional uses for the framework beyond making implementation improvements, specifically communicating with stakeholders and improving other evaluation elements.

Although our primary aim was to generate actionable data that could lead to implementation improvements, we found the process generated three other benefits: a culture of learning, a historical record, and increased buy-in from grantees. The process encouraged our team to adopt a culture of learning, where we actively sought to “test reality” about how the project was being implemented through data and then commit to reflecting and acting on what we were learning. By making the time and space to learn, the team was better able to conceptualize the project. We discovered that carrying out the rapid-feedback cycles in the first year helped us clarify and refine operational definitions of outcomes and the theory of change, which in turn allowed us to communicate more effectively to grantees and inform our summative and process evaluations.

The process also enabled us to document a historical record of the project. During the project period, the rapid feedback memos served as a collective memory of how the project was implemented. Project staff would periodically review the memos when asked for specific information about the project or when completing progress reports to the funder. The memos also aided in orienting new staff that joined the project after it began. Since the conclusion of the project, staff have revisited the memos when revising new requests for proposals or sharing ideas for others who are implementing similar capacity-building projects.

Last, the framework may have increased buy-in from grantees. As with many initiatives, grantees were not always 100% satisfied with project elements. Many rapid-feedback cycles included grantee representatives we interviewed to gauge their experiences with the supports being offered—coaching, trainings, and online documentation systems. In many cases, staff made adjustments to how these supports were offered and credited these changes to the rapid-feedback cycles. Knowing that their concerns were being addressed may have encouraged grantees to become more invested with the project.

Although the DELTA PREP team benefited from using the framework, we also experienced challenges during the process. A key challenge was determining the best actions to take based on data, even though that was the primary intent of the framework. One improvement we made during DELTA PREP was to specify a priori the top intended users and uses before any data were collected. This helped us prioritize actions in subsequent cycles. However, when we have presented the framework to staff from other programs, we find a common question is, “How do you decide when there is enough evidence to make a change?” It is an important question that requires more specific criteria or guideposts than the framework currently offers. A next step is to identify general principles for action. Some examples are when an action would remove a common barrier to implementation, increase flexibility while maintaining the program intent, and improve how the program element is received by participants.

Another challenge was balancing project resources. At times, we struggled with balancing staff time devoted to implementing the project versus carrying out the Data-to-Action processes. Similarly, evaluators had to balance limited resources for all three project evaluations—summative, process, and formative. The rapid-feedback cycles do require an ability to collect data and quickly synthesize findings to move to action. To reduce burden for the project team, we tried to find times during planned trainings for focus groups and interviews to gather our “good enough” data. And at times we decided to forgo our preferred formal memo with colorful graphics for a “quicker fix,” for example, the prefab report generated from an online survey system. In the end, we considered the time and labor investment during the project to greatly pay off after the project ended because we had documented the rich experience of project staff and participants and reflected on each project element by the time we were ready to tell the story of DELTA PREP and its outcomes. As program teams weigh the costs and benefits of using rapid-feedback cycles, they should consider the long-term value of information they might collect.

In sum, the lessons learned from using this framework during DELTA PREP can be instructive for both evaluators and practitioners. The framework offers opportunities for participatory evaluation as well as program improvement. It also can facilitate the use of data in ways that are relevant to the goals of project implementers as well as project participants.

Footnotes

Theme Section Note

This article is part of a Health Education & Behavior theme section on the DELTA PREP Project, sponsored by the Centers for Disease Control and Prevention (CDC) National Center for Injury Prevention and Control, Division of Violence Prevention. Funding for DELTA PREP and the issue was provided by the CDC Foundation through a grant from the Robert Wood Johnson Foundation. The findings and conclusions in this report are those of the author(s) and do not necessarily represent the official position of the Centers for Disease Control and Prevention, the CDC Foundation, or the Robert Wood Johnson Foundation. The theme section articles are available open access via .

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

References

Bauer

J. E.

Duffy

G. L.

Westcott

R. T.

(2006). The quality improvement handbook (2nd ed.). Milwaukee, WI: ASQ Press.

Beaudry

M. L.

Bialek

Moran

(2014). Using quality improvement tools and methods throughout the accreditation lifecycle. Journal of Public Health Practice, 20, 49-51.

Bialek

Duffy

G. L.

Moran

J. W.

(2009). The public health quality improvement handbook. Milwaukee, WI: ASQ Press.

Davis

M. V.

Vincus

Eggers

Mahanna

Riley

Joly

. . . Bowling

(2012). Effectiveness of public health quality improvement training approaches: Application, application, application. Journal of Public Health Practice, 18, E1-E7.

Dilley

J. A.

Bekemeier

Harris

J. R.

(2012). Quality improvement interventions in public health systems: A systematic review. American Journal of Preventive Medicine, 42(5 Suppl. 1), S58-S71.

Fixsen

D. L.

Naoom

S. F.

Blase

K. A.

Friedman

R. M.

Wallace

(2005). Implementation research: A synthesis of the literature. Tampa: National Implementation Research Network, University of South Florida.

Freire

K. E.

Zakocs

Hill

J. A.

Brown

Wheaton

(2015). Evaluation of DELTA PREP: A project aimed at integrating primary prevention of intimate partner violence within state domestic violence coalitions. Health Education & Behavior, 42(4), 436-448.

Glasgow

R. E.

Lichtenstein

Marcus

A. C.

(2002). Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. American Journal of Public Health, 93, 1261-1267.

Hargreaves

M. B.

(2014). Rapid evaluation methods from complex initiatives (HHS Office of the Secretary for Planning and Evaluation issue brief). Retrieved from http://aspe.hhs.gov/sp/reports/2014/EvalApproach/rs_EvalApproach.cfm

10.

Institute of Medicine. (2001). Crossing the quality chasm: A new health system for the 21st century. Washington, DC: National Academies Press.

11.

Livingood

W. C.

Sabbagh

Spitzfaden

Hicks

Wells

Puigdomenech

. . . Wood

D. L.

(2013). Quality improvement evaluation case study: Impact on public health outcomes and agency culture. American Journal of Preventive Medicine, 44, 445-452.

12.

McLees

A. W.

Thomas

C. W.

Nawaz

Young

A. C.

Rider

Davis

(2014). Advances in public health accreditation readiness and quality improvement: Evaluation findings from the national public health improvement initiative. Journal of Public Health Practice, 20, 29-35.

13.

McNall

Welch

V. E.

Ruh

K. L.

Mildner

C. A.

Soto

(2004). The use of rapid-feedback evaluation methods to improve the retention rates of an HIV/AIDS healthcare intervention. Evaluation and Program Planning, 27, 287-294.

14.

O’Sullivan

R. G.

(2004). Practicing evaluation: A collaborative approach. Thousand Oaks, CA: Sage.

15.

Patton

M. Q.

(2008). Utilization-focused evaluation (4th ed.). Thousand Oaks, CA: Sage.

16.

Patton

M. Q.

(2011). Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York, NY: Guildford Press.

17.

Riley

W. J.

Moran

J. W.

Corso

L. C.

Betsch

L. M.

Bialek

Cofsky

(2010). Defining quality improvement in public health. Journal of Public Health Management, 16(1), 5-7.

18.

Schober

Fawcett

(2015). Using action planning to build organizational capacity for the prevention of intimate partner violence. Health Education & Behavior, 42(4), 449-457.

19.

Tague

N. R.

(2005). The quality tool box (2nd ed.). Milwaukee, WI: ASQ Press.

20.

Thigpen

Puddy

R. W

Singer

H. H.

Hall

D. M.

(2013). Moving knowledge into action: developing the rapid synthesis and translation process within the interactive systems framework. American Journal of Community Psychology, 50, 285-294.

21.

Verma

Moran

J. W.

(2014). Sustaining a quality improvement culture in local health departments applying for accreditation. Journal of Public Health Practice, 20(10), 43-48.

22.

Wandersman

Duffy

Flaspohler

Noonan

Lubell

Stillman

. . . Saul

(2008). Bridging the gap between prevention research and practice: An interactive systems framework for building capacity to disseminate and implement innovations. American Journal of Community Psychology, 41, 171-181.

23.

Wilson

K. M.

Brady

T. J.

Lesesne

(2011). An organizing framework for translation in public health: The Knowledge to Action Framework. Prevention of Chronic Disease, 8, A46. Retrieved from http://www.cdc.gov/pcd/issues/2011/mar/10_0012.htm

24.

Woodhouse

L. D.

Toal

Nguyen

Keene

Gunn

Kellum

. . . Livingood

W. C.

(2013). A merged model of quality improvement and evaluation: Maximizing return on investment. Health Promotion Practice, 14, 885-892.

25.

Zakocs

Freire

K. E.

(2015). The DELTA PREP initiative: Accelerating coalition capacity for intimate partner violence prevention. Health Education & Behavior, 42(4), 458-470.