Scaling Up Implementation Support for Violence Prevention and Resilience Promotion in the Air Force

Abstract

Closing the gap between research and practice requires that organizations can consistently incorporate new ideas and best practices. The Department of the Air Force (DAF) Integrated Resilience Directorate is leading a coordinated effort to increase the adoption of evidence-based violence prevention and resilience promotion programs across the entire Department. To support this effort, DAF is using Getting To Outcomes (GTO), an evidence-based implementation support that helps organizations plan, implement, and self-evaluate programs. Thus, the DAF is not only scaling up prevention programs but is also embarking on the largest scale-up of GTO to date. The study team trained personnel from every Air Force installation across the world to use GTO for their programs. Quantitative findings from training questionnaires and ratings of implementation plans as well as qualitative results from resilience personnel interviews suggest some lessons learned for scaling up implementation support. This study builds on established implementation science frameworks for scaling up interventions by identifying critical tasks and unique supports needed to scale up evidence-based prevention. Results suggest GTO helped DAF scale-up prevention across the Department, and that establishing leadership buy-in, simplifying evidence-based program selection and adaptation, monitoring implementation and outcomes, and creating dedicated prevention practitioner roles are critical tasks to support scale-up of evidence-based prevention. Unique supports needed to scale-up evidence-based prevention include multitiered learning systems; integrated tools that improve access to data and evolving evidence; prevention personnel with skills in program management, adaptation, and evaluation; timely crosscutting data; continuous learning to support sustainability; and leadership buy-in.

Keywords

evidence-based programs implementation facilitation program evaluation

Closing the gap between research and practice requires that public health practitioners and organizations consistently incorporate new ideas and best practices into their work. A coordinated effort to increase the adoption of evidence-based practices across a large enterprise is being led by the Air Force Integrated Resilience Directorate, which is tasked with mitigating “all instances of interpersonal and self-directed violence within the Department of the Air Force (DAF), while promoting resilience, connectedness and respect for all” (DAF, 2022). To comply, every 2 years, staff from every U.S. Air Force installation are required to convene a team of representatives from their installation’s helping agencies (the Community Action Team) and create a Community Action Plan (CAP). However, staff very seldom have the capabilities needed to effectively plan and implement evidence-based programs (EBPs).

Historically, CAPs have not relied on EBPs or included evaluation plans. A recent review of U.S. Air Force sexual assault prevention programs found progress in implementing some best practices for effective prevention (e.g., tailoring prevention to the context) but noted overall low rates of compliance with others (i.e., conducting prevention with adequately trained personnel, utilizing outcome evaluations; Gedney et al., 2018). This was partly because there were no standard guidelines to ensure CAP quality or implementation success. Often, staff relied on interventions that were homegrown, heavily modified, or were not trained in prevention activities like needs assessment, goal setting, assessing and choosing EBPs, planning, outcome evaluation design, and quality improvement. As might be expected, this lack of capability, structure, planning, and accountability meant that CAPs typically did not demonstrate significant impacts (DAF, 2018). A coordinated system of implementation support to was needed to increase the ability of staff to incorporate EBPs in the installation CAPs and improve the quality of violence prevention and resilience promotion efforts across DAF. This article describes the incorporation of implementation support in the CAP development process and lessons learned from its scale-up across DAF’s 93 worldwide installations.

Implementation Support Strategies and Scale-Up

Implementation support strategies have been shown to improve the adoption of EBPs across multiple sites. Getting To Outcomes® (GTO®) is a 10-step implementation strategy bundle, which combines the Expert Recommendations for Implementing Change strategies of train and educate stakeholders, provide technical assistance, multiple evaluative and iterative strategies, adapt and tailor to the context, and support clinicians/practitioners (Chinman et al., 2008). GTO has been described extensively elsewhere (Chinman et al., 2008, 2013) Briefly, more than 20 years of research has shown that organizations that use GTO build their implementation and evaluation skills, run EBPs with better fidelity, and are more likely to achieve their intended outcomes. Much of this success depends on the training provided by GTO staff as well as materials in the community and program-specific GTO manuals (Chinman et al., 2016, 2018). In these studies, GTO improved the uptake and implementation of a single or limited number of EBPs in a single delivery system (i.e., Boys and Girls Clubs; Chinman et al., 2008, 2009, 2013) In contrast, Air Force installations not only need a wide range of new EBPs but must also tailor them to a military setting. DAF selected GTO as an implementation support process to guide the 2018 CAP development process across all DAF installations for numerous programs.

Framework for Going to Full Scale

Existing frameworks that support spread or scale-up provide useful constructs for examining the roll out of GTO in the Air Force. For example, the Institute for Health care Improvement (IHI) Framework for Going to Full Scale (Barker et al., 2015) draws on prior scale up and spread frameworks and describes the support system elements needed for successful scale-up: (1) a learning system that connects adopters and experts; (2) a data system to support measurement for improvement; (3) infrastructure (i.e., tools, communication systems); (4) training and support to build capability; and (5) a reliable process that supports sustainability (Hunter et al., 2015). GTO is an implementation support that aligns with these IHI scale-up infrastructure elements (Table 1).

Table 1.

GTO 10 Steps for Community Action Planning.

Getting To Outcomes’ 10 steps		Framework for going to full scale
10 Steps	Application to Department of the Air Force Community Action Plans	Alignment with support system infrastructure
1. Provides information about conducting a needs assessment	Gathering data available at an installation that specifies the nature of the problem to be addressed and documenting any resources that are currently being used to address the problem. This could involve collecting existing data or new data (e.g. interviews, focus groups).	Data systems
2. Has worksheets for creating measurable goals and objectives	Establishing goals and concrete desired outcomes to address the needs identified in Step 1. This includes running the goals and outcomes by local leaders for their input.	Learning systems
3. Overviews evidence-based programming	Searching online databases, the internet, and written documents to find programs that could be used to achieve the goals set in Step 2. This could also involve speaking with the developers on identified programs to learn more about them. Often this step results in multiple program options.	Infrastructure for scale-up
4. Ensures program fit with the host agency/other programs	Examining how well programs chosen in Step 3 fit with your specific installation to determine which program should be modified or rejected. This would likely require talking to local leaders to assess the possibility of running the program.	Infrastructure for scale-up
5. Ensures sufficient capacity to conduct the program	Talking to various leaders and stakeholders to assess whether the installation has all the necessary support, expertise, and resources needed to implement a chosen program. Like Step 4, this step can lead an installation to ultimately reject programs identified in Step 3.	Infrastructure and Human capability for scale-up
6. Presents information and tools to plan activities	Making a final decision about which program to implement. This step involves specifying the activities of the program, where they will take place, who will do them, and when they will be done. It also involves specifying a plan for evaluating the program, which could involve searching for ways to measure progress.	Infrastructure and Human capability for scale-up
7. Provides information and tools to do process evaluation	Steps 7-10 to take place after CAP is approved and begins to be implemented and evaluated.	Data and Learning systems
8. Provides information and tools to do outcome evaluation		Data and Learning systems
9. Prompts practitioners to improve the program		Data and Learning systems
10. Helps sustain an effective program		Design for sustainability

Note. GTO = Getting To Outcomes; CAP = Community Action Plan.

This study provides lessons learned from the scale-up of GTO implementation support to assist DAF to adopt and implement evidence-based approaches across the Department. This effort represents the largest scale-up of GTO to date, encompassing multiple types of military populations (active duty, families, civilian employees) and multiple delivery systems (e.g., equal opportunity, sexual assault prevention, alcohol, and drug prevention).

Method

GTO coaches assisted installations with identifying their community needs and priority problems, primarily using data from the Community Feedback Tool (Sims et al., 2019), a biannual self-reported needs assessment for all Airmen, their spouses, and Air Force civilian employees. Coaches helped installation staff set goals, select and adapt evidence-based or best practice programs, and plan for implementation and evaluation (GTO Steps 1–6). Completed CAPs consisted of a logic model (from Step 2) and implementation workplan and process and outcome evaluation plans (from Step 6). The goal was to help staff integrate these practices into routine operations, thereby improving the uptake, implementation, evaluation, and improvement of EBPs over time.

Materials

All participating DAF personnel had access to a customized GTO guide (Chinman et al., 2020) and four supplemental sets of GTO tools (Farris, 2020; Hardison et al., 2020; Shearer & Ebener, 2020; Troxel et al., 2020) addressing DAF-identified priority topics (suicide prevention, sexual harassment prevention, sleep health promotion, and stress management). Tools are worksheets that prompt the decision to complete GTO steps.

Training

A train-the-trainer approach was used to create a tiered implementation support structure. GTO coaches—doctoral and masters level experts in GTO—first trained the Major Command Community Support Program Managers (“MAJCOMs”) to provide coaching to installations in their command using the GTO process. Major Commands represent each division of DAF, organized by either function (e.g., special operations) or geographic area (e.g., Europe), and MAJCOMs from each division are responsible for providing support to installations for implementing resilience programming (N = 20). GTO coaches trained MAJCOMs through a 2-day in-person GTO training (May 2018).

GTO coaches then trained installation Community Action Team members charged with developing CAPs on the 10 steps of GTO, with MAJCOM assistance. Three representatives from each installation (N = 93) participated in two, 3-day in-person GTO trainings (“Summit 1” in June 2018 and “Summit 2” in August 2018). The summits were a mixture of large plenary sessions with all attendees (over 300) to introduce each GTO step, and 13 smaller sessions broken out by Major Command where they received individualized instruction from MAJCOMs and GTO coaches.

Coaching

Coaching support (i.e., technical assistance) was delivered over a 6-month period through the tiered structure. MAJCOMs took the lead in providing installation-specific support by holding calls with installations to problem solve and check-in on CAP progress and review the installation’s draft CAPs. GTO coaches checked in with MAJCOMs and helped troubleshoot any issues via biweekly support calls and ad hoc phone calls and emails.

Data Collection

Data sources consisted of evaluation questionnaires following GTO training, a quality review of CAPs submitted by individual installations, and interviews with MAJCOM and installation participants. This study was approved by the RAND Corporation’s Human Subjects Protection Committee (Study: 2018-0080, approved 10/9/18). All participants provided verbal informed consent.

Training Evaluations

Following the MAJCOM training in May and summits in June and August 2018, participants completed anonymous questionnaires to gauge satisfaction with the training (quality, relevance, organization, opportunity for discussion, handouts, and time allotted) on a Likert-type scale from 1 (very dissatisfied) to 4 (very satisfied); their self-efficacy with the 10 steps of GTO on a Likert-type scale from 1 (could do the step with a great deal of help) to 3 (could do the step without any help); and their interest (yes/no) in receiving additional help with any of the GTO steps. At Summit 1, the training evaluation survey for MAJCOM participants also asked respondents whether they would need more GTO support than they could provide alone to their installations (yes/no). Summit 2 training evaluations went to both MAJCOM and installation participants and asked about the quality of their 2019 CAP “as it is today” and the readiness of their Community Action Team to submit their CAP by November 1, 2018, on a Likert-type scale from 1 (poor) to 4 (excellent). Response rates for the three evaluation surveys were 73% (n = 26 MAJCOMs), 95% (n = 252), and 84% (n = 233), respectively.¹

CAP Quality Review

Two GTO experts reviewed the 74 CAPs submitted between December 2018 and June 2019 and rated their quality as high, moderate, or low, based on four modified dimensions of the Plan Quality Index (Courie & Tate, 2020), a checklist of elements critical for high-quality plans that covers the existence of measurable goals, well-specified activities, and evaluation plans. GTO experts rated whether CAPs fully, partially, or not at all included: (1) all completed and detailed CAP components; (2) logically linkages between components (i.e., goals, activities, and evaluation measures all target the same domain); (3) desired outcomes that were specific, measurable, achievable, realistic, and specified a time by when they should be achieved; and (4) implementation plans that were detailed, specific, appropriate and feasible.

MAJCOM and Installation Interviews

We used maximum variation sampling (Cohen & Crabtree, 2006) to capture a range of perspectives from installations in different MAJCOMs and with CAPs of varying quality. We asked via email 42 Community Action Team members from at least one installation in each of the 13 Major Commands, to participate. Interviews were completed with 29 individuals from 18 installations representing 11 of the 13 Major Commands. Three declined to participate, and 10 did not respond to repeated inquiries. The final sample included nine installation representatives with incomplete CAPs, 12 with moderate quality CAPs, and eight with good quality CAPs. In addition, we interviewed two MAJCOMs that coaches perceived by the study team as more engaged, and two MAJCOMs perceived as less engaged. All installation and MAJCOM interviewees attended Summit 1 and all except two (installation representatives) attended Summit 2. A study team member not involved in coaching conducted the approximately hour-long semi-structured phone interviews. Near-verbatim responses were captured by a notetaker. Interviewees were asked about their impressions of GTO for community action planning; their involvement in using GTO to develop a CAP; benefits and challenges of using GTO; recommendations for improvements; and expectations about future use of GTO. Interviews with MAJCOMs also asked about their support to installations.

Data Analysis

Training Evaluations

For each of the three training evaluation questionnaires, means and standard deviations were calculated for Likert-type scale items, and percentages were calculated for items with yes/no response options (see Table 2).

Table 2.

MAJCOM GTO Training (3/30/18-4/1/18) Evaluation Results.

Questions		Mean*	SD
Satisfactions with quality of training components: 1 = very dissatisfied; 2 = somewhat dissatisfied; 3 = somewhat satisfied; 4 = very satisfied	1. Quality of the information you received	3.64	0.51
	2. Relevance of the information to your work on community action planning in the AF	3.50	0.71
	3. Organization of the training	3.27	0.78
	4. Opportunity for questions/ discussion	3.78	0.67
	5. Handouts or materials	3.69	0.68
	6. Time allocated to tasks during the training	3.38	0.80
Level at which you could support your installations: 1 = could support but would need a great deal of help; 2 = could support but would need some help; 3 = could support without any help	7. Examining current integrated resilience/violence prevention needs in your community	2.32	0.63
	8. Determining the availability of resources to meet the community needs	2.60	0.50
	9. Developing CAP goals and objectives to address the needs	2.32	0.56
	10. Locating, assessing and/or selecting evidence-based EBP	1.96	0.45
	11. Examining whether selected EBP fit with AF culture and population	2.22	0.54
	12. Evaluating readiness to implement selected EBP	2.00	0.50
	13. Developing a detailed implementation plan (e.g., staff roles, timeline, target population) for EBP	2.24	0.52
	14. Evaluate whether EBP are implemented in accordance with the plan	2.22	0.61
	15. Evaluate whether EBP produce the desired outcomes	2.08	0.57
Which of the GTO Steps would you like additional training on? 1 = Endorsed; 0 = Not endorsed	16. Needs/Resources	0.19	0.40
	17. Goal Setting	0.27	0.45
	18. Selecting EBP	0.35	0.49
	19. Fit	0.19	0.40
	20. Readiness to implement EBP	0.38	0.50
	21. Planning EBP	0.31	0.47
	22. Planning a Process Evaluation	0.46	0.51
	23. Planning an Outcome Evaluation	0.42	0.50

Note. *(N = 26, 73% response rate). AF = Air Force; MAJCOM = Major Command; CAP = Community Action Plan; EBP = Evidence-based program; GTO = Getting To Outcomes; SD = standard deviation.

CAP Quality Review

Both researchers rated all CAPs, came to consensus on ratings, and assigned each CAP to a category based on their summed total score: Low quality for missing or incomplete plans (score of 0); Moderate quality for plans that needed improvement (score of 1 to 5); and High quality for plans that appear ready or nearly ready for the next step of evaluation support (score of 6–8). Researchers also reviewed programs included in the CAPs to identify whether they were EBPs by checking whether programs were listed (and their status e.g., promising, proven) in the Clearinghouse for Military Family Readiness at Penn State (Karre et al., 2017), a registry of best practice programs to which installations were directed during training.

MAJCOM and Installation Interviews

Interview notes were qualitatively coded by a team of three coders using the constant comparative method (Charmaz, 2006). Notes were coded for topic (e.g., leadership support, training) and sentiment (positive, negative, or implementation suggestions). To increase trustworthiness, coders regularly reviewed and discussed code applications, illustrative quotes, and overarching themes. The four MAJCOM interviews were used to contextualize the installation responses.

Results

Training Evaluation

MAJCOM training evaluation questionnaires indicated they felt moderately prepared to assist their installations following training and would need additional support with planning evaluations (Table 2). Overall, installations and MAJCOMs were satisfied with almost all components of both Summit 1 and Summit 2 (see Tables 3 and 4). After Summit 1, participants wanted more support with selecting an evidence-based intervention (GTO Step 3), the planning process, and with outcome evaluations (GTO Step 6). Following Summit 2, installations wanted additional support in creating an implementation work plan. Overall, participants disliked the large plenary sessions that included more than 300 attendees, finding them overwhelming and confusing. They preferred breakouts by Major Command where they received individualized feedback from MAJCOMs and GTO coaches and suggested that future training only be provided in this format. After using GTO tools for CAPs at Summit 2, installations rated their CAPs as “satisfactory” (M = 2.28 out of 4, SD = 0.82). They indicated their readiness to submit a completed CAP by the deadline was also “satisfactory” (M = 2.39 out of 4, SD = 0.94).

Table 3.

Summit 1: Analysis of June 2018 Training Feedback Forms.

Questions asked on both installation and MAJCOM survey
Please rate your satisfaction with the^a. . . .	All respondents (N = 252)	MAJCOMs (n = 21)	Installations (n = 231)
Please rate your satisfaction with the^a. . . .	Mean (SD)
1. Organization of the breakout sessions	3.45 (0.74)	3.57 (0.81)	3.44 (0.73)
2. Opportunity for questions/ discussion	3.78 (0.49)	3.86 (0.65)	3.77 (0.47)
3. Handouts or materials	3.70 (0.57)	3.75 (0.72)	3.69 (0.56)
4. Time allocated to tasks during the training	3.25 (0.76)	3.57 (0.60)	3.23 (0.77)
Questions asked only on installation survey (n = 231)
Please rate your satisfaction with the^a. . . .			Mean (SD)
5. Quality of the information you received			3.57 (0.55)
6. Relevance of the information to your work on community action planning in AF			3.51 (0.59)
7. Organization of the training overall			3.31 (0.72)
8. Organization of the plenary sessions			2.98 (0.80)
9. Overall, how satisfied are you with the training?			3.37 (0.63)
10. Which of the GTO Steps would you like additional training on? (check all that apply)			Percent
Needs/ Resources			11%
Goal Setting			12%
Selecting EBP			48%
Fit			13%
Readiness to implement EBP			21%
Planning			23%
Planning a Process Evaluation			24%
Planning an Outcome Evaluation			26%
Questions asked only on MAJCOM survey (n = 20)
11. In what ways could this training be improved? (check all that apply)			%
More time to prepare before training			25%
More discussion among participants			15%
More information about a GTO Step^b			35%
Other, write-in comments: • Assessment of CAT prior to training—most CATs completely ineffective • Show a bad example and a good clear concept at the beginning so everyone can understand • More time [for Steps] 1–3 • More examples of completed step (maybe during each breakout) • More time between first and second training • Just more time to process info so folks don’t feel rushed overwhelmed • Expanded audience past just one CAT member • More solid expectations set by HAF –> MAJCOM –> installations to ensure CSC/VPI were ready to begin GTO process + MANDATORY attendance required by all base CSCs and VPIs.			40%
12. For each of the activities below, please choose the level at which your installations are likely to be for different steps of GTO.		Some could complete but may need help at a level that I can provide	Some will likely need more support than I can provide alone
Examining prevention and resilience needs in your community (Step 1)		25%	10%
Determining the availability of resources to meet the community needs (Step 2)		45%	10%
Developing CAP goals and objectives to address the needs (Step 2)		50%	10%
Locating, assessing and/or selecting evidence-based EBP (Step 3)		40%	30%
Examining whether selected EBP fit with AF culture and population (Step 4)		25%	15%
Evaluating readiness to implement selected EBP (Step 5)		40%	10%
Developing a detailed implementation plan (e.g., staff roles, timeline, target population) for programs (Step 6)		25%	15%
Evaluate whether programs are implemented as planned (Step 6)		37%	32%
Evaluate whether programs produce the desired outcomes (Step 6)		45%	20%

Note. MAJCOM = Major Command; GTO = Getting To Outcomes; EBP = Evidence-based program; CAP = Community Action Plan; CAT = Community Action Team; HAF = Headquarters Air Force; CSC = Community Support Coordinator; VPI = Violence Prevention Integrator; AF = Air Force.

Scale: 1 = Very Dissatisfied, 2 = Somewhat Dissatisfied, 3 = Somewhat Satisfied, 4 = Very Satisfied. ^bOne respondent indicated Step 2, one indicated Step 6, and the rest left blank.

Table 4.

Summit 2: Analysis of August 2018 Training Feedback Forms.

Please rate your satisfaction with each of the following aspects of the GTO training^a:	Mean (SD)
	All respondents(N = 233)	MAJCOMs(n = 9)	Installations(n = 197)	Unknown(n = 27)
1. Usefulness of the plenary sessions	2.67 (0.83)	3.44 (0.53)	2.67 (0.81)	2.40 (0.91)
2. Usefulness of the breakout sessions	3.50 (0.68)	4.00 (0.00)	3.51 (0.67)	3.19 (0.69)
3. Information provided to identify and select EBP	3.30 (0.67)	3.67 (0.50)	3.32 (0.65)	2.91 (0.79)
4. Information & tools provided to make strong plans	3.32 (0.67)	3.78 (0.44)	3.33 (0.66)	3.04 (0.81)
5. Information & tools for EBP readiness building	3.16 (0.78)	3.78 (0.44)	3.17 (0.78)	2.84 (0.69)
6. Information & tools provided to help with evaluation of EBP	3.18 (0.71)	3.63 (0.52)	3.20 (0.71)	2.84 (0.62)
7. Overall, how satisfied are you with the training?	3.19 (0.75)	3.78 (0.44)	3.21 (0.73)	2.81 (0.86)
Please rate the following^b:	All respondents	MAJCOMs	Installations	Unknown
8. The quality of your 2019 Community Action Plan as it is today	2.31 (0.82)	2.43 (0.53)	2.28 (0.82)	2.53 (0.83)
9. The readiness of your CAT to complete and submit your 2019 CAP by November 1	2.43 (0.94)	2.56 (1.01)	2.39 (0.94)	2.87 (0.83)
10. Which of the GTO Steps (or parts of steps) would you like additional support with? (check all that apply)
	All respondents	MAJCOMs	Installations	Unknown
Needs/ Resources	12%	0%	12%	11%
Goal Setting	12%	0%	13%	15%
Selecting EBP	23%	0%	24%	19%
Fit	9%	11%	8%	11%
Readiness to implement EBP	14%	0%	15%	11%
Planning	21%	11%	21%	19%
Planning a Process Evaluation	21%	11%	22%	15%
Planning an Outcome Evaluation	28%	22%	29%	19%
Other Area (doesn’t have to be a GTO step)	12%	11%	12%	15%
11. What type of additional support would you like (with the above GTO steps)?
	All Respondents	MAJCOMs	Installations	Unknown
Teleconferences	24%	22%	26%	11%
On-site Training	34%	56%	34%	33%
Webinars	23%	0%	26%	4%
Additional written guidance/handouts	19%	0%	23%	0%
Peer networking	29%	22%	30%	19%

Note. GTO = Getting To Outcomes; MAJCOMs = Major Commands; EBP = Evidence-based program; CAP = Community Action Plan; CAT = Community Action Team.

Scale: 1=Very Dissatisfied, 2=Somewhat Dissatisfied, 3=Somewhat Satisfied, 4=Very Satisfied. ^b Scale: 1 = Poor, 2 =Satisfactory, 3 =Good, 4 = Excellent.

CAP Quality Review

Among the 87 installations from which CAPs were expected, 74 (85%) had submitted a CAP by the June 27, 2019, deadline. Of these, 63% (n = 47) were rated high quality for strong alignment with the four dimensions of plan quality (i.e., scored 6 or higher). More than a third (36%, n = 27) were rated as moderate quality because some GTO tools were missing or because the plan lacked a logical overview, misstated desired outcomes, or had limited detail on implementation. Thirteen were missing (n = 5) or incomplete (n = 8; i.e., rated as Low quality). CAPs primarily focused on workplace issues like communication and leadership skills; personal/family relationships and work–life balance; Airmen community engagement; awareness of available services and resources; personal resilience; and health and fitness promotion. Many different approaches and interventions were planned. However, only a quarter of the interventions were listed in the Clearinghouse for Military Family Readiness’ repository of programs (Karre et al., 2017), and the majority were categorized as having “unclear” evidence.

Installation Interviews

GTO Process

Executed a Tiered-Learning System

Overall, installation respondents indicated that engaged MAJCOMs facilitated knowledge transfer and provided a helpful structure for installations to work together, through MAJCOM-wide working meetings and phone calls:

[MAJCOMs] were a step ahead of us and they were working with [the GTO coaches] so I felt comfortable that they were knowledgeable about our objectives.

A few respondents did have negative experiences with MAJCOMs who did not have the knowledge or capacity to support them, commenting: “I don’t think they [the MAJCOM] had a good understanding and it wasn’t a priority.”

Provided Helpful, but Lengthy, Structure

Most installation respondents liked that GTO tools provided better guidance, structure, and uniformity to the process than was present with previous CAPs:

When I first started, I didn’t get a whole lot of guidance on CAP. There wasn’t a lot of consistency on format or anything else. . . the [GTO] Guide gave us a guideline.²

Preferred GTO resources included evaluation guidance, resources to find EBPs, and example tools. Although installations and MAJCOMs said the tools took a long time to complete, several liked that they ensured important details would not be overlooked:

The tools are exhaustive. There’s a lot of considerations that would be overlooked without the detail that GTO goes through.

Respondents suggested the tools be streamlined and transitioned to an online format to allow for certain fields within the tools to be auto-populated across tools:

The hardest thing [about the GTO tools] was the format. It took so long to do it.

Provided a Sustainable Structure

When asked about future CAP development, the majority (n = 21) intended to use some or all of the GTO process. A few (n = 6) said they would use it if required to, and two indicated they did not plan to use GTO at all. Respondents indicated that continued support from MAJCOMs would be necessary to continue using GTO. MAJCOMs noted that despite initial difficulties, installations developed strong plans once they understood the process. They were optimistic about future CAPs using GTO:

If we went back [and did it again], our packages and plans would be even stronger than before. In two years’ we’d be a lot better the next round. (MAJCOM respondent)

Impact of GTO

Helped Build Skills and Knowledge

Interviews suggested that installations and MAJCOMs were learning and strengthening prevention skills aligned with the GTO steps including looking at trends in data to identify priority problems; setting concrete desired outcomes; considering the evidence base of a program before selecting one; creating detailed workplans; and considering how to evaluate their work. In addition, installation respondents said they strengthened their teamwork and coordination skills; ability to view the CAP holistically, linking action planning to outcomes; and overall organization and planning skills. These are general prevention skills and can be applied by prevention practitioners regardless of the specific problem area being addressed by the CAP—an especially critical approach for any scale up:

I think about drilling down into what is causing the issue in order to help me find a way to measure and try to solve it. [GTO] forces you to look at the reason for the problem.

Improved Communication

Respondents suggested that sharing the GTO workload helped Community Action Teams strengthen their communication and coordination with each other, and with other installations in their MAJCOM:

Prior to GTO I did most of [the CAP] by myself. As this process came on board, instead of working in a silo, I got other people involved. . . There was more buy-in.

Several divided up the work among their Community Action Team by subject matter expertise. Respondents noted the benefits of using the same “GTO language,” learning from other installations, sharing ideas, and comparing CAPs. This was facilitated by MAJCOMs and GTO coaches, at the training, and through phone calls.

Improved CAPs

Installation respondents said GTO improved their CAP. Of those who were somewhat or very familiar with previous CAPs (n = 24), 71% (n = 17) thought the current CAP (i.e., the CAP created using GTO) was better than in previous years.

Barriers to Scale-Up

Respondents consistently said accountability and oversight from higher level leadership is needed for scale-up. A perceived lack of communicated support from Headquarters Air Force, Wing commanders, and other installation leadership, limited buy-in was perceived as undermining the process. Without proper buy-in, some installation leadership insisted that the CAP meet their own priorities, and vetoed the plans developed by the Community Action Team; others told the Community Action Team it was not a priority:

We have to have support from the highest command. . . If you don’t have that people won’t take it seriously.

Respondents reported having delays and difficulty in accessing installation-level data. As a result, at least half of the respondents had not shared their community needs data with their full Community Action Team and installation leadership ahead of training.

Respondents had numerous competing demands on their time, making it challenging to complete the GTO tools. For some, the work involved in conducting a thorough planning process was a collateral duty because there were no full- or part-time prevention practitioners at an installation:

We (and our helping agencies) don’t have time to do [this additional duty] . . . I did most of [GTO] outside of work hours. . . When I talked to others, they [also] said “I don’t have the time.”

Discussion

Interim results suggest that GTO has demonstrated important successes in supporting the scale-up of prevention and resilience programs across DAF. Through a tiered learning system and structured process, GTO improved practitioner capability, communication among Community Action Teams, and ultimately helped installations develop strong CAPs. Installations intend to sustain the use of GTO to create future CAPs but face several challenges including a lack of time for prevention, leadership buy-in, and timely needs assessment data.

The majority of submitted CAPs were high quality, and individuals with prior CAP experience said they were better quality than in previous years. A core benefit to the GTO roll out was that staff with no prevention background developed critical planning and evaluation skills that can be translated to a variety of contexts. Where previously CAPs were completed to fulfill a requirement, now staff are engaging in data-informed and collaborative decision-making, and creating implementable workplans and evaluation tools. MAJCOMs were the primary channel for communicating with and supporting installations to use GTO, and successful roll-out depended on their understanding of the GTO process. Previous research suggests that ongoing coaching and the inclusion of a consultation process in the supervision structure can also increase the likelihood that EBPs are sustained (Esslinger et al., 2020; Kitson et al., 2008). Once staff became familiar with GTO, they saw a benefit to having a structured process with explicit guidance. The common GTO lexicon also improved communication within and across teams. Additional research is needed to evaluate the impact and sustainability of GTO on implementation quality and subsequent resilience outcomes. This study suggests that GTO achieved this success via improved communication, a common and cross-cutting lexicon, and a more rigorous and standardized process for CAP development.

Some significant barriers limited the success of this scale-up. DAF designed the Community Action Planning process to be a locally led, bottom-up effort intended to produce CAPs tailored to local needs. Without strong leadership support from all levels, bottom-up implementation support like GTO was unable to achieve its full potential, putting its sustainability at risk. According to the Promoting Action on Research Implementation in Health Services (PARiHS) framework, successful implementation of EBPs depends on context, including the presence of transformational leaders (Kitson et al., 2008). Although leadership in Resilience Operations supported empowerment as a core goal, other leadership support varied. GTO was not a mandated process, which also created tension for installation staff whose direct leaders had other priorities. Leadership priorities is a well-documented predictor of EBP uptake (Hannes et al., 2010; Vroom et al., 2021). Gaps in infrastructure (lack of timely data) and human capability (lack of time for prevention) also need to be addressed to support scale-up. One way to decrease the burden on personnel may be to allow Community Action Teams to replace non-evidence-based, mandated programs with EBPs selected through the GTO CAP process. This would avoid increasing the overall number of programs being implemented.

Finally, despite the GTO process, most sites did not adopt an EBP. Barriers to EBP uptake were similar to those identified in other studies and included a limited number of EBPs specific to military populations (Perkins et al., 2016), very few brief interventions that could be fit into a busy training schedule, and limited site resources to purchase or manage new programs (Sridhar et al., 2021). Many sites opted to implement their own programs and instead use GTO to evaluate and improve them.

The multitiered GTO learning system was one way in which a broad array of experts, researchers, and practitioners worked together to improve the use of evidence-based prevention in DAF. Future scale-up efforts should consider whether a multitiered learning system would be advantageous in planning scale-up efforts—and whether the same type of learning system would be needed for implementation. Figure 1 shows how the Framework for Going to Full Scale could be adjusted to reflect the unique infrastructure and adoption mechanisms required for scale-up. Findings from this study not only add nuance to existing elements of the framework, such as specifying the need for timely, cross-cutting data, but also new elements, such as the need for accountability and oversight. In addition, while the Framework was developed based on research in primarily low-resource settings (Barker et al., 2015), the current study demonstrates its utility in, and necessary modifications for, scaling up in a high-resource environment. More research is needed to continue examining the relevance of the Framework for Going to Scale for scale-up efforts to determine whether the findings from this study are replicated, especially in lower resourced and nonmilitary service delivery systems.

Figure 1.

Extending the IHI Framework for Going to Full Scale Using Getting To Outcomes (GTO).

Limitations

Due to DAF mandated timeline for completing CAPs there was no ability to pilot GTO before going DAF-wide, a common practice before going to scale (Barker et al., 2015). In addition, the relationship between the CAP quality rating system and actual implementation quality is unknown because implementation progress is not yet monitored by DAF; however, prior research suggests programs that conduct the GTO steps with higher quality have stronger program fidelity, a key marker for improved program outcomes (Acosta et al., 2020). We were also not able to complete the full 10-step cycle of GTO with installations. Research shows that continuous quality improvement (Step 9) is the stage at which teams solidify their GTO knowledge and make programmatic changes that result in stronger implementation (Chinman et al., 2012; Hunter et al., 2015, 2017). Future work would ideally allow for a pilot process before going to full scale; include a system for tracking implementation; and allow for sites to complete the full 10 step cycle.

Conclusion

Using GTO to support the scale-up of evidence-based prevention in DAF CAPs represents the largest implementation of GTO to date, and demonstrated some success in improving the perceived quality of the resultant program plans as well as essential prevention practitioner skills. Future scale-ups should ensure they have the necessary support systems before tackling such an ambitious and resource-intensive effort. Finally, additional research and evaluation are needed to further develop the Framework for Going to Full Scale and assess the role of implementation support strategies.

Footnotes

Acknowledgements

We would like to thank the Integrated Resilience directors during the period of this work, Major General James C. Johnson (retired), for his leadership in adopting Getting To Outcomes for use in the Air Force Community Action Plan process, and Brigadier General Michael E. Martin, who offered his encouragement and support to the project. We would like to acknowledge and are particularly indebted to Andra L. Tharp, PhD, HQE, who, at the time of this work, was the senior advisor for prevention, AF/A1Z. We are grateful to the numerous Headquarters Air Force A1Z staff who offered their guidance and support to the work and also to those in the Resilience Operations office who maintained our communications with the major commands and the installations throughout this project. We also thank the Community Support Program Managers and Violence Prevention Integrators at the major commands. In addition, we would like to thank Jonathan Scaccia, PhD, of the Dawn Chorus Group, and Gordon Hannah, PhD, for their data analysis support throughout this effort.

Authors’ Note

The views expressed in this article represent the personal views of the author(s) and are not necessarily the views of the Department of Defense or of the Department of the Air Force.

Authors’ Contributions

AS substantively contributed to the study conception and design, acquisition, analysis and interpretation of data, and drafted and revised the manuscript. JA substantively contributed to study conception and design, interpretation of data, and drafted and revised the manuscript. PE substantively contributed to the study conception and design, acquisition and interpretation of data, and manuscript revisions. JS substantively contributed to manuscript revisions. MC substantively contributed to study conception and design, interpretation of data, and drafted and revised the manuscript.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: At the time of this research, JS directed resilience operations in the Airman and Family Division of the Air Force Personnel Center. The authors report no other competing interests.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was sponsored by the Air Force Integrated Resilience Directorate (AF/A1Z).

Ethics Approval and Consent to Participate

This study was approved by the RAND Corporation’s Human Subjects Protection Committee (Study: 2018-0080, approved 10/9/18).

ORCID iD

Amy L Shearer

Notes

References

Acosta

J. D.

Chinman

Ebener

Malone

P. S.

Cannon

D’Amico

E. J.

(2020). Sustaining an evidence-based program over time: Moderators of sustainability and the role of the Getting To Outcomes® implementation support intervention. Prevention Science, 21, 807–819.

Barker

P. M.

Reid

Schall

M. W.

(2015). A framework for scaling up health interventions: Lessons from large-scale improvement initiatives in Africa. Implementation Science, 11, Article 12.

Charmaz

(2006). Constructing grounded theory: A practical guide through qualitative analysis. SAGE.

Chinman

Acosta

Ebener

Burkhart

Malone

P. S.

Paddock

S. M.

Clifford

Corsello

Duffey

Hunter

(2013). Intervening with practitioners to improve the quality of prevention: One-year findings from a randomized trial of Assets-Getting To Outcomes. The Journal of Primary Prevention, 34(3), 173–191.

Chinman

Acosta

Ebener

Malone

P. S.

Slaughter

M. E.

(2016). Can implementation support help community-based settings better deliver evidence-based sexual health promotion programs? A randomized trial of Getting To Outcomes®. Implementation Science, 11(1), Article 78. https://doi.org/10.1186/s13012-016-0446-y

Chinman

Ebener

Malone

P. S.

Cannon

D’Amico

E. J.

Acosta

(2018). Testing implementation support for evidence-based programs in community settings: A replication cluster-randomized trial of Getting To Outcomes®. Implementation Science, 13(1), Article 131. https://doi.org/10.1186/s13012-018-0825-7

Chinman

Ebener

P. A.

Shearer

A. L.

Acosta

J. D.

Hunter

S. B.

(2020). Getting To Outcomes® operations guide for U.S. Air Force community action teams. RAND Corporation.

Chinman

Hunter

S. B.

Ebener

(2012). Employing continuous quality improvement in community-based substance abuse programs. International Journal of Health Care Quality Assurance, 25(7), 604–617.

Chinman

Hunter

S. B.

Ebener

Paddock

S. M.

Stillman

Imm

Wandersman

(2008). The Getting To Outcomes demonstration and evaluation: An illustration of the prevention support system. American Journal of Community Psychology, 41(3–4), 206–224.

10.

Chinman

Tremain

Imm

Wandersman

(2009). Strengthening prevention performance using technology: A formative evaluation of interactive Getting To Outcomes®. American Journal of Orthopsychiatry, 79(4), 469–481.

11.

Cohen

Crabtree

(2006). Qualitative research guidelines project. Robert Wood Johnson Foundation. http://www.qualres.org/HomeMaxi-3803.html

12.

Courie

A. F.

Tate

(2020). The usability of the plan quality index to support evaluation of community health action plans. Military Medicine, 185(11–12), e1908–e1912. https://doi.org/10.1093/milmed/usaa206

13.

DAF. (2018). Authors’ review of past Community Action Plans. In J.

Acosta

(Ed.). Department of the Air Force

14.

DAF. (2022). Department of the Air Force resilience: About us. Office of the Secretary of the Air Force. https://www.resilience.af.mil/About-Us/

15.

Esslinger

Sprang

Ascienzo

Silman

(2020). Fidelity and sustainability in evidence-based treatments for children: An investigation of implementation determinants. Journal of Family Social Work, 23(2), 177–196.

16.

Farris

(2020). Getting To Outcomes® operations guide for U.S. Air Force community action teams: Content area module for Air Force sexual harassment prevention. RAND Corporation. https://www.rand.org/pubs/tools/TL311z3.html

17.

Gedney

C. R.

Wood

D. S.

Lundahl

Butters

R. P.

(2018). Sexual assault prevention efforts in the US Air Force: A systematic review and content analysis. Journal of Interpersonal Violence, 33(3), 421–441.

18.

Hannes

Pieters

Goedhuya

Aertgeerts

(2010). Exploring barriers to the implementation of evidence-based practice in psychiatry to inform health policy: A focus group based study. Community Mental Health Journal, 46(5), 423–432. https://doi.org/10.1007/s10597-009-9260-1

19.

Hardison

C. M.

Vaughan

C. A.

Meredith

L. S.

Weilant

Ross

(2020). Getting To Outcomes® operations guide for US Air Force community action teams: Content area module for workplace stress prevention and reduction in the Air Force. RAND Corporation.

20.

Hunter

S. B.

Ebener

P. A.

Chinman

Ober

A. J.

Huang

C. Y.

(2015). Promoting success: A Getting to Outcomes® guide to implementing continuous quality improvement for community service organizations. RAND Corporation.

21.

Hunter

S. B.

Rutter

C. M.

Ober

A. J.

Booth

M. S.

(2017). Building capacity for continuous quality improvement (CQI): A pilot study. Journal of Substance Abuse Treatment, 81, 44–52.

22.

Karre

J. K.

Perkins

D. F.

Aronson

K. R.

DiNallo

Kyler

S. J.

Olson

Mentzer

C. E.

(2017). A continuum of evidence on evidence-based programs: A new resource for use in military social service delivery. Military Behavioral Health, 5(4), 346–355.

23.

Kitson

A. L.

Rycroft-Malone

Harvey

McCormack

Seers

Titchen

(2008). Evaluating the successful implementation of evidence into practice using the PARiHS framework: Theoretical and practical challenges. Implementation Science, 3(1), 1–12.

24.

Perkins

D. F.

Aronson

K. R.

Karre

Kyler

S. J.

DiNallo

J. M.

(2016). Reducing barriers to evidence-based practice with military families: Clearinghouse for military family readiness. Military Behavioral Health, 4(1), 47–57.

25.

Shearer

A. L.

Ebener

(2020). Getting To Outcomes® operations guide for U.S. Air Force community action teams: Content area module for Air Force suicide prevention. RAND Corporation. https://www.rand.org/pubs/tools/TL311z1.html

26.

Sims

C. S.

Miller

L. L.

Trail

T. E.

Woods

Kofner

Rutter

C. M.

Posard

M. N.

Hall

Kleykamp

(2019). 2017 U.S. Air Force community feedback tool: Key results report for Air Force headquarters. RAND Corporation. https://www.rand.org/pubs/research_reports/RR3084.html

27.

Sridhar

Drahota

Walsworth

(2021). Facilitators and barriers to the utilization of the ACT SMART Implementation Toolkit in community-based organizations: A qualitative study. Implementation Science Communications, 2(1), 1–14. https://implementationsciencecomms.biomedcentral.com/articles/10.1186/s43058-021-00158-1#Sec17

28.

Troxel

W. M.

Holliday

S. B.

Shih

R. A.

Ebener

(2020). Getting To Outcomes® operations guide for U.S. Air Force community action teams: Content area module for Air Force sleep health promotion. RAND Corporation. https://www.rand.org/pubs/tools/TL311z4.html

29.

Vroom

E. B.

Massey

O. T.

Tyson

D. M.

Levin

B. L.

Green

A. L.

(2021). Conceptualizing implementation practice capacity in community-based organizations delivering evidence-based behavioral health services. Global Implementation Research and Applications, 1(4), 246–257.