Abstract
Introduction. In 2014, the Centers for Disease Control and Prevention funded the American Heart Association to implement policy, systems, and environment-focused strategies targeting access to healthy food and beverages, physical activity, and smoke-free environments. Method. To understand factors affecting implementation and variations in success across sites, evaluators conducted a multiple case study. Based on past literature, community sites were categorized as capacity-building or implementation-ready, for comparison. A sample of six communities were selected using a systematic selection tool. Through site visits, evaluators conducted interviews with program staff and community partners and assessed action plans. Results. Evaluators identified important implications for nationally coordinated community-based prevention programming. Differences in implementation varied by the communities’ readiness, with the most notable differences in how they planned activities and defined success. Existing partner relationships (or lack thereof) played a significant role, regardless of the American Heart Association’s existing presence within the communities, in the progression of initiatives and the differences observed among phases. Last, goals in capacity-building sites were tied to organizational goals while goals in implementation-ready sites were more incremental with increased community influence and buy-in. Discussion. Using national organizations as a mechanism to carry out large-scale community-based prevention work is a viable option that provides coordinated, wide-scale implementation without sacrificing a community’s priorities or input. In funding future initiatives, the presence of relationships and the time needed to cultivate such relationships should be accounted for in the planning and implementation processes, as well as both local and national expectations.
In 2014, the Centers for Disease Control and Prevention (CDC) funded three national organizations to build community capacity and implement policy, systems, and environmental (PSE) interventions focused on increasing access to healthy food and beverages, physical activity opportunities, and smoke-free environments. PSE interventions aim to go beyond the individual and focus on the context in which people live (Frieden, 2010). The program leveraged the national organizations’ reach to work in communities not previously served by CDC grants. Grantees adopted new approaches to better reach communities and built infrastructure to centrally manage the geographically dispersed projects. The benefit of this funding model was that it allowed national organizations to influence many communities through coordinated approaches to local health interventions. This article describes evaluation findings and lessons learned from one of the national organizations, the American Heart Association (AHA).
The Accelerating National Community Health Outcomes Through Reinforcing Partnerships Program (ANCHOR)
The AHA participated in the grant program to accelerate its 2020 Impact Goal to “improve the cardiovascular health of all Americans by 20% while reducing deaths from cardiovascular diseases and stroke by 20% by 2020.” Traditionally, the AHA focuses on building federal- and state-level policy and systems changes, so a community-based model for implementing PSE interventions was new to the organization. To increase capacity, the AHA invested in training, technical assistance, and resources to ensure they had appropriate staff, processes, and organizational structure to implement the program.
Through its affiliates, the AHA built community support to implement the initiative starting in May 2015 with a cohort of 15 communities. As the project unfolded, unexpected community factors influenced implementation and subsequently the communities’ likelihood of success. To better understand these factors, evaluators at Texas A&M University designed a multiple case study to explore the influences on success and how the existing AHA infrastructure contributed to the project. The multiple case study findings presented here are part of a larger, mixed-methods evaluation of the AHA Program.
Method
Evaluators selected a multiple case study design because it is a proven method to detect patterns of influence in community-based settings (Stake,1995; Yin, 2009). Evaluators examined multiple communities to draw a set of cross-case conclusions that are generalizable to other communities (Yin, 2009). The ANCHOR program was an exploratory initiative with varying levels of success, and evaluators hoped to use the multiple case study to account for variations in community context (Yin, 2009). Communities were selected as the unit of analysis, or cases, because evaluators wanted to capture environmental aspects as well as program implementation factors.
Study Sample
The ANCHOR program implemented two cohorts; however, this multiple case study was conducted during Cohort 1 so results could be used to inform Cohort 2. Communities were selected based on characteristics of program implementation because the study aimed to understand why some communities were more successful than others (Chaskin, 2001). Evaluators used a systematic selection tool to sort and select communities (Kegler, Steckler, Malek, & McLeroy, 1998). The assessment tool first classified communities based on their intervention topic areas: (1) healthy food and beverages, (2) physical activity, and (3) smoke-free environments. Then, sites were divided into two categories—capacity-building or implementation-ready—based on stage of implementation. Capacity-building sites included those who were mobilizing partners, establishing organizational structure, identifying resources, and planning for action (Florin, Mitchell, & Stevenson, 1993). Implementation-ready sites included sites focused on intervention implementation, refinement, and sustainability (Florin et al., 1993). Evaluators worked with the AHA to select two communities from each topic area, one capacity-building and one implementation-ready, for a total of six cases. See Figure 1 for a visual depiction of Cohort 1 site distribution and sites selected.
Map of Cohort 1 ANCHOR sites.
Data Collection
To understand the implementation processes in each community, evaluators used the Consolidated Framework for Implementation Research (CFIR) to develop qualitative data collection instruments (Damschroder et al., 2009). CFIR provides an overarching typology, which helps identify what implementation practices work within various contexts. Evaluators conducted semistructured interviews during site visits as the main source of data. Evaluators developed interview questions based on previous CFIR instruments, as well as knowledge of program factors, organizational infrastructure, and gaps in knowledge identified by AHA leaders (CFIR Team, 2014). Questions aligned with each of the five major CFIR domains (Damschroder et al., 2009). Table 1 details CFIR domains and associated constructs, and Table 2 shows CFIR domains with adapted definitions, specific to ANCHOR.
Damschroder et al.’s (2009) Consolidated Framework for Implementation Research Domains and Constructs.
CFIR Domains and Adapted Definitions Used in ANCHOR Analysis.
Note. CFIR = Consolidated Framework for Implementation Research; ANCHOR = Accelerating National Community Health Outcomes Through Reinforcing Partnerships Program; AHA = The American Heart Association.
In addition to qualitative data, evaluators also conducted a review of each site’s community action plan (CAP), which described their project’s goals, objectives, time lines, and individuals responsible for implementation (Butterfoss & Dunět, 2005). This review was done to assess the caliber of each site’s planning tools to determine if the quality of action plan contributed to their likelihood of success.
Study Participants
During each site visit, evaluators interacted with three different categories of people. The project manager, employed by the AHA to lead the projects locally, provided information on daily activities, community engagement, and AHA involvement. They also worked with evaluators to assess their CAP. Each project manager was asked to identify a local community partner who was involved in ANCHOR initiatives. Because each site’s interventions and stakeholders differed, the choice of community partner was intentionally left to the discretion of the project manager with instructions to choose an individual who had knowledge of the program, as well as community context. Local community partners were interviewed about building support and implementing activities. Last, an employee of the AHA, not directly engaged in ANCHOR activities, was interviewed for insight into how the organization received the project and assisted with implementation indirectly.
Interview Instrument Development and Protocol
Prior to data collection, evaluators created a qualitative data collection protocol that was approved by the Texas A&M University Institutional Review Board. Informed consent was solicited from each participant prior to the interviews. Three trained evaluators conducted the interviews using the protocol.
Interview scripts differed based on the individual’s role. Consistent with CFIR, project managers were asked questions about intervention characteristics, outer setting (community), inner setting (AHA), and the implementation process (Damschroder et al., 2009). Community partners were asked about intervention characteristics and the implementation process, as it pertained to community implementation. Last, AHA employees were asked about the intervention characteristics, implementation process, and the inner setting (AHA). Table 3 lists the interview questions.
Interview Questions Used for the Nationwide ANCHOR Program Conducted From May 2015 to April 2016, Broken Down by Project Role and CFIR Domain.
Note. CFIR = Consolidated Framework for Implementation Research; AHA = The American Heart Association; ANCHOR = Accelerating National Community Health Outcomes Through Reinforcing Partnerships Program. The CFIR was used as the theoretical framework for question development (Damschroder et al., 2009).
Each interview lasted approximately 45 minutes. Participant responses were audio-recorded and transcribed verbatim by an outside transcription firm. Once transcribed, all audio-recordings were erased to ensure confidentiality.
Quality of Action Plan Assessment
To assess the quality of each site’s CAP, evaluators worked with each project manager to complete an action plan review tool adapted from the State Plan Index tool (Butterfoss & Dunět, 2005). This tool was adapted because it was originally developed to assess state-level chronic disease prevention plans. Evaluators selected this tool because, unlike others in the literature, it is clearly separated into distinct assessment components (i.e., stakeholder involvement, goals, objectives, evaluation, etc.; Butterfoss & Dunět, 2005). This allowed evaluators to look at individual category ratings, in addition to the plan’s overall score. Project managers helped complete the assessment, which was important because they understood why certain activities were included and were able to clarify discrepancies.
Data Analysis
To understand ANCHOR program implementation within each community, evaluators followed a constructivist-interpretivist paradigm to better cognize the lived experiences and challenges communities faced in implementing ANCHOR interventions (Schwandt, 1994). This paradigm was selected because it seeks to understand, rather than discover, a truth (Crotty, 1998). Through the qualitative data analysis, evaluators aimed to highlight and describe the participants’ voices and ways in which they navigated the implementation process within their community context. Within this paradigm, the evaluators and participants are linked, and findings are created through the shared experiences (Lincoln & Guba, 1985, p. 97).
During the qualitative analysis, evaluators broke data into segments, larger codes, and broader themes based on the major domains articulated in the CFIR theoretical framework (Damschroder et al., 2009). This approach is consistent with what Schwandt (2007) terms, “an a priori, content specific scheme” where the codes are “developed from careful study of the problem or topic under investigation and the theoretical interests that drive the inquiry” (p. 32). However, additional data that were not consistent with a CFIR domain were also recorded, which is in line with the constructivist-interpretivist paradigm.
Four coders with knowledge of the program independently coded the transcripts, and then came together to compare themes. Discrepancies were resolved through group consensus. After coding data, researchers then followed Lincoln and Guba’s (1985) recommendations for improving the credibility and trustworthiness of the findings. This included the use of interviewer debriefs to review codes, themes, and interpretations and offer peer feedback, thereby serving as a sort of audit for the work. Last, the interview data collected from each of the three perspectives were combined to triangulate themes (Denzin, 1970).
Action plans were analyzed using a numerical rating system in which the extent of presence (or lack) of information was scored on a scale of 1 (low) to 5 (high) with scores in between representing a subjective judgement of the extent to which the information was included. A total of 54 statements were scored, fitting into eight components: (1) Involvement of Key Stakeholders, (2) Presentation of Data on Disease Burden and Existing Efforts, (3) Goals, (4) Objectives, (5) Selecting Population and Strategies for Interventions, (6) Integration of Strategies With Other Programs and Implementation of Plan, (7) Evaluation, and (8) Accessibility of Plan (Butterfoss & Dunět, 2005). Each statement was assessed independently and assigned a score of 1 to 5. In order to receive full points, the statement had to be fully addressed. For example, short-term, intermediate-, and long-term objectives were all evaluated separately with their own 1 to 5 ranking. If short-term objectives were only partially present, the statement would receive a 2 or 3, which is independent from the assessment of long-term objectives. Scores within each component grouping were averaged for overall component scores, which were then used to assign an overall rating.
Results
To organize the findings, results are arranged according to stage of implementation—capacity-building or implementation-ready. The results are further organized by CFIR constructs and other identified themes.
Capacity-Building Cases
Three CFIR domains emerged as primary themes among capacity-building communities—intervention characteristics, inner setting, and outer setting (Damschroder et al., 2009). Outcome expectations was identified as a primary theme, which was not tied to CFIR but heard frequently in the analysis.
Intervention Characteristics
Across all capacity-building cases, evaluators heard that interventions were selected based on communities’ needs and AHA affiliates’ priorities (e.g., AHA’s desire to leverage their advocacy work to enhance ANCHOR). Project managers expected to achieve organizational outcomes related to policies (state or local) in the priority areas (healthy food and beverages, physical activity, and smoke-free environments). Conversely, community partners expected to make community-level changes with priority population impact. As one community partner focused on physical activity in schools stated, “The outcomes are these kids are getting more minutes of activity every week, thus they should be healthier, in better shape and the less chance for—decrease chance of obesity.”
Team members viewed having multiple perspectives as beneficial rather than competing. Including the communities’ focus on the priority population and the organization’s focus on PSE implementation ensured that community needs were at the forefront of initiatives, while still keeping longer term policy goals in mind. Multiple focuses created a comprehensive agenda with community-based efforts as stepping stones to build support for larger systems or policy changes. For example, multiple locations achieved organizational-level healthy vending policies, but initial conversations focused on implementing smaller changes to build support for the initiative among patrons, leadership, and/or vendors.
Outer Setting
Across cases, partners were engaged in developing action plans to guide their work. One project manager said, “We engaged a lot of partners . . . having collaboration is very key in any public health work, not just ANCHOR. But if you really want to have sustainable change implemented, you need to be working with partners.” Community partners served as gatekeepers and planned meetings between project managers and other community partners.
The project teams often, but not always, received the support needed to implement their initiatives. They experienced difficulties in engaging organizational decision makers, despite frequent outreach. One partner described the challenge: We wanted to make sure that the program or the work that ANCHOR did was really geared towards the community and geared toward what their needs were. And so, I think that in certain cases, yes, we did have some really great buy-in and in some cases, no we didn’t.
Additionally, the project timeline exacerbated these challenges because communities had just 13 months to implement initiatives. All communities experienced setbacks, which resulted in examination and adjustment of action plans.
Inner Setting
Overall, local and regional AHA staff supported ANCHOR within capacity-building sites. Staff experienced initial limitations due to communication challenges. However, as AHA leadership engagement and communication improved, project managers felt widespread organizational support. However, capacity-building sites did not report leveraging existing AHA ties to partners or organizations during ANCHOR.
Outcome Expectations
Capacity-building sites felt successful in partnership engagement and building support for initiatives. However, the successes did not necessarily reflect the CDC objectives set at the beginning of the project. For instance, CDC’s primary outcome measure was population reach, measured by the total number of individuals potentially affected by the interventions. Given the short implementation time frame and other limitations, reach numbers were not always achieved. Nonetheless, community partners viewed the project as successful because they were able to increase awareness and build a foundation for initiatives to be sustained. One of the project managers characterized this dilemma as “need[ing] to be able to show CDC what they’re doing in response to the grant.”
Quality of Action Plans
Out of a possible total of 270, scores ranges from 161 to 235, indicating a wide range of quality with variability in the extent to which each component was addressed. Aggregate data indicated that plans focused on long-term goals and included data to justify the strategies. Plans, however, rated lower on the inclusion of objectives and a plan to evaluate initiatives. For example, one plan had a goal of increasing SNAP/EBT (Supplemental Nutrition Assistance Program/Electronic Benefit Transfer) benefit acceptance at farmers markets, but as the needs assessment was still underway, little detail was provided regarding the steps to accomplish this goal. Sites reported having to revise their plan throughout implementation. This is consistent with the finding that capacity-building sites focused on the longer term goal rather than community-level initiatives in the beginning, likely because they did not yet have the community-level perspective to guide local-level initiatives that would be expected to come through in the CAP objectives. They were able to make their CAP more community specific as ANCHOR progressed. See Table 4 for the full CAP assessment results.
Community Action Plan Assessment Results by Component. a
Note. CB = capacity-building; IR = implementation-ready.
Percentage of component points received. bJudgments of low versus high are in comparison to implementation-ready communities’ scores, not a comparison between components.
Implementation-Ready Cases
Similar to the capacity-building sites, two CFIR domains, intervention characteristics and outer setting, were identified as primary themes; however, the implementation process was also noted, which differed from the capacity-building sites who identified the inner setting (Damschroder et al., 2009). Outcomes expectations was also identified as a primary theme that was not tied to CFIR.
Intervention Characteristics
Implementation-ready communities began ANCHOR with the goal of increasing partner involvement and building support for their work. Specific topic areas were selected based on communities’ needs and priorities; however, these communities had ongoing initiatives that were leveraged for ANCHOR. Existing AHA affiliate activities and connections were instrumental in the communities’ work. One AHA staff member said, “I think the outcome is an expansion of [AHA’s] influence to some new areas that we haven’t addressed before.” Similar to the capacity-building sites, community partners wanted to see impacts on their priority population, like academic and behavior outcomes among children in classrooms or availability of healthier food options.
Implementation Process
Implementation-ready communities felt confident in their action plans from the beginning. One project manager described this by saying, “Our [CAP] and what strategies we had were spot on with what we need to do, with some minor variations.” Participants described little discrepancy between what they intended to do and what actually occurred. Small delays in implementation were overcome relatively quickly. A major influence on planning processes for implementation-ready sites was that many sites already had similar interventions in place or planned. These communities had the benefit of existing community and leadership support, so they were able to go through the planning and needs assessment phase very quickly. In two of the three communities, the project manager was well known and had worked on similar issues for many years. They had existing relationships to build on instead of having to build the initial ground-level support.
Outer Setting
Implementation-ready sites focused on community buy-in but were able to leverage existing ties to quickly engage stakeholders and begin implementation. The partners had ongoing communication and worked closely with ANCHOR staff throughout implementation. Overall, community partners considered the program to have good buy-in and involvement from the larger community. However, challenges did arise while trying to engage organizational leaders in two communities. Extra time was needed to build relationships with these gatekeepers.
Outcome Expectations
ANCHOR was viewed as successful in implementation-ready communities, specifically related to community engagement. However, disconnect occurred in defining success because AHA affiliate staff did not see engagement as a critical outcome. Instead, they focused on building support for future policies as a primary metric of success. Community members considered the progress as beneficial to their priority populations but thought there was still work to do. The high level of partner engagement was an unexpected but positive outcome.
Quality of Action Plans
Out of a possible total of 270, scores range from 181 to 192, indicating consistency of action plan quality. Aggregate data indicated that plans focused on objectives rather than long-term goals. For example, one site focused on implementing physical activity in schools through providing professional development for teachers, rather than stating the larger goal of broadly increasing physical activity among students. While this was understood as a long-term goal, the CAP focused on the activities to be implemented. Additionally, plans ranked highly in terms of selection of priority population and strategies. The plans rated lower regarding inclusion of data to justify initiatives. This supports interview findings because these sites already had strong community connection and input in choosing strategies, so more emphasis was on local-level initiatives with community perspective rather than justifying strategies with broad data or focusing on population-level goals. See Table 4 for full CAP assessment results.
Triangulated Findings and Key Differences
Several interesting findings emerged. The first observation is the domains identified as primary themes. Capacity-building and implementation-ready sites identified intervention characteristics, outer setting, and outcome expectations as primary themes. The difference came in that capacity-building sites also focused on inner setting, whereas implementation-ready sites focused on implementation process. This is notable because capacity-building sites did not immediately focus on implementation; therefore, the implementation process itself was not as salient during the interview. However, they were guided by AHA affiliate goals, making the inner setting relevant. Conversely, implementation-ready sites executed interventions from the start, so their process was discussed throughout the interviews. Inner setting was not highlighted because they had existing projects in place that already aligned with the organizational priorities. This is supported by CAP assessment findings that implementation-ready sites focused on objectives regarding community implementation whereas capacity-building sites focused on longer term goals aligned with AHA affiliates.
Outcome Expectations
Both site types identified outcome expectations that illustrate notable differences in defining success. Capacity-building and implementation-ready communities wanted partner engagement, but capacity-building communities also focused on policy changes (larger goals) from the onset. Implementation-ready sites approached the project incrementally, in line with community priorities. Capacity-building sites likely felt more pressure from the AHA affiliate, which influenced their desire for immediate policy change, whereas implementation-ready sites had a better understanding of community needs with immediate partner engagement and existing work to build on.
Overall, implementation-ready sites achieved at least one of their stated outcomes and had others in progress at the time of the interview. Capacity-building sites met their internal, community-driven goals but not necessarily the outcomes initially defined.
Implementation Process and Outer Setting
The multiple case study also revealed differences in the planning processes. Implementation-ready communities began with solid CAPs that presented a strong justification for the selected focus areas and strategies. Conversely, capacity-building sites initially focused on long-term goals and data and had to revise their plans based on community input. While all sites were encouraged to revise plans as needed, implementation-ready plans were revised to a lesser extent because they already reflected community needs and context, and anticipated delays and challenges, whereas the capacity-building cases had to readjust with setbacks.
In implementation-ready communities, the project manager worked closely with community partners to implement interventions. However, capacity-building communities relied on the project managers to lead activities and used partners for networking and connecting to key players. Implementation-ready sites were able to rely on previous connections and partner support, whereas capacity-building sites focused on broad community engagement and had difficulty engaging key partners.
Lack of time, lack of follow-up, and difficulty accessing key stakeholders impeded capacity-building communities’ progress. Both types ultimately received community support; it just took longer in capacity-building communities. Implementation-ready communities felt they were most successful in community engagement, even though they were also able to successfully implement PSE interventions. Capacity-building sites laid the groundwork for change but struggled to achieve tangible outcomes.
Inner Setting
Both capacity-building and implementation-ready sites built support for future work; however, the AHA affiliates considered the work in progress and not yet successful for both site types as the larger goals had not been achieved.
Discussion
Overall, this funding stream accomplished its goal of reaching geographically diverse areas and implementing community-based interventions through a national organization, like the AHA. However, differences in community readiness—implementation-ready versus capacity-building—can substantially affect this type of large-scale initiative. Providing an adequate amount of time for communities to build capacity and support for PSE interventions through stakeholder involvement, complementary initiatives, leadership support, and the presence of resources is essential.
While the community work in this multiple case study may or may not have been carried out through formal partnerships, similar principles still apply to convening informal stakeholders. The process of moving through necessary stages, starting with formation, before moving on to implementation, and maintenance, is consistent with previous coalition literature (Butterfoss, Goodman, & Wandersman, 1993). Convening stakeholders, establishing partnership infrastructure and norms, and conducting needs assessment are necessary parts of community-based work. These functions are associated with capacity-building sites in this study, which emphasizes how this is an essential step in setting the stage for implementation of large-scale community initiatives (Clark et al., 2006).
In terms of implementation research, this study provides interesting insights and considerations for implementation of community-based initiatives. The analysis revealed that outcome expectations were important for the implementation of PSE interventions, regardless of the stage of readiness. Outcome expectations can be closely tied to CFIR’s construct of evidence strength and quality. However, this study found that in community-based worked it is essential to make sure not only that the intervention is acceptable to the community but also that it will also advance existing priorities and local work. In addition, the alignment of goals between the inner (organization) and outer (community) settings was essential to successful PSE changes, which is not covered in CFIR’s current structure. The interrelationship of these domains was an interesting aspect of this study, which may be specific to community-based interventions. These findings provide considerations for future implementation research in community settings.
Implications for the Future
Based on the findings of this study, additional factors may be needed to assess implementation processes within community-based initiatives, like outcome expectations and the alignment of community and organizational priorities. Including these aspects in future community-based implementation research can help explain how interventions are conducted and sustained in community settings. Furthermore, it captures an emergent property—the relationship between inner and outer settings—which is not detailed in CFIR. By looking at the relationship between CFIR domains, additional insight into community context comes to light.
Limitations
As with any study, this evaluation has boundaries that are important to consider. First, as a multiple case study it is impossible to separate the findings from the settings (communities) where the data collection took place. As a result, community context is incorporated in the findings, which limits the generalizability in a traditional, realist sense. However, results can be helpful to other communities with similar priorities and challenges. Additionally, data were collected in person by evaluators, so responses may have been biased toward evaluator expectations. The constructivist–intrepretivist paradigm acknowledges that you cannot separate the researcher from the study but should acknowledge potential research biases upfront (Lincoln & Guba, 1985). In this case, evaluators were biased toward the success of the program, which informed the study’s findings by identifying constructive areas of improvement and lessons learned, rather than harsher areas of weaknesses.
CAPs were also assessed from the evaluator and program manager perspectives, which introduces potential sources of bias for success, since the managers were responsible for the programs. However, evaluators felt that the program managers completed the tool truthfully and acknowledged areas for improvement. Additionally, the potential for bias was outweighed by the participant voice that their presence provided, which was vital to the interpretation of the project.
Last, this evaluation was used to inform the second cohort of AHA funded communities, which made categorizing communities as capacity-building or implementation-ready helpful to future work. AHA staff were able to adjust the technical assistance and resources provided to communities based on the needs identified in this evaluation. However, the two categories may not represent the full extent of community readiness, which is much more complex, of all the sites. Therefore, the categorization may limit the scope of inquiry.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was funded by FOA DP14-1418–National Implementation and Dissemination of Chronic Disease Prevention (Category A) from the Centers for Disease Control and Prevention.