Applying Theory of Change to a Structural Change Initiative: Evaluation of Model Communities in a Diverse County

Abstract

Structural change approaches, also called policy, systems, and environmental change approaches, have been increasingly promoted and adopted by public ealth agencies in the past decade. These interventions require attention to multilevel, complex and contextual influences on individual and community health outcomes, requiring a sound theoretical framework that links the many processes and outcomes over time. The Model Communities program of Cook County, Illinois Communities Putting Prevention to Work program employed a theory of change (ToC) framework to inform the evaluations’ design and execution. The main objective of this study was to apply findings from the longitudinal multiple case study evaluation to develop an adapted ToC. We conducted 97 key informant interviews across three waves, with a focus on Model Communities program participants’, Communities Putting Prevention to Work staff, and technical assistance providers’ experiences over time. Four analysts organized and coded the data using qualitative software; exploratory functions and data matrices were employed throughout three waves of analysis. Adaptations to the ToC included the addition of a construct, “change readiness,” as well as refinements to constructs: organizational capacity (human capital, technical assistance, informal and formal leadership), local partnerships, and the importance of sustainability. The findings offer a data-informed theoretical framework that may be considered for use in evaluations of structural change interventions in complex and diverse counties.

Keywords

policy,systems,and environmental changes structural change theory of change community-based evaluations qualitative research

In the past decade, the Centers for Disease Control and Prevention (CDC) have funded large-scale national initiatives—such as Communities Putting Prevention to Work (CPPW)—designed to improve access to healthy eating and physical activity and ultimately, support chronic disease and obesity prevention (Bunnell et al., 2012; CDC, 2012, 2014, 2016, 2017; Soler, Whitten, & Ottley, 2014). Such structural change or multilevel interventions (Schölmerich & Kawachi, 2016) have been defined as “modifications to the physical, social, political, and economic environment in which people make health-related decisions” (Lieberman, Golden, & Earp, 2013, p. 520) and are also referred to as “policy, systems, and environmental” changes. Examples of strategies promoted through CPPW include school nutrition policies (e.g., policies banning sugar-sweetened beverages) and improving access to safe community park spaces (CDC, 2014; Fagen et al., 2014). Other organizations, including the Robert Wood Johnson Foundation, have also created funding streams to support this work (Leviton & Strunk, 2012; Mockenhaupt & Woodrum, 2015; Nichols, Ussery-Hall, Griffin-Blake, & Easton, 2012; Pettibone, Friend, Nargiso, & Florin, 2013).

While structural changes are not new, most studies in the literature do not explicitly report on or do not employ theoretical frameworks to guide evaluation (Asada, Lieberman, Neubauer, Hanneke, & Fagen, 2017; Kegler et al., 2015; Lieberman et al., 2013; Schölmerich & Kawachi, 2016). One notable challenge is the identification of frameworks that can guide longitudinal, complex, multilevel evaluations (Kegler et al., 2015; Lieberman et al., 2015; Schölmerich & Kawachi, 2016). In addition, best practices for how to apply frameworks such as systems theory, developmental evaluation, and evolutionary evaluation models are also limited (Archibald, 2015; Buckley, Archibald, Hargraves, & Trochim, 2015; Vo, 2013).

One framework that has been defined, conceptualized, and applied in a number of ways is the theory of change (ToC), which can be broadly understood as a guiding framework outlining how an intervention creates a sequence of intermediate outcomes leading to long-term outcomes (Breuer, Lee, De Silva, & Lund, 2016). The framework includes the processes of change necessary to achieve outcomes, assumptions, and contextual factors that influence this process, as well as a list of other potential elements. Despite the limited consensus in the literature on conceptual definitions (Breuer et al., 2016), the ToC aligns well with structural changes in that it guides program and policy evaluators to consider complex, multilevel factors. To our knowledge, few studies to date have applied the ToC to guide a structural change intervention focused on improving access to physical activity and healthy eating.

To address this gap in the literature, this article examines the application of a ToC framework that guided planning, implementation, and evaluation of the Model Communities grant program (2011-2012) that was funded to suburban Cook County as one component of the CPPW initiative. The ToC applied in this study was developed by the Cook County Department of Public Health and revised and expanded for the Model Communities program; we assessed this ToC by examining salient constructs based on multiple case studies conducted at three time points before, between, and after implementation of the Model Communities program. Using data from the longitudinal case study, we developed an adapted ToC based on the salient contructs and the relationships observed between them.

Method

CPPW and the Theory of Change

In March 2010, the Cook County Department of Public Health, the state-certified public health authority for nearly all of suburban Cook County, in collaboration with the Public Health Institute of Metropolitan Chicago, was awarded nearly $16 million as part of the CPPW initiative to implement chronic disease-related structural change interventions to make healthy eating and active living the easy choice for individuals. Suburban Cook County was an ideal jurisdiction to examine structural changes as it offered unique implementation factors with a diverse range of geopolitical and cultural contexts. This region spans more than 700 square miles, comprises over 130 municipalities in nearly 30 townships, and is home to 21 acute care hospitals and 700 public schools.

Prior to CPPW, Cook County Department of Public Health had developed a ToC as a strategic approach to address comprehensive behavior change for public health initiatives to broadly support the county’s structural change interventions. To tailor the ToC specifically for CPPW, county officers and evaluation team researchers (members from the University of Illinois at Chicago and the Consortium to Lower Obesity in Chicago Children of Lurie Children’s Hospital of Chicago) revised and expanded the original ToC to apply to the design, implementation, and evaluation of the CPPW initiative. This ToC (Figure 1) combined three key components—health communications, capacity building, and alliance strengthening—thought to causally lead to increased organizational capacity, advance structural changes, and ultimately improve population level health outcomes, as indicated by unidirectional arrows (Figure 1). Notably, the framework assumed that interventions targeted to these three components and delivered at multiple levels in an integrated manner would be effective in facilitating structural changes. However, the framework had not previously been applied to a complex structural intervention like the Model Communities program and thus the team was unsure if these assumptions were accurate. In response, the purpose of this study was to examine how the ToC developed specifically for the Model Communities program (described next) aligned with actual implementation experiences from the multiple case study evaluation.

Figure 1.

Theory of change (ToC) for Cook County Department of Public Health (CCDPH).

Model Communities

Within suburban Cook County’s CPPW approach, the Model Communities grant program was a key health promotion strategy. Nearly $4 million was awarded to 38 grantees, of which 17 were defined as local municipalities, 12 as school districts, 12 as community-based organizations, and 2 as hospitals. While CPPW was funded for 2 years, the Model Communities project lasted 1 year (February 2011-February 2012) during which each grantee was expected to solidify and advance their respective structural changes. Additional details about the Model Communities program have been published elsewhere (Dombrowski et al., 2013). Since the Model Communities program’s planning, implementation, and evaluation was informed by the ToC (Figure 1), the three primary elements of Model Communities included (1) enhancing organizational and community capacity of Model Communities grantees, (2) strengthening alliances between Model Communities and like-minded partner organizations, and (3) implementing health communications strategies to promote reduction of sugar-sweetened beverage consumption (Dombrowski et al., 2013). The health communications component involved a plan to change social norms about health; however, this was evaluated independently by a separate team and is not addressed here. Instead, we focused on two key constructs of the ToC that align with the Model Communities program design. To promote organizational capacity, the Model Communities program offered various intervention support activities in the form of trainings and technical assistance (Table 1). Trainings were coordinated by the MidAmerica Center for Public Health Practice. In addition, technical assistance providers offered specialized consultation to each grantee.

Table 1.

Table of Intervention Support Activities.

Type of intervention support activity	Activity
Trainings and presentations	• Staff trainings
	• Pre-RFP trainings
	• AHAC quarterly meetings
	• Change Institute (3 in person; 2 webinars)
	• School trainings
	• Regional trainings
	• Place and health trainings and dialogues
	• School nurse leadership training
	• Example of trainings: building community coaliations
Technical assistance (one on one and group facilitation)	• Provided by TA partners
	• Provided by staff (included e-communications, in person and telephone)
	• Specialized consultation related to (a) community and roadway design for active living, (b) healthy school environments, (c) chronic disease self-management programs, (d) community gardening, (e) worksite wellness, and (f) breastfeeding
Communications/resource sharing	• Cook County Connects
	• AHAC Newsletter
	• CCDPH Website

Note. RFP = request for proposals; AHAC = Alliance for Healthy and Active Communities; TA = technical assistance; CCDPH = Cook County Department of Public Health.

To address alliance strengthening, grantees were encouraged to coordinate among multiple sectors within a single community: among partners from the same sector (e.g., school districts), across multiple communities, or both. They were also encouraged to partner with grassroots organizations, community residents, or youth groups. Last, as noted, the CPPW project team created opportunities to facilitate collaborations between the attending grantees at intervention support activities workshops and other meetings.

Study Design and Population

The longitudinal multiple case study design was intended to capture structural change implementation as an ongoing, process-related changes across time (Yin, 2009). With 38 total Model Communities grantees, a stratified maximum variation sampling strategy was applied (Patton, 2015); selection criteria included competency level, readiness level, organization type, funding level, and geographic region. The final sample consisted of grantee organizations from two local municipalities, two school districts, one school district foundation, and one community-based organization (Table 2). As is the goal with maximum variation sampling, these were selected to examine the process of implementation in various organizational contexts. A total of six Model Communities were selected based on the research team’s resources. For additional details about specific grantees, see Dombrowski et al. (2013).

Table 2.

Characteristics of Model Communities Grantees Included in Sample.

Model Community	Organization type	Geographic region^a	Population^b	Structural goals achieved
1	Local municipality	South	14,300	• Active transportation plan
				• Joint use agreement
				• Nutrition standards for vending machines
				• Installation of bike racks
2	Local municipality	North	75,000	• Bike signage
				• Revised bicycle plan
				• Intergenerational community garden
3	School district	West	13,680	• Worksite wellness
3	School district	West	13,680	• Healthy Schools program
4	School district	Southwest	1,300	• School food and physical activity changes
4	School district	Southwest	1,300	• Comprehensive action plan in schools
5	School district foundation	Northwest	3,600	• Worksite wellness
				• School meal and physical activity changes
				• School garden
				• Safe routes to school
6	Community-based organization	South	105,786	• Chronic disease self-management program

Region of suburban Cook County, Illinois. ^bPopulation catchment area that Model Communities organization targets.

Data Collection

From each Model Communities organization, a maximum of six participants were selected for interviews: two representatives from each Model Communities grantee (designated by the organization), up to two technical assistance providers, and two CPPW staff members (one manager and one coordinator) (Table 3). Three waves of data collection were conducted to reflect initial, middle, and final time points. For the final wave, CPPW directors and support staff were also interviewed. The analysis team was led by a faculty member with expertise in evaluation with three trained doctoral-level researchers. A total of 97 interviews were conducted, 89 of which were conducted in-person, 7 via phone, and 1 via e-mail. The various modes of data collection were designed to be flexible to stakeholders’ availability. In-person and phone sessions, lasting 45 to 70 minutes, were audio recorded and transcribed. The study was approved by the institutional review board of the University of Illinois at Chicago (#2010-0884).

Table 3.

Number of Key Informant Interviews by Respondent Type and Wave of Data Collection.

Respondent type	Wave 1, Sept-Oct 2011	Wave 2, Jan-Feb 2012	Wave 3, May-June 2012	Total
Model Communities representatives	12	12	12	36
Technical assistance providers^a	9	8	7	24
CPPW^a coordinators	6	6	3	15
CPPW managers	6	6	3	15
CPPW leadership and support staff^b	0	0	7	7
Total	33	32	32	97

Note. CPPW = Communities Putting Prevention to Work.

Since technical assistance providers and suburban Cook County CPPW staff were assigned to more than one Model Community, some respondents participated in multiple interviews.

Data Collection Instruments

Three interview guides specific to respondent type were created for each wave of interview; all instruments were informed by ToC constructs and iteratively revised. The original guides were pilot tested for flow and appropriate terminology prior to Wave 1; in addition, we expected and included revisions to the guide during and after each wave of interviews to provide a deeper understanding of implementation as it evolved. Additional revisions across waves were incorporated based on incoming data and analyst observations; pilot testing was not conducted on Waves 2 and 3 interview guides. The questions were broadly divided into the following: (a) provision and participation in intervention support activities; (b) change in organizational capacity and alliances over time; (c) facilitators/barriers to goal achievement; and (d) associations between structural change goal achievement and participation in intervention support activities, organizational capacity, and alliance strengthening.

Coding and Analysis

A coding guide was developed based on study objectives, ToC constructs, and evaluation questions. Code families were created in ATLAS.ti Qualitative Data Analysis Software to organize data by wave, type of respondent, and salient themes. Each subsequent coding guide was modified to reflect changes over time. Team coding was performed by four analysts in ATLAS.ti; for each wave, analysts first coded independently and then met several times to discuss code applications, discrepancies, and revise code descriptions. Each subsequent analysis was also conducted to look across waves. Memos were applied to capture project progress and analysts’ observations. Interrater agreement was over 80% for all three waves (Miles, Huberman, & Saldaña, 2014). Atlas.ti exploratory functions were employed to visualize and organize data; cooccurrence matrices were created to examine the associations between codes to elucidate contextual factors and how these factors shaped the specific manifestations of a given theme.

Results

This study aimed to examine a ToC based on a longitudinal case study evaluation and experiences of structural change implementation. The results presented here highlight adapations to the framework as a result of this examination. Since the ToC’s original design was aimed for broader county-level strategic change, our revisions to the ToC framework—specifically, the capacity-building and alliance-strengthening constructs—reflect the Model Communities program’s structural change initiative. Henceforth, the “original ToC” will refer to Figure 1 and revised model is referred to as “adapted ToC” (Figure 2). Table 4 lists sample illustrative quotes that correspond to the highlighted constructs in the adapted framework. The following section is organized by the salient constructs identified in the adapted ToC framework that highlight the key changes made to the original ToC (Table 5). The constructs align with those illustrated in Figure 2, which outline how the key constructs of capacity building and alliance strengthening lead to increased structural changes, sustainability of structural changes, and in the long term, improved population behaviors and chronic disease outcomes.

Figure 2.

Adapted theory of change (ToC).

Table 4.

Themes and Illustrative Quotes.

Themes	Illustrative quotes
Readiness/change readiness	• I think it made all of us start looking at opportunities differently and that’s really been the upsurge of this. It just kind of puts your focus in a different place. I’ve been here for almost 15 years and we’ve always focused on what’s going to be productive in terms of financial input because money is always tight. What this [Model Communities Grant Program] has done is changed our focus . . . so that’s been huge for us . . . I think it just opens your eyes and starts you focusing in a different direction. [Model Communities grantee]
Readiness/change readiness	• I feel like the more trainings and the more knowledge I gain, the more passionate I’m getting about this and then I can share that passion with other people. And when you see someone so enthusiastic about something then you get excited about it and it’s kind of like a snowball effect, it just keeps going to the next person and the next person and then it gets things done. [Model Communities grantee]
Enhanced organizational capacity	• Human Capital: For them it was really having that person who was coordinating everything and could reach out to all the schools and follow-up with them and kind of know what was going on, keep on top of things. That’s really made the biggest difference. [TA provider]
	• Direct TA: [TA provider] is like a cheerleader for everything. She’s so upbeat and so positive about everything. You feel like you can do anything when you sit down with her. [Model Communities grantee]
	• Direct TA: I do think that a lot of times we’re just building confidence in people, and giving them a lot of the tools that they need to actually execute because they don’t know where to start . . . I think we just expedited the process for her. [TA provider]
	• Formal Leadership: Our main contact is the Superintendent, which is great and I don’t think a lot of school districts have a Superintendent 100% involved actively participating in the grant. I think leadership and her desire to keep improving the district and her keeping the community involved. [CPPW staff]
Local partnerships	• As far as from the village staffing, yes. I think we were very fortunate with the garden because for instance in our Engineering Public Works Department we already had a full-time horticulturist. And so he worked with us and continues, like he and I have been meeting about making plans for the second year. [Model Communities grantee]
	• I think it’s so key, not only the school district who we collaborate with, but like I said with the hospital and with key businesses in the community as well, just help us get a wealth of people and resources to come out and have these initiatives happen. [Model Communities grantee]
	• Really like they say it takes a village to teach a child and it’s really true. The more partners you can have and the more people invested I think the better results you’re going to get for what you want to achieve. [Model Communities grantee]
Sustainability of structural change approaches	• I am looking for funding opportunities all the time. We received funding from [organization] . . . We’re looking at some smaller grants, a collective brands grant that has to do with more physical activity opportunities, the Lowe’s grant for the garden. We are currently about to partner with Home Depot and so we’re hoping they’re going to give us some resources. [Model Communities grantee]
	• Well, budget is always an impact for sustainability. We did receive an additional $1,000 grant from [organization]. I asked for $3,000, we were lucky to get $1,000, so that will help with assisting the gardens. As far as the after school club that is a concern we have because of the budgets. [Model Communities grantee]
	• I think the partnership that we developed with [TA provider] is a sustainable partnership and we are looking at other things that we are going to be doing together, so I think it was a good opportunity to bring some organizations together that had some common interest and common goals and objectives to link themselves around. [Model Communities grantee]

Note. TA = technical assistance.

Table 5.

Key Adaptions to Original ToC Framework.

Original ToC framework	Adapted ToC framework
Capacity building
• Considers provision of training, technical assistance and online collaborative space leading to increased organizational capacity	• In addition to direct technical assistance, adds contribution of human capital (designated Model Communities person) and strong informal/formal leaders to increased organizational capacity
	• Includes the role of leaders to improving local partnerships (below) as an important (bidirectional) relationship to improving structural changes
Alliance strengthening
• Focuses on enhanced partnerships broadly leading to countywide alliances at meetings/events to facilitate collaborations	• Focuses on local partnerships (rather than broad countywide alliances) as key facilitator in enhancing collaborations
	• Includes role of local partnerships and collaborations in improving sustainability of structural change
Increased structural change in Cook County leading to population behavior change	Addition of sustainability of structural change (in addition to increased structural change) leading to population of population behavior change
Consideration of readiness in design and implementation of Model Communities program; however, not included in original ToC	Inclusion of readiness/change readiness as important factors in success of structural change implementation

Note. ToC = theory of change.

Readiness and Change Readiness: Before and After Model Communities

While the original ToC did not explicitly include “readiness,” the Model Communities program and evaluation team conceptualized readiness as the extent that an “organization had the appropriate resources (e.g., human capital), policies and procedures, and intra- extra-organizational characteristics to facilitate change.” This definition was limited to initial readiness (individual or organizational) and its role as a predictor of success for implementation (Stevens, 2013). Grantees who perceived themselves or who were perceived by technical assistance providers/CPPW staff as having higher organizational capacity and strong existing partnerships at the start of the grant were better prepared to advance project activities. These initial indicators of readiness were particularly salient due to the short time frame of the grant. For example, organizations that had strong partnerships reported being more prepared to implement structural change steps in their communities:

We had the partnerships in place so we didn’t have to do a lot of work to bring people to the table . . . so we were able to hit the ground fairly well moving forward. [Model Communities grantee]

More recently, readiness has also been conceptualized as “change readiness,” which expands the definition beyond an initial state of set conditions and rather as a process or trajectory that considers the interaction of the individual, contextual, and collective factors and how they respond to one another over the course of time (Stevens, 2013). The case study also supported this conceptualization of “change readiness.” Model Communities grantees reported that a number of intervention support activities fostered new and positive attitudes toward their initiatives, which enhanced readiness for continued efforts and sustainability of their work. For example, participants noted that as they witnessed the positive impacts of their progress, enthusiasm for the work also increased.

Although initial readiness was perceived to be critical to “hitting the ground running,” overall, respondents reflected that readiness was not a static concept but one that iteratively evolved with time. The concept of “change readiness” aptly captured the dynamic evolution of readiness in Model Communities participants as they observed outcomes of their work. In Figure 2, both readiness and change readiness are represented in the largest box as a contextual factor because all other factors were influenced by the ongoing and evolving readiness of grantees as described here.

Enhanced Organizational Capacity: Capacity Building

The original ToC assumed that intervention support activities activities—such as still trainings—would improve individual and organizational capacity. As previously noted, however, this part of the ToC did not account for contextual and other factors that can facilitate or impede these processes, nor did it consider the relationship between individual and organizational capacity. The adapted ToC reflects a strong theme around organizational capacity rather than individual capacity. Organizational capacity was defined as “Model Communities’s ability to successfully apply its skills and resources to accomplish project goals and satisfy grant requirements.” Three constructs were most notable for enhancing organizational capacity: (1) human capital, (2) technical assistance, and (3) formal and informal leadership; each are illustrated in Figure 2 within capacity building and understood as facilitating an increase in organizational capacity to advance structural change. These relationships are represented as directional arrows from the constructs to increased organizational capacity.

Human capital. Adequate human capital—a designated Model Communities staff person—who had protected time for grant-related activities—was identified as a key success factor for advancing implementation:

So what I have observed amongst all the grantees is that the people that hired somebody specifically to work on this are getting more done, which is not that surprising. To have a dedicated person to work on the PSE [policy, systems, and environmental; structural] change makes it happen. All my districts where it’s just like they added it to someone’s job responsibilities, those people just don’t have time to do this and it’s not moving forward. [CPPW staff]

Technical assistance. While organizational capacity building activities included both technical assistance and intervention support activities (Table 2), respondents overwhelmingly identified technical assistance as the most critical facilitating factor for the advancement of structural change goals. The technical assistance providers’ intimate knowledge of the Model Communities’ strengths and issues and provided customized assistance.

[I thought] Can we do it, do we have the capacity? But then working with all the outside assistance from . . . the TA provider helped make that a reality because then, it’s not the strain that I kind of anticipated. [Model Communities grantee]

Notably, Model Communities grantees highlighted the importance of technical assistance given the short grant period. Since structural change initiatives were new to most Model Communities grantees, technical assistance “expedited the process.”

Formal and informal leadership. The original ToC had not explicitly outlined the role of leadership as a construct of interest; yet, leadership emerged as a critical construct to enhancing organizational capacity. Model Communities initiatives were primarily conducted in hierarchical organizations, such as school districts; not surprisingly, formal leadership—defined as “strategic guidance toward goal attainment from persons with formal positions of authority within the Model Communities organization”—was reported as a critical construct. Formal leaders were high-level decision makers such as superintendents, board members, or government officials. Informal leaders were defined as “people who champion structural change despite not holding a position of designated authority.” In addition, leaders were described as having extensive local knowledge of the community as well as influential partnerships (discussed next) with change agents.

[Informal leader] is a very competent, very strong woman. She knew what she wanted, she got it done and the way she was able to do that is she already had relationships built in the village. She had the relationship with Public Works . . . teen center and the senior center, so I feel like she knew what she wanted going in. [CPPW staff]

In addition to leadership as a construct within capacity building, Figure 2 illustrates how strong leaders also facilitated local partnerships that supported structural changes (bidirectional arrows linking these constructs), which enhanced collaborations.

Alliance Strengthening: Local Partnerships

The original ToC focused on strengthening alliances (Figure 1), as well as providing opportunities for Model Communities to increase collaborative and synergistic efforts toward structural change. Respondents pointed to existing local partnerships as a critical element to their success:

I think our local collaboration was the key to the success of this project. It was definitely the Village, the Park District, and the School District. [Model Communities grantee]

While no grantees explicitly noted that building alliances over the course of the grant was challenging, their limited descriptions of new collaborations indicated that the latter was less attainable in their limited time frame. Overall, however, all respondents indicated that partnerships was a critical success factor.

Sustainability of Structural Change

One limitation of the original ToC was that the relationships between constructs ended at “increase structural changes” (Figure 1) and did not explicitly include sustainability beyond the grant period. The Model Communities program and evaluation team defined sustainability as “a grantee’s capacity to support and uphold its structural change activities beyond the grant period with collaborations that continue to be operational, cohesive, and growing.” Technical assistance providers and CPPW staff provided supports for Model Communities grantees to address sustainability. When asked how grantees would continue to uphold and advance the work achieved, they reported identifying funding sources or applying for future grants, critical activities to sustaining their efforts. Additionally, new partnerships with technical assistance providers and other community organizations were also noted to increase potential for sustainability (Table 4). This relationship is illustrated in Figure 2 as the arrow pointing from alliance strengthening to sustainability of structural changes. Last, grantees felt that the short time frame of the grant posed a challenge for sustainability planning:

The limitation of this grant, and we are hoping that this is something that gets looked at in the future, the one year does not really extend itself to real sustainability . . . so the idea of a one-year grant to produce real change is . . . I just think it’s something that does not lend itself to that, because it’s one time and then everybody disappears. [Model Communities grantee]

Despite these reported challenges with time frame, many grantees continued to advance their structural change goals or even adopt new strategies beyond the Model Communities grant, suggesting sustainability of the program. For example, some municipalities in Cook County continue to receive technical assistance support for implementing Complete Streets from Active Transportation Alliance (a CPPW technical assistance provider) due to continued funding supports. In addition, some initiatives were intentionally ended after the Model Communities program but communities advanced to other structural changes (e.g., farmers markets) after the grant period (Massuda-Barnett, 2017, personal communication). To highlight the importance of sustainability, Figure 2 points to this construct eventually leading to change in population behaviors and finally, improved chronic disease outcomes.

Discussion

In response to the dearth of literature for structural change evaluations, this study offers an adapted ToC framework informed by data-driven implementation experiences from a longitudinal multiple case study evaluation in a complex and diverse county. The study design allowed us to evaluate stakeholders’ implementation experiences throughout the Model Communities program, identifying time- and context-specific constructs. The adapted ToC highlighted nuanced constructs specific to the Model Communities program participants’, CPPW staff, and providers’ experiences, including: “change readiness,” organizational capacity (human capital, direct technical assistance, informal and formal leadership), local partnerships, and the importance of sustainability. In addition to potential implications for practice (described below), the findings also offer a formative theoretical contribution to the structural change evaluation literature that currently is limited in its understanding of salient constructs to consider.

The challenge of adapting a framework to comprehensively address the dynamic processes of structural change implementation cannot be understated (Asada et al., 2017; Brownson, Brennan, Evenson, & Leviton, 2012; Leviton & Strunk, 2012; Lieberman et al., 2015). As observed by Kegler et al. (2015), grantees implemented a “constellation” of activities, adding complexity and necessitating a flexible evaluation approach. Despite these challenges, our findings align with previous structural change literature. For example, the importance of technical support, strong leadership, and partnerships as key facilitators is consistent with other CPPW projects (Kegler et al., 2015). In addition, our inclusion of organizational capacity to the framework is also consistent with studies indicating structural change interventions are more successfully sustained when they integrate with or “become part of the organizations’ norms and fabric” (DeJoy et al., 2012, p. 416).

Notably, while some constructs were not explicitly included in the original ToC, the Model Communities evaluation team considered readiness and sustainability in the evaluation. For example, grantees’ self-reported readiness was measured via survey prior to the evaluation and considered in our sampling strategy for the multiple case study. Sustainability was considered but not comprehensively integrated into the evaluation as a result of not being explicitly included in the ToC. As a key lessons learned, an earlier conceptualization of such constructs and how they were integrated with the overall ToC would have better served our evaluation efforts, highlighting the importance of applying a well aligned framework at the design stage.

Limitations

There are several notable limitations in this study. First, our findings from six Model Communities are not intended to be representative of communities nationwide. Second, since constructs were measured by Model Communities respondents’ experiences, this study was limited in its ability to triangulate constructs with other outcome measurements. Other evaluations have noted the challenge of including behavioral or health outcomes given the short grant time frames (Kegler et al., 2015). Furthermore, grantees may have received other county-level support systems (Cook County Department of Public Health support but external to the Model Communities grant) that were not measured in this evaluation. Future studies may include a wider account of structural change within the context of their communities. Third, as discussed, the 1 year evaluation period influenced almost all constructs; for example, it is likely that given more time, grantees would have expanded to broader countywide alliances (as noted in the original ToC) rather than focused on local partnerships. Future evaluations may more intentionally consider the impact of the time frame and adapt their evaluations and ToCs accordingly. Last, Kegler et al. (2015) noted that evaluators should be involved early in the planning phases of the initiative; in the CPPW evaluation, CPPW leadership and support staff were interviewed only during the final wave. In hindsight, this was a lost opportunity and future studies should include leadership and support staff in the study design, sampling, and instrument development.

Implications for Practice

The ToC was applied as an integration of theory and implementation, stressing change over time in this complex and dynamic setting. While this formative work requires additional evaluations with community-based structural interventions, a refined ToC may provide evidence-informed guidance for local public health departments as they design, implement, and evaluate structural interventions. First, the refined ToC can inform an assessment to determine readiness needed to advance structural changes and inform key strategy development. For example, if formal leadership is not on board, it may be critical to implement a systems-wide approach (e.g., across the region), while also supporting engagement at the local level (e.g., school district). This may further lead to the refined ToC guiding decision making and use of resources to expand structural changes or their supporting actions. Second, the refined ToC may support development of a more comprehensive evaluation plan or serve as an evaluation tool for structural interventions in geopolitically complex jurisdictions. The constructs in the ToC identify key areas for consideration and can help demonstrate areas of alignment between constructs that are (1) strongest, and most effective; (2) aligned but gaps in the supporting actions toward structural change may be required, and (3) not aligned to the particular context and requiring further assessment. Last, applying the ToC for evaluation may also call for additional evaluation designs, such as development evaluation, to track ongoing implementation and learning with the adapted ToC, with the engagement of key stakeholders. Public health departments can use this framework to continuously adapt and possibly engage multiple evaluation designs to inform structural change and promote sustainable implementation strategies. In addition, demonstrating “what works” through the refined ToC may inform more detailed operational definitions for evaluation measurement. Such new details may help practitioners and evaluators alike begin to track the specific supporting actions to document their performance.

Footnotes

Acknowledgements

The authors would like to thank Wesley Epplin and Rachel Dombrowski for their invaluable contributions to the CPPW evaluation.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The CPPW evaluation was made possible by a cooperative agreement from the Centers for Disease Control and Prevention (Grant No.: 1U58DP002623-01) to the Public Health Institute of Metropolitan Chicago (PHIMC) and the Cook County Department of Public Health (CCDPH). The authors would also like to gratefully acknowledge support of the Centers for Disease Control and Prevention and ICF International to attend a CPPW writing workshop, and Jennifer Patashnick for her helpful guidance and expertise. The findings and conclusions in this article are those of the authors and do not necessarily represent the views of the U.S. Department of Health and Human Services, the Centers for Disease Control and Prevention, CCDPH and PHIMC.

References

Archibald

(2015). “They just know”: The epistemological politics of “evidence-based” non-formal education. Evaluation and Program Planning, 48, 137-148.

Asada

Lieberman

Neubauer

L. C.

Hanneke

Fagen

M. C.

(2017). Evaluating structural change approaches to health promotion: An exploratory scoping review of a decade of US progress. Health Education & Behavior, 45, 153-166.

Breuer

Lee

De Silva

Lund

(2016). Using theory of change to design and evaluate public health interventions: A systematic review. Implementation Science, 11, 63. Retrieved from https://implementationscience.biomedcentral.com/articles/10.1186/s13012-016-0422-6

Brownson

R. C.

Brennan

L. K.

Evenson

K. R.

Leviton

L. C.

(2012). Lessons from a mixed-methods approach to evaluating active living by design. American Journal of Preventive Medicine, 43(5 Suppl. 4), S271-S280.

Buckley

Archibald

Hargraves

Trochim

W. M.

(2015). Defining and teaching evaluative thinking: Insights from research on critical thinking. American Journal of Evaluation, 36, 375-388.

Bunnell

O’Neil

Soler

Payne

Giles

W. H.

Collins

Bauer

(2012). Fifty communities putting prevention to work: Accelerating chronic disease prevention through policy, systems and environmental change. Journal of Community Health, 37, 1081-1090.

Centers for Disease Control and Prevention. (2012). Action communities for health, innovation, and environmental change. Retrieved from http://www.cdc.gov/nccdphp/dch/programs/healthycommunitiesprogram/communities/achieve/index.htm

Centers for Disease Control and Prevention. (2014). Communities putting prevention to work. Retrieved from http://www.cdc.gov/communitiesputtingpreventiontowork/

Centers for Disease Control and Prevention. (2016). Community transformation grants. Retrieved from http://www.cdc.gov/communitytransformation/

10.

Centers for Disease Control and Prevention. (2017). Partnerships to improve community health. Retrieved from https://www.cdc.gov/nccdphp/dch/programs/partnershipstoimprovecommunityhealth/

11.

DeJoy

D. M.

Wilson

M. G.

Padilla

H. M.

Goetzel

R. Z.

Parker

K. B.

Della

L. J.

Roemer

E. C.

(2012). Process evaluation results from an environmentally focused worksite weight management study. Health Education & Behavior, 39, 405-418.

12.

Dombrowski

R. D.

Mason

Welch

S. B.

Welter

Barnett

G. M.

Cedeño

(2013). Model communities as a strategy for achieving policy, systems and environmental change for obesity control and reduction. Building Community Capacity, 5, 1-18.

13.

Fagen

M. C.

Asada

Welch

Dombrowski

Gilmet

Welter

. . . Mason

(2014). Policy, systems, and environmentally oriented school-based obesity prevention: Opportunities and challenges. Journal of Prevention & Intervention in the Community, 42, 95-111.

14.

Kegler

M. C.

Honeycutt

Davis

Dauria

Berg

Dove

. . . Hawkins

(2015). Policy, systems, and environmental change in the Mississippi Delta: Considerations for evaluation design. Health Education & Behavior, 42(1 Suppl.), 57S-66S.

15.

Leviton

L. C.

Strunk

S. L.

(2012). Active living by design: The value of evaluation. American Journal of Preventive Medicine, 43(5 Suppl. 4), S267-S268.

16.

Lieberman

Golden

S. D.

Earp

J. A. L.

(2013). Structural approaches to health promotion: What do we need to know about policy and environmental change? Health Education & Behavior, 40, 520-525.

17.

Miles

M. B.

Huberman

A. M.

Saldaña

(2014). Qualitative data analysis: A methods sourcebook. Thousand Oaks, CA: Sage.

18.

Mockenhaupt

Woodrum

(2015). Developing evidence for structural approaches to build a culture of health: A perspective from the Robert Wood Johnson Foundation. Health Education & Behavior, 42(1 Suppl.), 15S–19S.

19.

Nichols

Ussery-Hall

Griffin-Blake

Easton

(2012). The evolution of the steps program, 2003-2010: Transforming the federal public health practice of chronic disease prevention. Preventing Chronic Disease, 9, E50.

20.

Patton

M. Q.

(2015). Qualitative research & evaluation methods (4th ed). Thousand Oaks, CA: Sage.

21.

Pettibone

K. G.

Friend

K. B.

Nargiso

J. E.

Florin

(2013). Evaluating environmental change strategies: Challenges and solutions. American Journal of Community Psychology, 51, 217-221.

22.

Schölmerich

V. L. N.

Kawachi

(2016). Translating the socio-ecological perspective into multilevel interventions: Gaps between theory and practice. Health Education & Behavior, 43, 17-20.

23.

Soler

R. E.

Whitten

K. L.

Ottley

P. G.

(2014). Communities putting prevention to work: Local evaluation of community-based strategies designed to make healthy living easier. Preventive Medicine, 67(Suppl. 1), S1-S3.

24.

Stevens

G. W.

(2013). Toward a process-based approach of conceptualizing change readiness. Journal of Applied Behavioral Science, 49, 333-360.

25.

(2013). Toward a definition of evaluative thinking. Retrieved from https://escholarship.org/uc/item/26t5x5f6

26.

Yin

R. K.

(2009). Case study research: Design and methods (5th ed). Thousand Oaks, CA: Sage. doi:10.1097/FCH.0b013e31822dda9e