Abstract
Collective impact (CI) is a structured approach that helps drive multi-sector collaborations to address social problems through systems changes. While the CI approach is gaining popularity, practitioners experience challenges in evaluating its implementation and intended outcomes. We conducted a systematic scoping review to understand evaluation methods specific to CI initiatives, identify challenges or limitations with these evaluations, and provide recommendations for the design of CI evaluations. Eighteen studies met the inclusion criteria. Process evaluations were the most frequently used evaluation design. Most studies collected cross-sectional data to evaluate their efforts. The complexity of CI was most frequently cited as the greatest evaluation challenge. Study recommendations primarily focused on improvements during the evaluation planning phase. Taking careful consideration in the planning of CI evaluations, developing context-specific data collection methods, and communicating results intentionally and effectively could prove useful to sufficiently capture and assess this systems-level approach to address social problems.
Keywords
Introduction
Frequently within the social sector, multi-sector collaborations are used to break down organizational silos and tackle complex, community-level challenges. These collaborations, often labeled as community coalitions or partnerships, have been used for decades to address various concerns of a given community (Goodman, et al., 1996; Kubisch, 1997; McLeroy, et al., 1994). By definition, the term “community coalition” refers to a group of community partners and/or organizations who work together to achieve a common goal (Butterfoss & Kegler, 2009; Granner & Sharpe, 2004). This collaborative action often aims to address disparities that emerge from existing social and economic differences within a community (Giachello et al., 2003; Wolff et al., 2016). While coalitions have demonstrated success in bringing together diverse perspectives, evidence suggests that a lack of centralized management and structure in coalitions may present limitations to overall effectiveness (Valente, Chou, & Pentz, 2007).
The collective impact (CI) approach helps address this limitation by proposing a centralized coalition strategy guided by a structured framework (Kania & Kramer, 2011). CI is a systematic framework that describes how a methodical, multi-sector approach can be used to address complex problems (Kania & Kramer, 2011). The centralized approach of the framework helps offset recurrent issues, such as a lack of participation and commitment by members (Valente et al., 2007). Rather than organizations working in silos, the framework also supports collaboration towards an achievable, common goal (Hanleybrown, Kania, & Kramer, 2012; Kania & Kramer, 2011).
CI is implemented using a set of five distinct conditions. These conditions include: a common agenda, shared measurement systems, mutually reinforcing activities, continuous communication, and backbone support organization(s) (Kania & Kramer, 2011, 2013). Within CI, a common agenda is established when CI members share a vision and common understanding of a problem, then engage in a joint approach to solve the problem. Shared measurement refers to consistent data collection across members to ensure efforts remain aligned and everyone is held accountable. Mutually reinforcing activities involve the coordination of complementary activities through a mutually reinforcing plan of action. Relatedly, continuous communication involves consistent, open member communication conducted in a way to build trust, mutual objectives, and common motivation. Finally, CI has at least one organizational member who serves as a backbone support organization, which provides staff focused on the coordination of participating organizations (Kania & Kramer, 2011).
CI has gained popularity over the last decade, causing many communities to begin to explore the benefits of using this approach in community development efforts (Flood et al., 2015). CI models in practice have implemented interventions focused on obesity prevention, substance use, teenage pregnancy, and other topics (Flood et al., 2015). For example, Shape Up Somerville was cited as a successful example of CI implementation in the seminal article on CI (Kania & Kramer, 2011). This CI initiative was formed in 2003 to address the issue of obesity in children in Somerville, Massachusetts by engaging schools, city governments, community groups, businesses, and other stakeholders (Collective Impact Forum, 2013). Through shared programming, infrastructure improvements, and policy enactment, the initiative made contributions to the improvement of health outcomes (i.e., reduction in children's BMI scores) along with positive financial return on investments (Coffield et al., 2019; Economos et al., 2013). However, given the nature of the community-based intervention, challenges in evaluating the initiative were cited, such as difficulty to randomize groups and the modest amount of attrition as children moved from the community (Coffield et al., 2019; Economos et al., 2013).
As described, evaluation has been a long-standing challenge of CI. CI is complex and has a dynamic nature, leading to challenges in establishing a set of best evaluation practices that could be applicable across the initiative (Preskill et al., 2014). Traditionally, program evaluations focus on isolating specific effects of a single intervention; however, CI involves multiple projects where stakeholders are faced with challenges in the attribution of specific outcomes, leading to complications with funding and contractual obligations (Smart, 2017). Furthermore, various community stakeholders (e.g., businesses, government agencies, community-based organizations, individuals) with differing approaches and goals are involved. Developing mutual agreements in outcomes, goals, and measurement tools that are meaningful for stakeholders can prove challenging and can affect the establishment of a shared evaluation approach (Cabaj, 2014). This process of establishing cross-sector coordination for evaluation can create time delays and can be costly and resource-intensive (Cabaj, 2014). Additionally, since CI projects often involve large communities, the dynamics of individuals moving in and out of the community and other community or policy level changes can complicate continuous data collection, especially when attempting to take into account contextual factors that influence the initiative's success (Kania & Kramer, 2011). Due to its complexity, developmental, realist, formative, and summative evaluation activities are essential to ensure that CI is being carried out correctly and produces its intended outcomes (Preskill, Parkhurst, & Juster, 2014). Further, ongoing monitoring can provide stakeholders with information to make changes and improve efficiency.
Prior to establishing the CI framework, an important piece of work for building and evaluating community change initiatives was proposed by the Aspen Institute in their three-part series, Voices from the Field (Connell et al., 1995; Kubisch, 1997; Kubisch et al., 2010). This work has served as a basis for evaluating community change efforts; however, much has changed in the landscape and capacity to accomplish systems changes within a community. Drawing from these seminal pieces and years of community change work, attempts have been made to establish a set of principles for evaluating CI. For example, shortly after the seminal article on CI was published, the CI Forum produced a set of guides for evaluating CI (Preskill et al., 2014) and summarized this information later that year (Preskill et al., 2014). Within the same year, Cabaj (2014) published an article that included five rules for evaluating CI, emphasizing strategic learning, use of multiple evaluation designs, careful consideration toward shared measurement, ascertainment of intended and unintended consequences, and a focus on contribution analysis as opposed to attribution. While informative, numerous articles have been published since then critiquing the original CI framework and suggesting the need for additional conditions to guide CI work and evaluation (Christens & Inzeo, 2015; LeChasseur, 2016; Weaver, 2014; Wolff et al., 2016). For example, LeChasseur (2016) conducted a case analysis of two equity-focused partnerships and found that there are potential flaws within the original CI framework caused by structural racism and other institutionalized inequalities that can compromise the initiative's ability to address community-wide challenges. The previous year, Christens and Inzeo (2015) concluded similar findings, stating that the roles that power can play in collaborative formation, maintenance, and the achievement of goals is an important consideration for community efforts. As demonstrated, the evidence-base for CI is still evolving and developing, and this approach has not been established long enough to settle on a framework that can be holistically tested (Moore et al., 2014; Smart, 2017).
Practitioners need an updated, more informed evaluation framework for CI to reflect these expanded conditions and ensure that it is implemented successfully with maximum community benefit. Therefore, the purpose of this study is to explore the existing peer-reviewed literature to understand methods of evaluating CI in practice, including challenges or limitations with these evaluations, and offer a list of recommendations for the design and implementation of CI evaluations. The study sought to answer the following research questions:
What methods are used to evaluate CI initiatives? What are the challenges of carrying out CI evaluations? What recommendations have been proposed for CI evaluations?
Methods
To systematically explore existing CI evaluation work within the literature, this study followed the Preferred Reporting Items for Systematic Review and Meta-Analysis statement to guide the reporting of methods and findings (Moher et al., 2009). Systematic methods of review have been used to synthesize studies that evaluate complex interventions, similar to CI (Shepperd et al., 2009). Thus, a systematic scoping review was deemed appropriate as the study sought to bring together emerging evidence over a particular period of time (Peters et al., 2015). A Cochrane technology platform, Covidence Systematic Review Software, was used to manage the review process.
Search Strategy
The Sample, Phenomenon of Interest, Design, Evaluation, Research (SPIDER) type framework was used to develop the search strategy and search terms to specifically identify relevant studies (Cooke, Smith, & Booth, 2012). The framework's definitions and description of each component as defined by this study can be found in Table 1. The first and second authors, both with PhD-level training, conducted a review of published literature using three databases in December 2020: Web of Science, MEDLINE, and Google Scholar. Search terms included “collective impact” AND “evaluation” OR “assessment” and included articles in the English language that were published between 2011 and 2020. This time frame was chosen to capture all articles that have been published since the original Kania and Kramer (2011) article describing CI.
Sample, Phenomenon of Interest, Design, Evaluation, Research (SPIDER) Components Associated With Search Strategy (Cooke et al., 2012).
Article Selection
Results were combined into Covidence, and duplicates were removed. The first two authors independently screened the included articles first by title and abstract, then followed with a full-text review before extracting data. The SPIDER tool conceptualized eligibility criteria as outlined in Table 1 (Cooke et al., 2012). Selection criteria were set to include articles that specifically focused on assessing or evaluating a CI initiative; used quantitative, qualitative, or mixed methods study designs; reported on experiences, perceptions, and/or outcomes; and reported either longitudinal or cross-sectional data as determined by the study purpose. Articles that consisted solely of published abstracts or described a collaborative model or CI initiative, but did not include an assessment or evaluation of a CI initiative, were excluded. At each stage of the screening, two reviewers independently screened articles using the Covidence platform to guide the process. The agreement score was greater than 90% for each stage; the reviewers discussed the remaining discrepancies to reach a consensus and be in full agreement before proceeding to the next stage. For a visual depiction of the study selection process (see Figure 1).

Study selection procedure.
Data Extraction and Management
Included articles were analyzed using a data extraction form that was developed based on features of CI initiatives (e.g., CI collaboration description, problem addressed) and evaluation studies (e.g., evaluation type, evaluation data sources). The two reviewers then piloted the extraction form using a random sample of three articles. Other relevant categories were determined by consensus of both reviewers, and the form was changed based on the findings from the pilot test. The two reviewers then independently extracted data using a data extraction table within Covidence to organize the important study characteristics. The variables chosen to be extracted into the matrix include title, citation/author information, year of publication, the aim of the study, evaluation design, whether the study was conducted longitudinally or cross-sectionally, theoretical framework used, study funding sources, evaluation data sources, whether the study was conducted by an internal or external evaluator, intervention description, key findings, implications for practice, challenges/limitations, and study recommendations. The reviewers initially reached an agreement score of 94% after independent extraction, meaning they agreed on the majority of the data extracted, but had few differences, particularly related to key findings and implications for practice. Reviewers met to discuss discrepancies and referenced articles, as needed, to reach a full agreement.
Data Synthesis
Data was exported from Covidence into a Microsoft Excel file, and the main article characteristics were synthesized across articles. Quantitative summaries of characteristics and methodological aspects of studies were determined. Challenges or limitations of studies and recommendations were analyzed using thematic analysis based on the study's research questions but were also sensitive to topics that emerged from the data (Braun & Clarke, 2012). The first and second authors carried this process out by first coding specific challenges/limitations identified within each study, then categorizing these challenges/limitations into specific themes and quantifying them to determine the themes that occurred the most frequently. The same process was followed for the study recommendations.
Results
The initial search yielded 150 published articles, see Figure 1. After the removal of 46 duplicates, the remaining 104 articles were screened based on title and abstract, resulting in 46 articles for full-text review. After full-text screening based on the inclusion criteria, 18 studies were included for analysis.
Characteristics of Included Studies
The characteristics of the eighteen articles included in this review are summarized in Table 2. The majority of the evaluation studies occurred in the United States (n = 14; 77.8%); two were conducted in Australia (n = 2; 11.1%), and two were conducted in Canada (n = 2; 11.1%). The CI initiatives focused on varied social problems, with the highest number of studies addressing obesity (n = 5; 27.8%), followed by education (n = 4; 22.2%), the healthcare system (n = 2; 11.0%), healthy living (n = 2; 11.0%). The remaining initiatives focused on socioeconomic status (n = 1; 5.6%), teen pregnancy (n = 1; 5.6%), substance abuse in the juvenile justice system (n = 1; 5.6%), child well-being (n = 1; 5.6%), and maternal and child health (n = 1; 5.6%). Most studies were published between 2018–2020 (n = 12; 66.7%), five were published between 2015 and 2017 (n = 5; 27.7%), and one was published between 2011–2014 (n = 1; 5.6%). More than one-third of the studies were funded by government agencies (n = 7; 38.8%) at either the state or national level, followed by private foundations (n = 6; 33.3%) and academic institutions (n = 1; 5.6%). One study was funded by a government agency and private foundation (n = 1; 5.6%). Three studies either stated that they did not receive funding for their research or did not state otherwise (n = 3; 16.7%).
Characteristics of Included Studies.
Methodological Aspects of Studies
One of the key aims of this study was to understand how the evaluation of CI initiatives is carried out in practice. Most studies were carried out by an external evaluator (n = 14; 77.8%), and the rest were conducted internally (n = 4; 22.2%), with one evaluation conducted internally with external support. The majority of evaluations were process evaluations (n = 12; 66.7%); few were outcome evaluations (n = 4; 22.1%). One study gathered data for both formative and summative evaluation (n = 1; 5.6%), and another study referred to their evaluation design as a developmental evaluation (n = 1; 5.6%). A little more than half of the studies used cross-sectional data (n = 10; 55.6%), and the remaining collected data longitudinally (n = 8; 44.4%). Eight of the evaluations had a mixed-methods study design (n = 8; 44.5%), six were quantitative (n = 6; 33.3%), and four were qualitative (n = 4; 22.2%). Most studies used surveys that allowed for both quantitative and qualitative responses; others utilized outcome measures as a form of analysis for quantitative studies; interviews and archival document reviews were the most common tools used for qualitative studies. Less than half of the studies used the CI framework as the basis for their evaluation (n = 8; 44.4%).
Challenges and Limitations With Evaluation
In addition to the methodological aspects of studies, we also explored the challenges and limitations faced by evaluators in assessing CI (see Table 3). The most cited challenge was related to the complexity of CI (n = 13), which made it difficult to determine evaluation measures, posed challenges in analyzing data in an impactful and meaningful way, and limited the ability to attribute systems-level changes while accounting for confounding results (e.g., Barata-Cavalcanti et al., 2020a, 2020b). Other challenges included the possibility of biased data (n = 4) due to self-reporting, social desirability, and the majority of data coming from a single organization or a small group of organizations (e.g., Gao et al., 2019); limitations at the organizational level (n = 4), such as resistance to negative feedback, the added burden to track relevant measures at the organizational level, and organizational focus on impact rather than process of implementing CI (e.g., Rorrer et al., 2019); limited data (n = 4) due to small sample sizes and/or incomplete responses (e.g., Jenkins et al., 2020); and time and cost limitations (n = 2; e.g., Brown, Rizzuto, & Singh, 2019).
Challenges, Limitations, and Recommendations Made by Included Studies.
Recommendations by Studies
Based on their experience with evaluating CI, the included study authors made recommendations for CI evaluation best practices (see Table 3). The most frequently noted recommendations were related to planning the evaluation (n = 16; e.g., Easterling, Mayfield Arnold, Jones, & Smart, 2013). These included the use of mixed methods for richer data, involving partners in evaluation development, understanding organizational funding models to use as a basis for developing measures, utilizing a developmental evaluation approach for co-learning, using an external evaluator, and a recommendation for funders to provide technical assistance to support CI evaluation. The studies also made methodological recommendations for data collection (n = 3; e.g., Salihu, Wilson, & Berry, 2018). These recommendations included conducting longitudinal studies to learn the long-term benefit of the collaborative, focusing on systems-level measures rather than individual-level measures, utilizing meaningful measures that relate to the desired community change, assessing both formative and summative aspects of CI, considering the assessment of short-term indirect effects of programs, and breaking down evaluation instruments in specific segments in accordance with the collaborative's structure. The final set of recommendations were related to communicating results (n = 5; e.g., Barata-Cavalcanti et al., 2020a, 2020b). These recommendations include presenting data anonymously, communicating with audiences that outcomes of CI are emergent and iterative, utilizing and communicating results to reinforce community ownership and guide future planning, and communicating results with simple language.
Discussion
As the CI approach is more frequently implemented to address social problems, it is important to develop a holistic, relevant framework for evaluating these initiatives. Thus, the purpose of this study was to (1) explore the ways in which CI initiatives have been evaluated, (2) identify challenges or limitations experienced when evaluating CI, and (3) synthesize recommendations from the literature for future practitioners interested in evaluating CI. We accomplished these objectives by conducting a systematic scoping review and found 18 articles that met our inclusion criteria and were published following the release of the seminal article on CI (Kania & Kramer, 2011).
The first step in establishing a CI initiative is developing a common agenda (Kania & Kramer, 2011). This shared vision not only drives the collaboration's work but also supports its evaluation by answering the questions, “what are we trying to accomplish?” and “how will we measure its success?” Many CI efforts struggle to answer the latter question given the complexity of multiple initiatives occurring simultaneously and can turn to previous efforts with similar common agendas to serve as a guideline of how they should structure their evaluation (Jenkins et al., 2020). Our search found that there was some variation in the common agenda or focus of each evaluated initiative, with the majority focusing on obesity reduction. This is not surprising given that the most well-known and earliest documented CI initiative is Shape Up Somerville, which focused on utilizing a systems approach to reduce childhood obesity in Somerville, Massachusetts (Kania & Kramer, 2011). This collaborative approach has served as a model for applying CI to a complex, social problem (Coffield et al., 2019). Furthermore, in the last decade, there has been a call for system approaches to address obesity within the literature as outlined in the special themed issue of the American Journal of Public Health and the 2011 and 2015 Lancet Series on Obesity (Kleinert & Horton, 2015; Mabry & Bures, 2014; Rutter, 2011). While the evaluation tools and lessons learned from obesity-focused initiatives can carry over to other complex problems, the peer-reviewed literature would benefit from CI initiatives focused on other social problems to serve as examples for future initiatives.
Another important aspect of the studies was the funding for the evaluation. Most studies were funded by government agencies. Within the CI literature, it has been noted that funding from government entities tend to create hierarchies within collaborative structures, and evaluative measures are likely influenced by grant requirements rather than the collaborative's activities (Sagrestano, Clay, & Finerman, 2018). This finding is not limited to government entities, but has also been cited by initiatives that receive financial support from other types of funders. In our review, one article described a CI initiative that a private foundation proposed to its funded partners, rather than the typical case of stakeholders approaching funders to support their initiative. Although the funder's efforts were well-intentioned, the authors discuss that this approach created power imbalances from the CI initiative's onset and led to counterproductive expectations in terms of accountability (Landers, Price, & Minyard, 2018). Another article noted that respondents resisted sharing negative feedback because the continuity of their programs relied on external funding (Barata-Cavalcanti et al., 2020b). Furthermore, the need to demonstrate the effectiveness of CI initiatives in a way that is appealing to funders is complicated by the limited timeframe (e.g., 1–3 years) of funding, restricting the ability to measure the true impact of CI in the long-term (Sagrestano et al., 2018). This can also place an added burden on collaboratives and smaller funders to provide the financial support to sustain these initiatives (Sagrestano et al., 2018). Due to the systems nature of CI, short evaluation periods are less likely to reveal the real impact of collaborative work, which takes many years to see evidence (Rorrer et al., 2019). Funders that choose to provide financial support for CI initiatives must take into account the complexity, time delays, and continuous learning that occur and must challenge their traditional assumptions, approaches, and role to better fit these efforts.
In looking at the methodological aspects of the studies, most evaluations were conducted by an external evaluator; this approach has been cited extensively within evaluation research as a way to avoid bias, protect the objectivity of evaluation, and minimize the influence of leadership and pressure to present evaluation findings in a favorable light (American Evaluation Association, 2010; Patton, 2008). For example, one study conducted by an internal evaluator noted a limitation of their findings was social desirability bias (i.e., the tendency to respond in a way that will be viewed as favorable to others) that resulted from the known relationship between the interviewer (i.e., evaluator) and one of the collaborative's steering group organizations (Jenkins et al., 2020). Moreover, as stated by another study in our review, the use of an external evaluator could minimize the burden on organizational partners to collect and analyze data, provide an objective approach to standardize the evaluation across organizations, and ensure quality control of data collection (Rorrer et al., 2019). This finding was reiterated by another study that found that an external evaluator provided accountability, benchmarks, and timelines for progress that furthered the collaborative's work beyond what they would have been able to achieve on their own (Landry, Collie-Akers, Foster, Pecha, & Abresch, 2020). Thus, while costlier, external evaluators are especially critical during the beginning and middle stages of CI initiatives to help support start-up evaluation efforts and provide an objective approach for continuous quality improvement.
Most of the included studies reported on formative evaluation results. While summative evaluations are useful in assessing the impact of interventions, formative evaluations are particularly useful to measure progress and are valuable during the early stages of developing and implementing new methods, practices, policies, and approaches (Scott et al., 2019). For example, one study noted that formative data were used to make adjustments and increase the collaboration's capacity (Brown et al., 2019). Given that CI is a relatively new, innovative approach to multi-sector collaboration, applying a formative evaluation framework could prove useful for rapid-cycle testing for program refinement (Scott et al., 2019). Similarly, another evaluation approach that is particularly relevant for CI initiatives, but was only applied in one of our included studies, is the use of a developmental evaluation. Developmental evaluations focus on strategic learning to understand the activities of a program or programs operating in a dynamic, new environment that encompasses complex interactions (Landers et al., 2018). This approach allows for frequent cycles of data collection and sense-making, requires a participatory effort with stakeholders, and was found to be a critical support for learning and adaption throughout the CI process (Landers et al., 2018). Lastly, a final evaluation approach that would be useful for CI evaluation, but was not applied by any of the included studies, is the use of a realist evaluation. Realist evaluations take into account an intervention's implementation context and aims to identify and explain the causal mechanisms that result in specific outcomes (Dalkin et al., 2015; Rhodes & Lancaster, 2019). Most importantly, the realist approach recognizes that complex interventions are relatively unpredictable, involve feedback loops, and undergo iterative adaptations (Bhaskar, 2008). One approach to applying a realist evaluation to CI is developing a causal loop diagram or system map of relevant factors that influence the common agenda within the community (Meadows, 2008); evaluators can use this visual depiction to assess specific mechanisms or relationships that give rise to observed behaviors or outcomes. Given that CI involves multiple moving parts, a realist approach could prove useful to teasing out the impact of specific mechanisms or interventions and could address some of the evaluation challenges with the complexity of CI. Overall, while moving through the beginning, middle, and mature stages of CI, it is important to adapt the evaluation approach (i.e., through developmental, realist, formative, and summative approaches) or include elements of each to assess the work of the collaboration.
Approximately half of the studies used a mixed methods approach for their evaluation design. While mixed methods can be costlier and possibly more burdensome, it has been cited as critical during planning and design stages of a project (Bamberger, Tarsilla, & Hesse-Biber, 2016). Applying this methodology during the beginning stages of an initiative can often anticipate and help address potential unintended consequences that may occur during implementation (Bamberger et al., 2016). Mixed methods also provide additional richness and context for the data collected (Barata-Cavalcanti et al., 2020a). In our review, one study that employed a mixed methods approach during year 2 of their CI effort found that the quantitative data collected provided insight into preliminary outcome effects (i.e., improved literacy scores) while the qualitative data helped clarify areas for improvement (e.g., lack of consistent and strategic communication; Smith et al., 2019). Investing in the collection of both quantitative and qualitative data through multiple sources, especially during the early phases, could prove useful for refining and steering the CI initiative.
The core conditions of CI served as a framework for the evaluation design and data collection tools in about half of the studies. While this is important and crucial to understanding the extent to which the CI framework was implemented and how it allowed the collaborative to achieve its goals, CI is highly context specific, and the way in which these definitions are understood and carried out by collaborative members may vary. From an evaluation perspective, it is important to tailor and redefine these core conditions to accurately assess their extent of implementation. One study identified their collaboration's mutually reinforcing activities as “partners [utilizing] a strategic planning process to detail activities and respective roles of each of the priority areas” (Meinen et al., 2016; p. 271). Having this systematic way of specifying activities and roles within the collaborative was found to be important for accountability and guiding evaluation activities, such as asset mapping and formative assessment during the initiative's early stages (Meinen et al., 2016). Furthermore, recent studies have cited additional elements that were not emphasized within the initial CI framework (Christens & Inzeo, 2015). Other contextual elements (e.g., program awareness, capacity, cultural norms) were incorporated in one of our studies to assess CI's individual initiatives; the authors found that the findings from these additional domains highlighted areas for further investment to achieve greater and more sustained impact (Barata-Cavalcanti et al., 2020b).
In synthesizing the methods, challenges, and recommendations proposed by this study we developed a list of seven recommendations to support the evaluation of future CI initiatives. It is important to note that these recommendations are not in any particular order and can be prioritized based on the evaluation needs of the CI initiative.
Draw on similar CI initiatives to support evaluation work. CI approaches address complex social issues and are constantly evolving and changing over time. Using other CI initiatives with a similar common agenda or contextual factors as an example in the evaluation planning phase and throughout could assist in identifying and capturing the complexity of these initiatives. Furthermore, as the CI framework is more frequently applied, validated tools to specifically evaluate these types of community collaborations are being developed and can easily be applied to emerging initiatives (e.g., Barata-Cavalcanti et al., 2020a, 2020b; Salihu et al., 2018). Target supportive funding for community-based intervention. Not only are community-based initiatives gaining popularity among practitioners, but these efforts are also capturing the attention of funders. For example, in 2010, the U.S. Department of Education released a funding opportunity, called the Promise Neighborhoods Initiative, to provide cradle-to-career services within distressed neighborhoods, calling for a comprehensive, community-level approach. Funding mechanisms that support community-based work may be better suited for CI evaluation and could also provide the technical assistance partners may need to carry out their evaluation (Blake-Lamb et al., 2018). CI initiatives should be intentional when applying for funding streams to ensure it coincides with the systems-level approach they are carrying out. Engage an external evaluator. An external evaluator can provide objectivity, technical experience and expertise, and a unique perspective from collaborative members to assess the breadth of the CI initiative's activities and identify areas for improvement (Jenkins et al., 2020; Landry et al., 2020; Rorrer et al., 2019). These individuals are important to engage, at minimum, during the evaluation planning and early implementation phases to provide a foundation of support for subsequent years and can be further engaged, as needed, once the collaborative develops their evaluation plan and shared measurement system. External evaluators skilled in complex, systems-level evaluation are particularly useful for capturing the complexity and dynamic nature of CI and should also be sought out. Use an adaptive evaluation approach. As emphasized, CI is complex and operates in a continuously dynamic environment. Adapting the evaluation approach (i.e., developmental, realist, formative, and summative) based on the current state of the initiative or incorporating elements of each approach is important to capture the dynamic nature to better understand how the initiative can be improved, the elements that contribute to its success, and how the initiative will be or has been able to “move the needle” for the social problem of interest (Bhaskar, 2008; Brown et al., 2019; Landers et al., 2018). Diversify data collection methods. Mixed methods approaches are especially important to capture both objective data (i.e., quantitative) and contextual information (i.e., qualitative). CI involves multiple moving parts and collecting rich, informative data can help drive the work, garner buy-in, and support the initiative's progress. Furthermore, CI evaluations should utilize multiple sources of data that are context-specific and address multiple ecological levels (Brown et al., 2019; Klaus & Saunders, 2016). For example, individual-level data that assess outcomes could be supported by community-level measures that evaluate how the collaborative's multiple activities concomitantly supported those outcomes (Shaw, Sweet, McBride, Adair, & Martin Ginis, 2019). Involve key stakeholders in all aspects of evaluation. The lack of engagement of context experts and those with lived experience is one of the criticisms of the CI framework and is a missed opportunity for evaluation (Christens & Inzeo, 2015). These committed stakeholders are not only important for evaluation planning to ensure the efficacy of the partnership is appropriately measured, but could also serve as data collection recruiters to address challenges with small sample sizes and poor response rates (Smith et al., 2019). Furthermore, the involvement of stakeholders within the evaluation could support relationship building and trust within the community as well as an appreciation for the work being done, further supporting the initiative's sustainability (Jenkins et al., 2020; Shaw et al., 2019). Communicate evaluation results in a relevant, useful way. CI evaluation is essential to improve processes and make adaptations to meet the needs of the target population. Identifying channels to communicate evaluation results in a relevant, nontechnical manner is important to ensure the audience understands findings and can utilize them in a productive way (e.g., strategic thinking and future planning; Easterling et al., 2013; Jenkins et al., 2020). Furthermore, transparency in terms of evaluation is key for trust building and helps in identifying what can reasonably be achieved within a given timeframe.
Limitations
We acknowledge that limitations are present in the current study. The first limitation includes the sources of literature searched; since authors only searched academic databases, evaluation studies within grey literature were potentially missed. Given that CI is typically carried out at the grassroots level, there could be white papers, technical reports, or funder reports that have been developed to describe methods for evaluating CI. However, the purpose of this review was to identify CI efforts from the literature that have undergone rigorous peer review to provide sound evidence for future CI initiatives. Additionally, although the authors utilized broad search terms, there is a possibility that we missed articles that did not appear using these terms. Lastly, because there are variations in the ways CI is implemented and evaluated, it was difficult to categorize relevant information from studies and produce a coherent set of recommendations for CI evaluations. This speaks to the complexity of CI and emphasizes the importance of developing CI evaluations that are context-specific. However, we believe the recommendations provided are generalizable across CI initiatives and could provide a foundation for practitioners involved in CI evaluation development.
Conclusions
Our review revealed a variety of approaches currently being used to evaluate CI. Given that CI is a relatively new framework for community collaboration, evaluators are confronted with the need to understand how to effectively capture and assess this systems-level approach to address social problems in a relevant, context-specific manner. Nevertheless, the recommendations provided in this article speak to some of the key aspects of evaluation that should be considered when assessing CI. Future studies could compare different evaluation approaches and explore their impact on CI to gain a better understanding of the most effective methods for evaluating CI.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
