Evaluating Collective Impact Initiatives: A Systematic Scoping Review

Introduction

Frequently within the social sector, multi-sector collaborations are used to break down organizational silos and tackle complex, community-level challenges. These collaborations, often labeled as community coalitions or partnerships, have been used for decades to address various concerns of a given community (Goodman, et al., 1996; Kubisch, 1997; McLeroy, et al., 1994). By definition, the term “community coalition” refers to a group of community partners and/or organizations who work together to achieve a common goal (Butterfoss & Kegler, 2009; Granner & Sharpe, 2004). This collaborative action often aims to address disparities that emerge from existing social and economic differences within a community (Giachello et al., 2003; Wolff et al., 2016). While coalitions have demonstrated success in bringing together diverse perspectives, evidence suggests that a lack of centralized management and structure in coalitions may present limitations to overall effectiveness (Valente, Chou, & Pentz, 2007).

The collective impact (CI) approach helps address this limitation by proposing a centralized coalition strategy guided by a structured framework (Kania & Kramer, 2011). CI is a systematic framework that describes how a methodical, multi-sector approach can be used to address complex problems (Kania & Kramer, 2011). The centralized approach of the framework helps offset recurrent issues, such as a lack of participation and commitment by members (Valente et al., 2007). Rather than organizations working in silos, the framework also supports collaboration towards an achievable, common goal (Hanleybrown, Kania, & Kramer, 2012; Kania & Kramer, 2011).

CI is implemented using a set of five distinct conditions. These conditions include: a common agenda, shared measurement systems, mutually reinforcing activities, continuous communication, and backbone support organization(s) (Kania & Kramer, 2011, 2013). Within CI, a common agenda is established when CI members share a vision and common understanding of a problem, then engage in a joint approach to solve the problem. Shared measurement refers to consistent data collection across members to ensure efforts remain aligned and everyone is held accountable. Mutually reinforcing activities involve the coordination of complementary activities through a mutually reinforcing plan of action. Relatedly, continuous communication involves consistent, open member communication conducted in a way to build trust, mutual objectives, and common motivation. Finally, CI has at least one organizational member who serves as a backbone support organization, which provides staff focused on the coordination of participating organizations (Kania & Kramer, 2011).

CI has gained popularity over the last decade, causing many communities to begin to explore the benefits of using this approach in community development efforts (Flood et al., 2015). CI models in practice have implemented interventions focused on obesity prevention, substance use, teenage pregnancy, and other topics (Flood et al., 2015). For example, Shape Up Somerville was cited as a successful example of CI implementation in the seminal article on CI (Kania & Kramer, 2011). This CI initiative was formed in 2003 to address the issue of obesity in children in Somerville, Massachusetts by engaging schools, city governments, community groups, businesses, and other stakeholders (Collective Impact Forum, 2013). Through shared programming, infrastructure improvements, and policy enactment, the initiative made contributions to the improvement of health outcomes (i.e., reduction in children's BMI scores) along with positive financial return on investments (Coffield et al., 2019; Economos et al., 2013). However, given the nature of the community-based intervention, challenges in evaluating the initiative were cited, such as difficulty to randomize groups and the modest amount of attrition as children moved from the community (Coffield et al., 2019; Economos et al., 2013).

As described, evaluation has been a long-standing challenge of CI. CI is complex and has a dynamic nature, leading to challenges in establishing a set of best evaluation practices that could be applicable across the initiative (Preskill et al., 2014). Traditionally, program evaluations focus on isolating specific effects of a single intervention; however, CI involves multiple projects where stakeholders are faced with challenges in the attribution of specific outcomes, leading to complications with funding and contractual obligations (Smart, 2017). Furthermore, various community stakeholders (e.g., businesses, government agencies, community-based organizations, individuals) with differing approaches and goals are involved. Developing mutual agreements in outcomes, goals, and measurement tools that are meaningful for stakeholders can prove challenging and can affect the establishment of a shared evaluation approach (Cabaj, 2014). This process of establishing cross-sector coordination for evaluation can create time delays and can be costly and resource-intensive (Cabaj, 2014). Additionally, since CI projects often involve large communities, the dynamics of individuals moving in and out of the community and other community or policy level changes can complicate continuous data collection, especially when attempting to take into account contextual factors that influence the initiative's success (Kania & Kramer, 2011). Due to its complexity, developmental, realist, formative, and summative evaluation activities are essential to ensure that CI is being carried out correctly and produces its intended outcomes (Preskill, Parkhurst, & Juster, 2014). Further, ongoing monitoring can provide stakeholders with information to make changes and improve efficiency.

Prior to establishing the CI framework, an important piece of work for building and evaluating community change initiatives was proposed by the Aspen Institute in their three-part series, Voices from the Field (Connell et al., 1995; Kubisch, 1997; Kubisch et al., 2010). This work has served as a basis for evaluating community change efforts; however, much has changed in the landscape and capacity to accomplish systems changes within a community. Drawing from these seminal pieces and years of community change work, attempts have been made to establish a set of principles for evaluating CI. For example, shortly after the seminal article on CI was published, the CI Forum produced a set of guides for evaluating CI (Preskill et al., 2014) and summarized this information later that year (Preskill et al., 2014). Within the same year, Cabaj (2014) published an article that included five rules for evaluating CI, emphasizing strategic learning, use of multiple evaluation designs, careful consideration toward shared measurement, ascertainment of intended and unintended consequences, and a focus on contribution analysis as opposed to attribution. While informative, numerous articles have been published since then critiquing the original CI framework and suggesting the need for additional conditions to guide CI work and evaluation (Christens & Inzeo, 2015; LeChasseur, 2016; Weaver, 2014; Wolff et al., 2016). For example, LeChasseur (2016) conducted a case analysis of two equity-focused partnerships and found that there are potential flaws within the original CI framework caused by structural racism and other institutionalized inequalities that can compromise the initiative's ability to address community-wide challenges. The previous year, Christens and Inzeo (2015) concluded similar findings, stating that the roles that power can play in collaborative formation, maintenance, and the achievement of goals is an important consideration for community efforts. As demonstrated, the evidence-base for CI is still evolving and developing, and this approach has not been established long enough to settle on a framework that can be holistically tested (Moore et al., 2014; Smart, 2017).

Practitioners need an updated, more informed evaluation framework for CI to reflect these expanded conditions and ensure that it is implemented successfully with maximum community benefit. Therefore, the purpose of this study is to explore the existing peer-reviewed literature to understand methods of evaluating CI in practice, including challenges or limitations with these evaluations, and offer a list of recommendations for the design and implementation of CI evaluations. The study sought to answer the following research questions:

What methods are used to evaluate CI initiatives?

Methods

To systematically explore existing CI evaluation work within the literature, this study followed the Preferred Reporting Items for Systematic Review and Meta-Analysis statement to guide the reporting of methods and findings (Moher et al., 2009). Systematic methods of review have been used to synthesize studies that evaluate complex interventions, similar to CI (Shepperd et al., 2009). Thus, a systematic scoping review was deemed appropriate as the study sought to bring together emerging evidence over a particular period of time (Peters et al., 2015). A Cochrane technology platform, Covidence Systematic Review Software, was used to manage the review process.

Search Strategy

The Sample, Phenomenon of Interest, Design, Evaluation, Research (SPIDER) type framework was used to develop the search strategy and search terms to specifically identify relevant studies (Cooke, Smith, & Booth, 2012). The framework's definitions and description of each component as defined by this study can be found in Table 1. The first and second authors, both with PhD-level training, conducted a review of published literature using three databases in December 2020: Web of Science, MEDLINE, and Google Scholar. Search terms included “collective impact” AND “evaluation” OR “assessment” and included articles in the English language that were published between 2011 and 2020. This time frame was chosen to capture all articles that have been published since the original Kania and Kramer (2011) article describing CI.

OPEN IN VIEWER

Table 1.

Sample, Phenomenon of Interest, Design, Evaluation, Research (SPIDER) Components Associated With Search Strategy (Cooke et al., 2012).

SPIDER Component	Definition	Study Description
Sample (S)	Unit of analysis included in the research study	Cross-sector collaborations applying the collective impact approach
Phenomenon of interest (P, I)	Encompasses behaviors, experiences, decisions, and interventions	Collective impact initiatives
Design (D)	Corresponding method(s) used	Named types of quantitative and qualitative data collection and analysis
Evaluation (E)	Evaluation outcomes that may include more subjective outcomes (views, attitudes, perspectives, etc.)	Experiences, perceptions, outcomes
Research type (R)	The research type searched for	Quantitative, qualitative, or mixed methods; Longitudinal or cross-sectional

Article Selection

Results were combined into Covidence, and duplicates were removed. The first two authors independently screened the included articles first by title and abstract, then followed with a full-text review before extracting data. The SPIDER tool conceptualized eligibility criteria as outlined in Table 1 (Cooke et al., 2012). Selection criteria were set to include articles that specifically focused on assessing or evaluating a CI initiative; used quantitative, qualitative, or mixed methods study designs; reported on experiences, perceptions, and/or outcomes; and reported either longitudinal or cross-sectional data as determined by the study purpose. Articles that consisted solely of published abstracts or described a collaborative model or CI initiative, but did not include an assessment or evaluation of a CI initiative, were excluded. At each stage of the screening, two reviewers independently screened articles using the Covidence platform to guide the process. The agreement score was greater than 90% for each stage; the reviewers discussed the remaining discrepancies to reach a consensus and be in full agreement before proceeding to the next stage. For a visual depiction of the study selection process (see Figure 1).

OPEN IN VIEWER

Figure 1.

Study selection procedure.

Results

The initial search yielded 150 published articles, see Figure 1. After the removal of 46 duplicates, the remaining 104 articles were screened based on title and abstract, resulting in 46 articles for full-text review. After full-text screening based on the inclusion criteria, 18 studies were included for analysis.

Characteristics of Included Studies

The characteristics of the eighteen articles included in this review are summarized in Table 2. The majority of the evaluation studies occurred in the United States (n = 14; 77.8%); two were conducted in Australia (n = 2; 11.1%), and two were conducted in Canada (n = 2; 11.1%). The CI initiatives focused on varied social problems, with the highest number of studies addressing obesity (n = 5; 27.8%), followed by education (n = 4; 22.2%), the healthcare system (n = 2; 11.0%), healthy living (n = 2; 11.0%). The remaining initiatives focused on socioeconomic status (n = 1; 5.6%), teen pregnancy (n = 1; 5.6%), substance abuse in the juvenile justice system (n = 1; 5.6%), child well-being (n = 1; 5.6%), and maternal and child health (n = 1; 5.6%). Most studies were published between 2018–2020 (n = 12; 66.7%), five were published between 2015 and 2017 (n = 5; 27.7%), and one was published between 2011–2014 (n = 1; 5.6%). More than one-third of the studies were funded by government agencies (n = 7; 38.8%) at either the state or national level, followed by private foundations (n = 6; 33.3%) and academic institutions (n = 1; 5.6%). One study was funded by a government agency and private foundation (n = 1; 5.6%). Three studies either stated that they did not receive funding for their research or did not state otherwise (n = 3; 16.7%).

OPEN IN VIEWER

Table 2.

Characteristics of Included Studies.

Author (Year)	Social Problem	Evaluation Design	Funding Source(s)	Methods	Data Source(s)	External vs Internal Evaluator
Addy and Dube (2018)	Obesity	Longitudinal, process evaluation	Private foundation	Qualitative	Review of organizational materials; Semi-structured interviews; participant observations	External
Barata-Cavalcanti et al. (2020a)	Obesity	Cross-sectional, process evaluation	Private Foundation	Mixed methods	Key Informant Interviews; Focus Groups; Direct Observations; Survey	External
Barata-Cavalcanti et al. (2020b)	Healthy living	Cross-sectional, process evaluation	Private Foundation	Mixed methods	Survey	External
Blake-Lamb et al. (2018)	Obesity	Longitudinal, outcome evaluation	Private Foundation, Government Agency	Quantitative	Clinical data; survey; process measures	Internal
Brown et al. (2019)	Low socioeconomic status	Cross-sectional, process evaluation	Unknown	Mixed methods	Survey; observation; qualitative feedback (Informal discussion)	External
Easterling et al. (2013)	Substance abuse in juvenile justice system	Longitudinal, outcome evaluation	Private foundation	Quantitative	Client screening tools; client tracking database	External
Gao et al. (2019)	Education	Longitudinal, outcome evaluation	Unknown	Quantitative	Country-level database of child reading scores	External
Homel et al. (2020)	Child well-being	Longitudinal, outcome evaluation	Government Agency	Quantitative	School-level database of student social-emotional well-being, student behavioral challenges, student numeracy and literacy	External
Jenkins et al. (2020)	Obesity	Cross-sectional, process evaluation	Government Agency	Qualitative	Interview	Internal
Klaus and Saunders (2016)	Teen pregnancy	Cross-sectional, process evaluation	Government Agency	Mixed methods	Partner Calls and Progress Reports; Survey	External
Landers et al. (2018)	Healthcare system	Longitudinal, developmental evaluation	Private Foundation	Qualitative	Document Review; Key Informant Interviews; Polling; Participant Observation; Feedback Loops	External
Landry et al. (2020)	Maternal and child health priorities	Longitudinal, Process evaluation	Government Agency	Quantitative	Survey	External
Meinen et al. (2016)	Obesity	Cross-sectional, process evaluation	Academic Institution	Mixed methods	Survey; Archival Document; Interviews	Internal
Rorrer et al. (2019)	Education	Longitudinal, process evaluation	Government Agency	Mixed methods	Survey, Narrative Reports, Score Cards	Internal with external support
Salihu et al. (2018)	Healthcare system	Cross-sectional, process evaluation	Government Agency	Quantitative	Survey	External
Shaw et al. (2019)	Healthy living	Cross-sectional, process and outcome evaluation	Government Agency	Mixed methods	Survey; Telephone Interview	External
Smith et al. (2019)	Education	Cross-sectional, process evaluation	Unknown	Mixed methods	Interviews; Dynamic Indicators of Basic Early Literacy Skills (DIBELS) testing	External
Zuckerman, Garrett, Sarver, and Huddleston-Cases (2020)	Education	Cross-sectional, process evaluation	Private Foundation	Qualitative	Observations; Interviews	External

Challenges and Limitations With Evaluation

In addition to the methodological aspects of studies, we also explored the challenges and limitations faced by evaluators in assessing CI (see Table 3). The most cited challenge was related to the complexity of CI (n = 13), which made it difficult to determine evaluation measures, posed challenges in analyzing data in an impactful and meaningful way, and limited the ability to attribute systems-level changes while accounting for confounding results (e.g., Barata-Cavalcanti et al., 2020a, 2020b). Other challenges included the possibility of biased data (n = 4) due to self-reporting, social desirability, and the majority of data coming from a single organization or a small group of organizations (e.g., Gao et al., 2019); limitations at the organizational level (n = 4), such as resistance to negative feedback, the added burden to track relevant measures at the organizational level, and organizational focus on impact rather than process of implementing CI (e.g., Rorrer et al., 2019); limited data (n = 4) due to small sample sizes and/or incomplete responses (e.g., Jenkins et al., 2020); and time and cost limitations (n = 2; e.g., Brown, Rizzuto, & Singh, 2019).

OPEN IN VIEWER

Table 3.

Challenges, Limitations, and Recommendations Made by Included Studies.

Author (Year)	Challenges/Limitations	Recommendation(s)
Addy and Dube (2018)	Complexity	Data collection methods Evaluation planning
Barata-Cavalcanti et al. (2020a)	Complexity Organizational Limitations	Communicating results Data collection methods
Barata-Cavalcanti et al. (2020b)	Biases Complexity Limited data	Data collection methods
Blake-Lamb et al. (2018)	Complexity	None
Brown et al. (2019)	Time and cost limitations	Evaluation pPlanning Data collection methods
Easterling et al. (2013)	None	Evaluation planning Communicating results
Gao et al. (2019)	Complexity Biases	Data collection methods
Homel et al. (2020)	Limited data Organizational limitations Time and cost limitations	Evaluation planning Data collection methods
Jenkins et al. (2020)	Limited data Biases	Evaluation planning Communicating results
Klaus and Saunders (2016)	None	Data collection methods
Landers et al. (2018)	Biases	Evaluation planning
Landry et al. (2020)	None	Evaluation planning
Meinen et al. (2016)	Complexity	None
Rorrer et al. (2019)	Complexity Organizational limitations Time and cost limitations	Data collection methods Evaluation planning Communicating results
Salihu et al. (2018)	Complexity	Data collection methods
Shaw et al. (2019)	Organizational limitations	Data collection methods Evaluation planning
Smith et al. (2019)	Time and cost limitations	Evaluation planning Data collection methods
Zuckerman et al. (2020)	Complexity Time and cost limitations	Evaluation planning Data collection methods

Discussion

As the CI approach is more frequently implemented to address social problems, it is important to develop a holistic, relevant framework for evaluating these initiatives. Thus, the purpose of this study was to (1) explore the ways in which CI initiatives have been evaluated, (2) identify challenges or limitations experienced when evaluating CI, and (3) synthesize recommendations from the literature for future practitioners interested in evaluating CI. We accomplished these objectives by conducting a systematic scoping review and found 18 articles that met our inclusion criteria and were published following the release of the seminal article on CI (Kania & Kramer, 2011).

The first step in establishing a CI initiative is developing a common agenda (Kania & Kramer, 2011). This shared vision not only drives the collaboration's work but also supports its evaluation by answering the questions, “what are we trying to accomplish?” and “how will we measure its success?” Many CI efforts struggle to answer the latter question given the complexity of multiple initiatives occurring simultaneously and can turn to previous efforts with similar common agendas to serve as a guideline of how they should structure their evaluation (Jenkins et al., 2020). Our search found that there was some variation in the common agenda or focus of each evaluated initiative, with the majority focusing on obesity reduction. This is not surprising given that the most well-known and earliest documented CI initiative is Shape Up Somerville, which focused on utilizing a systems approach to reduce childhood obesity in Somerville, Massachusetts (Kania & Kramer, 2011). This collaborative approach has served as a model for applying CI to a complex, social problem (Coffield et al., 2019). Furthermore, in the last decade, there has been a call for system approaches to address obesity within the literature as outlined in the special themed issue of the American Journal of Public Health and the 2011 and 2015 Lancet Series on Obesity (Kleinert & Horton, 2015; Mabry & Bures, 2014; Rutter, 2011). While the evaluation tools and lessons learned from obesity-focused initiatives can carry over to other complex problems, the peer-reviewed literature would benefit from CI initiatives focused on other social problems to serve as examples for future initiatives.

Another important aspect of the studies was the funding for the evaluation. Most studies were funded by government agencies. Within the CI literature, it has been noted that funding from government entities tend to create hierarchies within collaborative structures, and evaluative measures are likely influenced by grant requirements rather than the collaborative's activities (Sagrestano, Clay, & Finerman, 2018). This finding is not limited to government entities, but has also been cited by initiatives that receive financial support from other types of funders. In our review, one article described a CI initiative that a private foundation proposed to its funded partners, rather than the typical case of stakeholders approaching funders to support their initiative. Although the funder's efforts were well-intentioned, the authors discuss that this approach created power imbalances from the CI initiative's onset and led to counterproductive expectations in terms of accountability (Landers, Price, & Minyard, 2018). Another article noted that respondents resisted sharing negative feedback because the continuity of their programs relied on external funding (Barata-Cavalcanti et al., 2020b). Furthermore, the need to demonstrate the effectiveness of CI initiatives in a way that is appealing to funders is complicated by the limited timeframe (e.g., 1–3 years) of funding, restricting the ability to measure the true impact of CI in the long-term (Sagrestano et al., 2018). This can also place an added burden on collaboratives and smaller funders to provide the financial support to sustain these initiatives (Sagrestano et al., 2018). Due to the systems nature of CI, short evaluation periods are less likely to reveal the real impact of collaborative work, which takes many years to see evidence (Rorrer et al., 2019). Funders that choose to provide financial support for CI initiatives must take into account the complexity, time delays, and continuous learning that occur and must challenge their traditional assumptions, approaches, and role to better fit these efforts.

In looking at the methodological aspects of the studies, most evaluations were conducted by an external evaluator; this approach has been cited extensively within evaluation research as a way to avoid bias, protect the objectivity of evaluation, and minimize the influence of leadership and pressure to present evaluation findings in a favorable light (American Evaluation Association, 2010; Patton, 2008). For example, one study conducted by an internal evaluator noted a limitation of their findings was social desirability bias (i.e., the tendency to respond in a way that will be viewed as favorable to others) that resulted from the known relationship between the interviewer (i.e., evaluator) and one of the collaborative's steering group organizations (Jenkins et al., 2020). Moreover, as stated by another study in our review, the use of an external evaluator could minimize the burden on organizational partners to collect and analyze data, provide an objective approach to standardize the evaluation across organizations, and ensure quality control of data collection (Rorrer et al., 2019). This finding was reiterated by another study that found that an external evaluator provided accountability, benchmarks, and timelines for progress that furthered the collaborative's work beyond what they would have been able to achieve on their own (Landry, Collie-Akers, Foster, Pecha, & Abresch, 2020). Thus, while costlier, external evaluators are especially critical during the beginning and middle stages of CI initiatives to help support start-up evaluation efforts and provide an objective approach for continuous quality improvement.

Most of the included studies reported on formative evaluation results. While summative evaluations are useful in assessing the impact of interventions, formative evaluations are particularly useful to measure progress and are valuable during the early stages of developing and implementing new methods, practices, policies, and approaches (Scott et al., 2019). For example, one study noted that formative data were used to make adjustments and increase the collaboration's capacity (Brown et al., 2019). Given that CI is a relatively new, innovative approach to multi-sector collaboration, applying a formative evaluation framework could prove useful for rapid-cycle testing for program refinement (Scott et al., 2019). Similarly, another evaluation approach that is particularly relevant for CI initiatives, but was only applied in one of our included studies, is the use of a developmental evaluation. Developmental evaluations focus on strategic learning to understand the activities of a program or programs operating in a dynamic, new environment that encompasses complex interactions (Landers et al., 2018). This approach allows for frequent cycles of data collection and sense-making, requires a participatory effort with stakeholders, and was found to be a critical support for learning and adaption throughout the CI process (Landers et al., 2018). Lastly, a final evaluation approach that would be useful for CI evaluation, but was not applied by any of the included studies, is the use of a realist evaluation. Realist evaluations take into account an intervention's implementation context and aims to identify and explain the causal mechanisms that result in specific outcomes (Dalkin et al., 2015; Rhodes & Lancaster, 2019). Most importantly, the realist approach recognizes that complex interventions are relatively unpredictable, involve feedback loops, and undergo iterative adaptations (Bhaskar, 2008). One approach to applying a realist evaluation to CI is developing a causal loop diagram or system map of relevant factors that influence the common agenda within the community (Meadows, 2008); evaluators can use this visual depiction to assess specific mechanisms or relationships that give rise to observed behaviors or outcomes. Given that CI involves multiple moving parts, a realist approach could prove useful to teasing out the impact of specific mechanisms or interventions and could address some of the evaluation challenges with the complexity of CI. Overall, while moving through the beginning, middle, and mature stages of CI, it is important to adapt the evaluation approach (i.e., through developmental, realist, formative, and summative approaches) or include elements of each to assess the work of the collaboration.

Approximately half of the studies used a mixed methods approach for their evaluation design. While mixed methods can be costlier and possibly more burdensome, it has been cited as critical during planning and design stages of a project (Bamberger, Tarsilla, & Hesse-Biber, 2016). Applying this methodology during the beginning stages of an initiative can often anticipate and help address potential unintended consequences that may occur during implementation (Bamberger et al., 2016). Mixed methods also provide additional richness and context for the data collected (Barata-Cavalcanti et al., 2020a). In our review, one study that employed a mixed methods approach during year 2 of their CI effort found that the quantitative data collected provided insight into preliminary outcome effects (i.e., improved literacy scores) while the qualitative data helped clarify areas for improvement (e.g., lack of consistent and strategic communication; Smith et al., 2019). Investing in the collection of both quantitative and qualitative data through multiple sources, especially during the early phases, could prove useful for refining and steering the CI initiative.

The core conditions of CI served as a framework for the evaluation design and data collection tools in about half of the studies. While this is important and crucial to understanding the extent to which the CI framework was implemented and how it allowed the collaborative to achieve its goals, CI is highly context specific, and the way in which these definitions are understood and carried out by collaborative members may vary. From an evaluation perspective, it is important to tailor and redefine these core conditions to accurately assess their extent of implementation. One study identified their collaboration's mutually reinforcing activities as “partners [utilizing] a strategic planning process to detail activities and respective roles of each of the priority areas” (Meinen et al., 2016; p. 271). Having this systematic way of specifying activities and roles within the collaborative was found to be important for accountability and guiding evaluation activities, such as asset mapping and formative assessment during the initiative's early stages (Meinen et al., 2016). Furthermore, recent studies have cited additional elements that were not emphasized within the initial CI framework (Christens & Inzeo, 2015). Other contextual elements (e.g., program awareness, capacity, cultural norms) were incorporated in one of our studies to assess CI's individual initiatives; the authors found that the findings from these additional domains highlighted areas for further investment to achieve greater and more sustained impact (Barata-Cavalcanti et al., 2020b).

In synthesizing the methods, challenges, and recommendations proposed by this study we developed a list of seven recommendations to support the evaluation of future CI initiatives. It is important to note that these recommendations are not in any particular order and can be prioritized based on the evaluation needs of the CI initiative.

Draw on similar CI initiatives to support evaluation work. CI approaches address complex social issues and are constantly evolving and changing over time. Using other CI initiatives with a similar common agenda or contextual factors as an example in the evaluation planning phase and throughout could assist in identifying and capturing the complexity of these initiatives. Furthermore, as the CI framework is more frequently applied, validated tools to specifically evaluate these types of community collaborations are being developed and can easily be applied to emerging initiatives (e.g., Barata-Cavalcanti et al., 2020a, 2020b; Salihu et al., 2018).

Limitations

We acknowledge that limitations are present in the current study. The first limitation includes the sources of literature searched; since authors only searched academic databases, evaluation studies within grey literature were potentially missed. Given that CI is typically carried out at the grassroots level, there could be white papers, technical reports, or funder reports that have been developed to describe methods for evaluating CI. However, the purpose of this review was to identify CI efforts from the literature that have undergone rigorous peer review to provide sound evidence for future CI initiatives. Additionally, although the authors utilized broad search terms, there is a possibility that we missed articles that did not appear using these terms. Lastly, because there are variations in the ways CI is implemented and evaluated, it was difficult to categorize relevant information from studies and produce a coherent set of recommendations for CI evaluations. This speaks to the complexity of CI and emphasizes the importance of developing CI evaluations that are context-specific. However, we believe the recommendations provided are generalizable across CI initiatives and could provide a foundation for practitioners involved in CI evaluation development.

Conclusions

Our review revealed a variety of approaches currently being used to evaluate CI. Given that CI is a relatively new framework for community collaboration, evaluators are confronted with the need to understand how to effectively capture and assess this systems-level approach to address social problems in a relevant, context-specific manner. Nevertheless, the recommendations provided in this article speak to some of the key aspects of evaluation that should be considered when assessing CI. Future studies could compare different evaluation approaches and explore their impact on CI to gain a better understanding of the most effective methods for evaluating CI.