Lessons From the Social Innovation Fund: A Tested Evaluation Technical Assistance Approach to Strengthening Evaluation Practice and Building a Body of Evidence

Abstract

The Social Innovation Fund (SIF), a program of the Corporation for National and Community Service that received funding from 2010 to 2016, is one of a set of tiered evidence initiatives that was designed and implemented at the federal level during President Obama’s administration. The key objectives of the initiative were to (1) invest in promising interventions that address social and community challenges and grow their impact and (2) invest in evaluation and capacity building in order to support the development and use of rigorous evidence to measure the effectiveness of each funded intervention (i.e., to “move the evidence needle”) and inform decision making. The SIF proved successful in strengthening and sustaining the capacity of its implementing partners to conduct rigorous evaluations when put through a robust impact evaluation of its own at the national level. It has also spurred high-quality local evaluations that are building knowledge and a body of evidence across the supported program models to inform practice. The SIF’s evaluation technical assistance program was critical to its success, and as such, its design and approach holds interesting lessons for the larger field. This article discusses the structure and key features of the SIF as a grant making model, its evaluation requirements, and embedded approach and process for evaluation capacity building and the delivery of technical assistance, the tools and resources that it generated to support its goals, the evidence supporting its success, and how those lessons can inform other organizations and initiatives.

Keywords

rigorous evaluation evaluation technical assistance capacity building evidence-based grant making Social Innovation Fund tiered evidence

Introduction

During the last 2 decades, there has been a growing interest in and demand for evidence-based solutions to persistent social problems (Haskins & Margolis, 2014; Nussle & Orszag, 2014). In practice, this has meant an increased emphasis and reliance on evidence produced through evaluations designed to assess causal relationships between social programs and improvements in the negative conditions and outcomes that often affect vulnerable populations in low-income communities across the country (Maynard et al., 2016). Within the context of the federal government, tiered evidence initiatives have been an important strategy for incentivizing the use of causal evidence in the design and implementation of social programs (Government Accountability Office [GAO], 2016; Haskins & Baron, 2011; Haskins & Margolis, 2014; Lester, 2015). More specifically, these initiatives use evidence to inform grant making, guiding decisions about which applicants receive funding and the dollar value of grant awards. Evidence is also used to assess implementation of social programs, document their impacts, and inform future funding priorities (GAO, 2016). In addition, all of these federal initiatives have some form of embedded evaluation technical assistance (ETA) component in order to make them operational. As such, they offer interesting case examples in terms of how they not only build the evidence base for social programs but also strengthen evaluation capacity and practice by providing technical assistance.

The Corporation for National and Community Service’s (CNCS) Social Innovation Fund (SIF)

The SIF, administered by the CNCS, is one of the federal tiered evidence initiatives that offers an interesting case example of ETA. The SIF received government funding from 2010 to 2016. Annual Congressional appropriation for the initiative was US$50 million, except for 2014, when the program received a US$70 million appropriation. According to CNCS program and administrative data, SIF generated over US$1 billion in investments for evidence-based community solutions in 46 states and Washington, DC. Private and local funders multiplied the US$352 million direct federal investment in awards, providing more than US$707 million in additional funding to expand evidence-based programming. These investments provided support to 553 nonprofit organizations in close to 2,000 service locations across the country and provided social programming to more than 857,000 individuals and their families.

The SIF was committed to investing in innovative and promising solutions that had demonstrated some evidence of effectiveness through prior evaluation studies. For the SIF, “social innovation” meant “new ways to solve old problems that are faster, cost-effective, data-driven, and lead to better results for the public good.” SIF made grant awards to experienced intermediary organizations (“grantees”) with strong skills and track records of success in selecting, validating, and growing or scaling high-performing programs; using evaluation findings for making programmatic decisions; and providing training and technical assistance to their past grantees. Funded organizations possessed the capacity to manage a federal grant and engage with third-party evaluators. Organizations were funded to implement social programs designed to improve education, economic, and health outcomes for vulnerable communities. Funded organizations were required to conduct third-party evaluations of the sponsored social programs with the goal of advancing the body of evidence.

The SIF had two distinct grant programs: the SIF Classic Grant Program and the SIF Pay for Success Grant Program.² The focus of this article is on the SIF Classic Grant Program that implemented six rounds of grant making starting in 2010 and awarded a total of 39 grants. These SIF grantees were selected through an open, competitive process. The SIF grants ranged from US$1 million to US$10 million per year, for up to 5 years. Table 1 summarizes the number of awards by year.

Table 1.

SIF Classic Grants by Award Year.

Award Year	Number of Grants to SIF Grantees
2010	11
2011	5
2012	4
2014	7
2015	8
2016	4
Total	39

Note. No new awards were made in 2013. SIF = Social Innovation Fund.

Table 2 lists all of the SIF grantees, their projects, and number of awards.

Table 2.

SIF Classic Grantees, Their Projects, and Number of Awards.

Grantee, Award Year(s), and Project Name(s)	Number of Awards	Grantee, Award Year(s), and Project Name(s)	Number of Awards
AARP Foundation (2014, 2015) Back to Work 50+: Women’s Economic Stability Initiative Scaling Reading Success	2	Mile High United Way (2011) Early Literacy Social Innovation Fund	1
AIDS United (2010) Access to Care Initiative (A2C)	1	Missouri Foundation for Health (2010) Strategic Innovation in Missouri	1
Annie E. Casey Foundation (2015) Learn and Earn to Achieve Potential (LEAP)	1	Nebraska Children and Families Foundation (2015) Connected Youth Initiative	1
Boston Foundation (2014) Boston Coaching for Completion Initiative	1	New Profit Inc. (2010) Pathways Fund	1
Capital Area United Way (2012) Capital Area United Way Social Innovation Fund	1	The North Carolina Partnership for Children, Inc. (2016) Shape NC: Healthy Starts for Young Children Initiative	1
Capital Impact Partners (2011) Stewardship Capacity Fund	1	REDF (2010, 2015) REDF Social Innovation Fund Scaling the Impact of Social Enterprise	2
Corporation for Supportive Housing (2011) Reducing Risky Behaviors Through Supportive Housing	1	Share Our Strength (2014) No Kid Hungry	1
The Edna McConnell Clark Foundation (2010) The Edna McConnell Clark Foundation Social Innovation Fund	1	Silicon Valley Community Foundation (2014) The Big Lift	1
Foundation for a Healthy Kentucky (2010) Kentucky Health Futures Initiative	1	United Way for Southeastern Michigan 2011, 2016) Greater Detroit Early Childhood Innovation Fund Bib to Backpack	2
Friends of the Children (FOTC) (2016) Friends of the Children	1	United Way of Central Indiana (2016) Great Places 2020	1
Greater Twin Cities United Way, in Partnership with Generation Next (2012) STRIVE Alliance	1	United Way of Greater Cincinnati (2010) Greater Cincinnati Social Innovation Fund	1
GreenLight Fund (2012) GreenLight Fund Social Innovation Fund	1	United Way of Greenville County (2014) OnTrack Greenville	1
Jobs for the Future, Inc. (2010, 2014) National Fund for Workforce Solutions Opportunity Works	2	United Way of Lane County (2015) Kids in Transition to Schools (KITS)	1
The John A. Hartford Foundation (2012) IMPACT	1	U.S. Soccer Foundation (2011) Soccer for Success	1
Local Initiatives Support Corporation (2010, 2015) Expanding Financial Opportunity Centers Bridges to Career Opportunity Program	2	Venture Philanthropy Partners (2010) youthCONNECT Social Innovation Fund	1
Mayor’s Fund to Advance New York City (2010, 2015) Economic Opportunity Social Innovation Fund Connections to Care	2	Youthprise (2015) Opportunity Reboot	1
Methodist Healthcare Ministries of South Texas, Inc. (2014) Sí Texas: Social Innovation for a Healthy South Texas	1	Total Number of Awards	39

Note. SIF = Social Innovation Fund; CNCS = Corporation for National and Community Service.

Although CNCS made its last set of SIF grant awards in fiscal year 2016, SIF grantees and their subaward recipients (“subgrantees”) will continue to administer their programs until their federal funding is exhausted.

The SIF ETA Program

Given the high evaluation and evidence bar of the initiative, SIF grantees and subgrantees were required to commit significant time and resources to ensure formal evaluations of each program model that received funding from the SIF. The agency quickly realized the need to provide resources if the initiative was to fully meet its inherent evidence building goals and developed a comprehensive ETA program (see Figure 1 for the theory of change).

Figure 1.

Social Innovation Fund theory of change.

By statute, the SIF was to reserve not more than 5% of the funds appropriated for a fiscal year to support, directly or through contract with an independent entity, research and evaluation activities to evaluate the eligible entities and community organizations receiving grants under the initiative. The SIF ETA program was supported by third parties³ with a value of between US$4 million and US$5 million for services provided to an approximately US$350 million funding stream over a 7-year period of funding (representing around 1.3% of the initiative’s appropriation funds and 1.6% of the federal grant funds based on 80% grant disbursement ratio). This only reflects the external costs of the ETA program and does not include the agency’s costs, which primarily included CNCS staff time. Management and technical direction of the work at the agency level required at least 1.5–2 full-time equivalent research and evaluation staff and a significant amount of time on the part of the SIF program officers and leadership.

Although participation in the array of ETA activities was not mandatory for SIF grantees, subgrantees, and their evaluation partners, SIF evaluation requirements and expected milestones were requirements of the program, and as such, SIF stakeholders engaged in those activities in order to meet program requirements and expectations. The array of activities that comprised the ETA program and the process for their delivery to SIF grantees and their partners are described in the next section.

SIF ETA Activities and Delivery Process

The ETA program activities were tied to the evaluation planning, implementation, monitoring, and reporting expectations and requirements of the SIF. The evaluation planning process took place during the first year of the grant. Once an evaluation plan was approved, the evaluation moved to the second phase: implementation and reporting. The time frame for evaluation plan implementation and reporting was based on the length of the study and varied for different interventions. On average, the studies were expected to take around 3.5 years to complete. During this period, grantees would submit interim evaluation reports, which were reviewed and compiled by CNCS. The evaluation reports were used to meet CNCS’s information needs and provide feedback to grantees and their partners. The technical guidance and standardized feedback provided to the grantees, their subgrantees, and respective evaluation partners were all intended to strengthen the capacity and deliverables for the program at each phase of the evaluation. Figure 2 provides a visual overview of this process (Zandniapour & Deterding, 2017).

Figure 2.

Social Innovation Fund evaluation process.

CNCS developed a comprehensive ETA program organized around a flexible coaching model that included individual consultation and customized resource development. This framework of teaching and learning strategies (see Figure 3) is consistent with the empirical and theoretical evaluation capacity building (ECB) literature (e.g., Baizerman et al., 2002a; Labin et al., 2012) and the evaluation capacity building approach used by CNCS for other programs.

Figure 3.

Corporation for National and Community Service’s Evaluation Capacity Building (ECB) framework and ECB literature crosswalk.

Evaluation training and technical assistance were provided on emerging “hot topics” of importance to the SIF evaluation work (e.g., one example was on Propensity Score Matching, a strategy that many of the SIF evaluations were using to develop their comparison group and was deemed important to cover). CNCS also supported quarterly meetings of an Evaluation Workgroup, peer learning, and evaluation-focused sessions at convenings. Access to resources and expert advice was provided to grantees on the SIF Knowledge Network, the program’s virtual platform for the grantee community of practice.

The following case example (Table 3) illustrates how grantees were supported in their efforts to strengthen their evaluation capacity and advance the evidence base of their programs.

Table 3.

Case Example of SIF Evaluation Technical Assistance.

Case Description	Evaluation Challenges	Technical Assistance Provided
Methodist Healthcare Ministries of South Texas, Inc. (MHM) had eight separate subgrantees. These subgrantees all had slightly different interventions and target populations but were generally implementing an integrated health care intervention for a low-income, predominantly Hispanic population in the Rio Grande Valley. The grantee came to the SIF program with the goal of also doing a meta-analysis of all subgrantees, in addition to individual subgrantee-level evaluations. Results:	Each subgrantee had its own intervention and distinct eligibility criteria, research questions, and study designs The evaluation proposals of the subgrantees varied in quality and subgrantees proposed to use different measures and methods	CNCS worked with MHM to develop individual subgrantee evaluation proposals and strengthen the study design and analytic strategies of each subgrantee CNCS developed a common framework across subgrantees so that all subgrantees were using similar measures and asking similar research questions CNCS developed the meta-analysis and strategies for how they could conduct an overarching evaluation based on the new subgrantee study designs
The MHM completed all subgrantee evaluations (four conducted experimental design impact evaluations and four used quasi-experimental design impact studies. MHM meta-evaluation of Social Innovation for a Healthy South Texas (Sí Texas) was completed successfully.
The meta-evaluation study included a portfolio-wide impact study using a quasi-experimental design across the eight subgrantees with pooled individual-level data. The study examined the effectiveness of enhanced Integrated Behavioral Health (IBH) on improving patient health outcomes compared to participants receiving the subgrantee’s standard of care or similar services. The study produced a range of findings. One of the positive findings was that after 12 months of care, Sí Texas participants showed significantly lower depressive symptoms and blood sugar levels than did participants who received each subgrantee’s standard of care.

Note. SIF = Social Innovation Fund; CNCS = Corporation for National and Community Service.

Evidence of SIF ETA Effectiveness

The key question about the SIF ETA program is whether or not it worked. Empirical evidence of the SIF ETA’s effectiveness is found in three sources as follows: (1) findings from the national assessment of the SIF which provides rigorous evidence that the program was successful in building the evaluation capacity and practices of participating organizations, (2) the caliber of the evaluation designs implemented by SIF grantees, and (3) the credibility of the positive and causal findings obtained through successful implementation of these evaluation designs.

Results From the SIF National Assessment

The SIF national assessment was designed to facilitate program evaluation and management by CNCS and support the greater goal of learning from the federal government’s tiered evidence initiatives.⁴ A rigorous quasi-experimental impact study was conducted at the funding stream level to assess the effectiveness of the program in bringing about positive capacity change within the participating organization, particularly the intermediary organizations who were the most immediate sphere of influence for the SIF (Zhang et al., 2017; Zhang & Barrero, 2016; Zhang et al., 2015; Zhang & Sun, 2016; Zandniapour et al., 2017). The study aimed to measure change in a number of areas but inclusive of evaluation capacity, a key focus of this program, and relied on two comparison groups (a nonselected but eligible sample of SIF applicants and a nationally representative sample of similar nonprofit organizations across the country) to capture program impact. At the core of this study were the following questions: “Is SIF a transformative program in terms of improving organizational practices and policies of its grantee organizations?” and “What can be learned from the implementation and results of this program?”

Findings from this evaluation showed that the program has been successful in meeting its goals and has indeed facilitated organizational change (Zhang et al., 2015). The study found evidence of improved organizational capacities among the SIF grantees in 13 of 14 areas tied to the key features of the SIF. Of note is that impact findings were most pronounced and statistically significant in the domains of evaluation and evidence-driven decision making: (1) conducting rigorous evaluations of the programs, (2) using evaluation findings to improve programs, and (3) using evaluation findings to demonstrate and communicate effectiveness of programs funded by the organization.

The study found a combination of program features that were key to SIF’s impact and success in improving the capacity of partner organizations. These included financial investment and resourcing of evaluation; accountability (clear requirements and intensive coaching and ongoing monitoring of progress); and training, technical assistance, and other hands on support from CNCS researchers and their ETA provider, and support and engagement of the SIF program staff. The program also produced a community of professionals supporting one another in carrying out evaluation. Peer support among the SIF grantees and support from SIF grantees to the organizations they fund were important to successfully implementing the SIF program.

Strengthening Evaluation Designs and Building Evidence

The SIF grantees were not mandated to use a specific evaluation design to build evidence for their social programs. SIF grantees and subgrantees adopted a variety of study designs when evaluating their programs and impacts. The type of study pursued depended on the program’s size, service approach, the stage of existing evidence and implementation, allocated budget, and other considerations. Decisions around the design were made at the SIF grantee or local level with CNCS consultation and feedback. SIF’s portfolio of local evaluations consists of about 135 studies that are in different stages of development, implementation, and completion and vary in terms of design, scope, and outcomes.

Most of these use a combination of designs including process and cost studies to build knowledge about the workings of these promising interventions and improve implementation and include a comparison group to assess effectiveness. A subset examines fidelity questions or questions regarding feasibility of conducting future experimental or quasi-experimental studies (Spera et al., 2015). Among SIF-sponsored evaluations, 30% (40 studies) are implementing an experimental design, 53% (72 studies) employ a quasi-experimental design, and 17% (23 studies) have nonexperimental designs.

As has been stated previously, the SIF grantees and their subgrantees were expected to advance the evidence base of social programming and potentially increasing the number of interventions with moderate or strong evidence of effectiveness. The evidence tiers used by the SIF are described as follows.

Evidence Tiers

As outlined in the Social Innovation Fund Notices of Funding Availabilities, the tiers of evidence are defined as follows:

Strong evidence means evidence from previous studies on the program, the designs of which can support causal conclusions (i.e., studies with high internal validity), and that, in total, include enough of the range of participants and settings to support scaling up to the state, regional, or national level (i.e., studies with high external validity). The following are examples of strong evidence: (1) More than one well-designed and well-implemented experimental study or well-designed and well-implemented quasi-experimental study that supports the effectiveness of the practice, strategy, or program or (2) one large, well-designed, and well-implemented randomized controlled multisite trial that supports the effectiveness of the practice, strategy, or program.

Moderate evidence means evidence from previous studies on the program, the designs of which can support causal conclusions (i.e., studies with high internal validity) but have limited generalizability (i.e., moderate external validity). This can also include studies for which the reverse is true—studies that only support moderate causal conclusions but have broad general applicability. The following examples would constitute moderate evidence: (1) At least one well-designed and well-implemented experimental or quasi-experimental study supporting the effectiveness of the practice, strategy, or program, with small sample sizes or other conditions of implementation or analysis that limit generalizability; (2) at least one well-designed and well-implemented experimental or quasi-experimental study that does not demonstrate equivalence between the intervention and comparison groups at program entry, but that has no other major flaws related to internal validity; or (3) correlational research with strong statistical controls for selection bias and for discerning the influence of internal factors.

Preliminary evidence means evidence that is based on a reasonable hypothesis supported by credible research findings. Thus, research that has yielded promising results for either the program model or a similar program model will meet CNCS criteria. Examples of research that meet the standards include (1) outcome studies that track participants through a program and measure participants’ responses at the end of the program and (2) third-party pre- and posttest research that determines whether participants have improved on an intended outcome.

In terms of the level of evidence targeted by these evaluation studies, 15% (20 evaluations) targeted a strong level of evidence for the program, while 56% (76 evaluations) targeted moderate level of evidence and 29% (39 evaluations) aimed to achieve a preliminary level of evidence. Among studies that have been completed and published to date (106 evaluations), 10% (11 studies) have achieved a strong level of evidence, 29% (31 studies) achieved a moderate level of evidence, and the remaining 60% (64 studies) produced a preliminary level of evidence based on the review of the study design, implementation, and results.

Of the evaluations that were completed, 53% (56 studies) achieved the level of evidence they were targeting, 5% (five evaluations) achieved a higher level of evidence (three aimed for moderate and achieved a strong evidence level and two aimed for preliminary and achieved a moderate level of evidence), and 42% (45 evaluations) achieved a lower than targeted evidence level in the end.⁵ Among the latter group, five were aiming for a strong level of evidence and attained a moderate evidence level, 34 targeted moderate and landed in the preliminary tier and six aimed at strong and ended in preliminary level of evidence. There are various explanations for why the program evaluations did not achieve the intended evidence results and a great many have to do with what transpires in the process of implementation (in terms of data collection such as issues with sample attrition, access to administrative data, statistical power, baseline equivalence of treatment and counterfactual group, and others) that affect the internal and external validity of studies and in some cases it may be due to findings and results. The SIF National Assessment’s meta-synthesis of findings based on a subset of evaluation reports across the SIF evaluations provided the following insights: “Evaluation-related lessons concerned adequate funding for data collection, sizable administrative datasets, tracking longitudinal outcomes, and others, such as statistical power, detail about the control condition, and selection of inexperienced sites” (Zhang, 2015, p. 2).

Taken together, these success indicators provide evidence that the SIF ETA program was indeed effective in building evaluation capacity, as intended, strengthening evaluation practices among funded organizations and advancing the body of evidence for social programs designed to improve educational, economic, and health outcomes in underresourced communities.

Discussion

The process of operationalizing the SIF ETA program was a rich and generative experience that produced many findings, resources, and lessons that could be used to strengthen evaluation capacity and practices and increase the potential for improving the evidence base for social programs. This discussion is organized in two sections: (1) challenges identified and agency responses to these challenges and (2) lessons and takeaways to inform and strengthen future efforts and initiatives.

Early Challenges and Opportunities

CNCS’s operationalization of the ETA program and its requirements experienced a number of early challenges. These challenges required adjustments and in many ways informed the development of the ETA program. These challenges and responses are summarized in Table 4.

Table 4.

Challenges Encountered and Corresponding Responses.

Challenges	Responses
Inadequate and uneven preparation among funded grantees for meeting all SIF evaluation requirements (e.g., accurately assessing existing body of evidence, finding the right evaluation partner, budgeting for high-quality evaluations, assessing evaluation readiness among subgrantees, implementing high-quality evaluations) Significance: The agency quickly realized the need to provide resources if the initiative was to fully meet the inherent evidence-building goals of the initiative	CNCS developed a comprehensive evaluation technical assistance program organized around a flexible coaching model that included individual consultation and customized resource development
Inadequate early communication of expectations with grantees and their partners about evaluation requirements Significance: Given that SIF grantees and their subgrantees had developed standing contracts with their research partners on their own, when issues were discovered with their proposed designs and processes, it was difficult and complicated to fix those issues and ensure that they align with CNCS expectations	Clarified goals and expectations for grantees and their partners Clarified roles and responsibilities of CNCS/SIF evaluation team versus the SIF program office and the evaluation technical assistance provider to address issues of monitoring, compliance enforcement, communication, and the like Developed standardizing review forms to ensure a systematic and objective, and transparent approach to the review processes
Failure to foresee or account for inevitable lags in communications, given the number of various parties involved in the initiative Significance: Time lines established for evaluation plan development, refinement, review and approval were often unrealistic and in some cases created delays in implementation of the funded project	Revised turnaround times for reviews of evaluation plans and final reports

Note. SIF = Social Innovation Fund; CNCS = Corporation for National and Community Service.

Tiered evidence initiatives, including SIF, were the first of their kind. While there is a literature on evaluation capacity building and a handful of case examples that offer some practice guidance (e.g., Baizerman et al., 2002b; Cheng & King, 2017; Cousins & Bourgeois, 2014; Hilton & Libretto, 2017), there was very limited experience within federal government with designing and operationalizing a comprehensive technical assistance program like the one developed for the SIF. Early challenges noted above are only a handful of examples, as there were many more details and issues that required careful considerations and response. The ETA program developed in response to these early challenges may be of broad use by practitioners, grant makers, and evaluators alike.

Lessons Learned From the SIF ETA Program

Leadership prioritization

Staff participation in ETA was facilitated by engaged and supportive organizational and program leadership. SIF was very intentional on this score and reinforced buy-in by engaging intermediaries’ executive directors throughout the grant period and communicating about the shared objectives of the initiative to ensure there would be no lapse in prioritization on the part of the initiative’s primary stakeholders—SIF’s grantees. It is also worth noting that the SIF grantees represented high-capacity organizations, and this general capacity paired with leadership support facilitated a readiness for the evaluation demands of this initiative.

Trusting relationships

Planning and conducting evaluations in the real world poses many challenges and issues. Being able to openly discuss concerns and issues as they come up and having confidence in that all parties are vested in the successful resolution of those challenges are key for effective capacity building and assistance. Having a mindset that encourages holding back rather than sharing is a detriment to open and effective communication.

Timing matters

ETA is most effective when it is delivered “in time” to meet the needs of the organizations where they are and when there is absorption capacity. Due to the technical nature of many of the concepts and issues that are covered in ETA, when these types of information and requirements are communicated generally, they are difficult to process and absorb by the lay person. Communicating relevant content at the right time, when there is attention and focus on them by the recipient, enables them to connect to it and take it in. Moreover, effective ETA requires flexibility to meet the needs of the organizations and their partners, not overburdening them and ensuring that they are able to engage.

Customized coaching models

Different people have different learning styles. In the absence of direct knowledge of the learning style of the individuals, it is important to use different formats and delivery mechanisms to respond to diverse learning styles and facilitate take-up. Diversity of readiness and specific needs of the organizations within their particular contexts requires a model that is tailored or customized for the individual recipients and appropriate to the status of the existing evidence base and level of program maturity.

Dedicated resources

ETA is most effective and meaningful when it is offered in addition to financial and nonfinancial resources for evaluation to the grantee. This makes for a holistic package of support and acknowledges the value and benefits of evaluation capacity while recognizing that there are costs associated with evaluation and participation in capacity building. SIF recognized the need for both financial and nonfinancial resources. In addition to allocating financial resources for ETA, many resources were developed to assist grantees along the way. Table 5 shows some of the resources and tools developed for the SIF grantees and their partners for each phase of the evaluation process.

Table 5.

Sample ETA Resources by Topic and Evaluation Phase.

ETA Delivery Phase	Sample ETA Topics	ETA Resource
Evaluation Planning	Evaluation plans Establishing a baseline assessment of a program’s evidence base Screening and selecting a third-party evaluator Evaluation designs and budgets Impact evaluation readiness Institutional review boards	SIF Evaluation Plan (SEP) Guidance^a The SIF Rubric ^b Evaluator Screening Tips^c Evaluation Budgeting Guidance^d Impact Evaluability Assessment Tool, ^e Working with institutional review boards^f Self-paced Modules (tutorials)^g
Evaluation Implementation & Monitoring	Using administrative data Data collection Trouble-shooting design implementation challenges	Secondary/Administrative Data Use and Reporting, Implementation Reporting Outcomes/Impact Reporting Checklists
Evaluation Reporting	Evaluation reports	SIF Reporting Guidance document^h

Note. ETA = evaluation technical assistance.

^a Social Innovation Fund Evaluation Plan Guidance can be accessed at https://www.nationalservice.gov/impact-our-nation/evidence-exchange/social-innovation-fund-evaluation-plan-guidance

^b The Social Innovation Fund rubric for assessing levels of evidence can be accessed at http://www.nationalservice.gov/sites/default/files/resource/SIF-Rubric-Assessing-levels-of-evidence.12.19.14.pdf

^c Evaluator Screening Tips can be accessed at https://www.nationalservice.gov/sites/default/files/resource/Evaluator-Screening-Tips.pdf

^d Evaluation Budgeting Guidance can be accessed at https://www.nationalservice.gov/impact-our-nation/evidence-exchange/SIF-Budgeting-Rigorous-Evaluation

^e The Impact Evaluability Assessment Tool can be accessed at http://www.nationalservice.gov/sites/default/files/resource/FR_SIFImpactEvaluabilityAssessmentTool_Final_2016.pdf

^f Working with Institutional Review Boards can be accessed at https://www.nationalservice.gov/sites/default/files/resource/SIF_IRB_Tips_7_2014.pdf

^g The self-paced modules can be accessed at http://sep-tutorials.org/.

^h Social Innovation Fund Evaluation Reporting Guidance can be accessed at https://www.nationalservice.gov/impact-our-nation/evidence-exchange/social-innovation-fund-evaluation-reporting-guidance

Progress measurement and a strong feedback loop are important for continuous program improvement and achieving the intended goals of ETA

Identifying indicators and milestones for the ETA effort and capturing challenges, successes, and lessons learned helped promote learning, improvement, and increased ETA effectiveness. CNCS in consultation with the ETA provider developed a set of dashboard indicators and metrics for tracking progress. Among indicators and milestones used were total number of TA hours, as well as average number of hours of TA by grantee, the status of evaluation plans and reports as well as status of feedback and progress on those plans and reports, the types of study designs, evaluation budgets by types of study designs, and incoming, targeted and attained levels of evidence for the interventions. The dashboard report also included ETA topics by phase, common challenges faced by grantees or their subgrantees, best practices or lessons learned that could inform more targeted support. Progress on the ETA activities was reported and tracked on a weekly and quarterly basis.

The SIF and more specifically its ETA program’s design and implementation provides a successful framework for supporting the planning and implementation of rigorous local evaluations while building the infrastructure and capacities of its partner organizations to conduct such studies and use the generated evidence for decision making purposes. An independent national impact evaluation has verified the program’s success in building evaluation capacity. Important information about the 135 interventions that are addressing pressing challenges in local communities has been produced and their evidence base strengthened. SIF has increased the number of interventions with moderate and strong levels of evidence of addressing important social problems. The SIF ETA program provides another case example that advances the science and practice of evaluation capacity building.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

Author Biographies

Lily Zandniapour, PhD Research and Evaluation Manager, is an economist with more than 20 years of experience in the applied research and evaluation field across the private, public, and nonprofit sectors. She supports CNCS’ mission, programs, and policy priorities through the design, implementation and oversight of research, evaluation, and capacity building activities, as well as knowledge management and communications.

Mary Hyde, PhD Director of Research and Evaluation, is a community psychologist with more than 20 years of research, evaluation, technical assistance, and training experience in social and human services. She is responsible for advancing the evidence base for national service programs, sponsoring scholarship on volunteering and other forms of civic engagement, as well as fostering a culture of evaluation within the agency and the field.

References

Baizerman

Compton

D. W.

Stockdill

S. H.

(2002a). The art, craft, and science of evaluation capacity building. New Directions for Evaluation, 93, 1–120.

Baizerman

Compton

D. W.

Stockdill

S. H.

(2002b). Summary and analysis of the case studies: Themes across the cases. New Directions for Evaluation, 93, 101–108.

Cheng

King

J. A.

(2017). Exploring organizational evaluation capacity and evaluation capacity building: A Delphi study of Taiwanese elementary and junior high schools. American Journal of Evaluation, 38, 521–539.

Corporation for National and Community Service. (n.d.). Social Innovation Fund evaluation plan guidance: A step-by-step guide to designing a rigorous evaluation. https://www.nationalservice.gov/impact-our-nation/evidence-exchange/social-innovation-fund-evaluation-plan-guidance

Corporation for National and Community Service, Office of Research and Evaluation. (2014). Impact evaluability assessment tool (lead author, Lily Zandniapour). http://www.nationalservice.gov/sites/default/files/resource/FR_SIFImpactEvaluabilityAssessmentTool_Final_2016.pdf

Cousins

J. B.

Bourgeois

(2014). Cross-case analysis and implications for research, theory, and practice. New Directions for Evaluation, 141, 101–119.

Government Accountability Office. (2016). Tiered evidence grants: Opportunities exist to share lessons from early implementation and inform future federal efforts. http://www.gao.gov/assets/680/679917.pdf

Haskins

Baron

(2011). Building the connection between policy and evidence: The Obama evidence-based initiatives. Brookings Institution Press. https://www.brookings.edu/wp-content/uploads/2016/06/0907_evidence_based_policy_haskins.pdf

Haskins

Margolis

(2014). Show me the evidence. Brookings Institution Press.

10.

Hilton

Libretto

(2017). Evaluation capacity building in the context of military psychological health: Utilizing Preskill and Boyle’s multidisciplinary model. American Journal of Evaluation, 38, 393–404.

11.

Labin

Duffy

Meyers

D. C.

Wandersman

Lesesne

C. A.

(2012). A research synthesis of the evaluation capacity building literature. American Journal of Evaluation, 33, 307–338.

12.

Lester

(2015). Social Innovation Fund: Early results are promising. The Social Innovation Research Center. http://www.socialinnovationcenter.org/wp-content/uploads/2015/07/Social_Innovation_Fund-2015-06-30.pdf

13.

Maynard

Goldstein

Nightingale

D. S.

(2016). Program and policy evaluation in practice: Highlights from the federal perspective. In Peck

L. R.

(Ed.), Social experiments in practice: The what, why, when, where, and how of experimental design & analysis. https://doi.org/10.1002/ev.20209

14.

Nussle

Orszag

(Eds.). (2014). Moneyball for government. Disruption Books.

15.

Spera

DiTommaso

Hyde

Zandniapour

(2015). The Social Innovation Fund: Pioneering an evidence-based investment model. Corporation for National Community Service. https://www.nationalservice.gov/impact-our-nation/evidence-exchange/SIF-Pioneering-Investment-Model

16.

Zandniapour

Deterding

N. M.

(2017). Lessons from the Social Innovation Fund: Supporting evaluation to assess program effectiveness and build a body of research evidence. American Journal of Evaluation. https://journals-sagepub-com.web.bisu.edu.cn/doi/pdf/10.1177/1098214017734305

17.

Zandniapour

Griffith

J. D.

Zhang

Sun

Pershing

(2017). Strengthening organizational practice: Evidence from the Social Innovation Fund national assessment. The Journal of Nonprofit Education and Leadership, 7(4). http://js.sagamorepub.com/jnel/article/view/8642

18.

Zandniapour

Vicinanza

(2013). Budgeting for rigorous evaluation: Insights from the Social Innovation Fund. Corporation for National Community Service. http://www.nationalservice.gov/impact-our-nation/evidence-exchange/Budgeting_Rigorous_Evaluation

19.

Zhang

(2015). Meta-synthesis of evaluations across the SIF: Interim report (Prepared for the Corporation for National and Community Service, Office of Research and Evaluation). ICF International. https://nationalservice.gov/sites/default/files/evidenceexchange/Interim_FR_SIFNA_SynthesisReport_Final.pdf

20.

Zhang

Barrero

(2016). Issue Brief #5: Designing and implementing rigorous evaluation for the Social Innovation Fund (Prepared for the Corporation for National and Community Service, Office of Research and Evaluation). ICF International.

21.

Zhang

Griffith

Pershing

Sun

Malakoff

Marshland

Peters

Field

(2017). Strengthening organizational capacity and practices for high-performing nonprofit organizations: Evidence from the national assessment of the Social Innovation Fund-A public-private partnership. Public Administration Quarterly, 41(3).

22.

Zhang

Griffith

Sun

Malakoff

Pershing

Marshland

Peters

Field

(2015). The SIF’s impact on strengthening organizational capacity (Prepared for the Corporation for National and Community Service, Office of Research and Evaluation). ICF International.

23.

Zhang

Sun

(2016). Meta-analysis of evaluations across the Social Innovation Fund program: Final report (Prepared for the Corporation for National and Community Service, Office of Research and Evaluation). ICF International.