ETA Special Issue Editor Essay

Paper Summaries

The first paper (Zandniapour and Hyde, 2020) describes ETA provided by the Corporation for National and Community Service for the Social Innovation Fund (SIF), ¹ one of the Obama Administration’s Tiered Evidence initiatives. From the start, one of SIF’s key objectives was to invest in both evaluation and capacity building in the implementing partners in order to support the development and use of rigorous evidence. The ETA provider delivered ETA in all phases of the evaluation process (e.g. design, implementation, analysis, and reporting) through a flexible grantee-specific coaching model that was augmented with regular group meetings, peer learning, and a virtual platform. The authors suggest that the ETA was effective based on findings from a national assessment that found improved organizational capacities among grantees in most areas, high-quality evaluation designs, and credible findings that resulted from those evaluations that helped advance the body of evidence.

The second paper (Knab and Cole, 2020) considers ETA provided for another Tiered Evidence program, the Teen Pregnancy Prevention Program (TPP), ² in the Office of Adolescent Health, now part of the Office of Population Affairs in the Department of Health and Human Services (HHS). Here, the ETA was provided to 65 recipients across two cohorts. As a condition of their funding, recipients were required to conduct rigorous evaluations that could meet standards of the TPP Evidence Review. ³ Satisfying these evidence standards provided an explicit goal for the TPP evaluations. Funding was provided to recipients through cooperative agreements, not grants. This cooperative agreement mechanism allowed the federal government to withhold funding if recipients did not comply with funding conditions, including both participating in ETA and also making changes to evaluations when the ETA team had strong recommendations. As was the case for SIF programs, ETA for TPP was provided across the evaluation lifespan and used a similar mix of one-on-one individualized support, group meetings, and supplemental resources. The authors assert that ETA built substantial evaluation capacity in both cohorts and that the evidence base of TPP programs was broadened and enhanced.

The third paper (Zaid et al., 2021) considers ETA in the Maternal, Infant, and Early Childhood Home Visiting program (MIECHV) ⁴ administered by the Health Resources and Service Administration in collaboration with the Administration for Children and Families at HHS. As with SIF and TPP, the objective of ETA was to support and build capacity among both awardee project directors and evaluators to conduct rigorous evaluations of their programs. ETA was provided through all phases of the evaluation process and included individual support, group support, and universal webinars and guidance documents to address cross-cutting issues. The authors explain how the ETA was grounded in adult learning principles and utilized a tiered and phased approach starting with identifying technical assistance needs, priorities, and goals. ETA success is suggested by virtue of common strengths in the resulting evaluations.

The fourth paper (Gothro et al., 2020) considers ETA in two different interventions that were in the design and early implementation phases. The two programs—Youth At-Risk of Homelessness program at the Children’s Bureau in the Administration for Children and Families and the Performance Partnership Pilots (P3) ⁵ —were new programs in very early stages of development and roll-out. ETA included both individual and group support and was provided to both grantees and local evaluators; notably, however, ETA providers at times went beyond evaluation support to assist the programs with developing their interventions. Given the very early stages of the programs and their evaluations at the time of writing, it is not known if the ETA improved the quality and success of local evaluation. However, the authors assert that the ETA helped improve communication and understanding among the many players involved in these programs and helped grantees lay the groundwork for future rigorous evaluations.

Findings

These four papers suggest that ETA is generally an effective use of limited resources. In particular, these papers provide suggestive evidence that ETA can increase the evidence base, build capacity in grantees to engage productively with evaluators, build capacity for smaller or less-experienced evaluators to successfully conduct rigorous evaluations, and facilitate relationships between program staff and evaluators. If funders such as the federal government aim to increase the evidence base in a particular area, then they need high-quality, robust evaluations that will produce credible information about the effectiveness of the funded programs. In the absence of ETA, funders may waste resources on poorly designed or executed evaluations. Such evaluations do not produce useful information, which could cause funds to be wasted on ineffective programs themselves. Furthermore, the programs need the information from high-quality evaluations to inform their own learning and improvement efforts.

Beyond producing credible evaluation results, the importance of building grantee capacity is a critical goal of ETA. In most cases, it is not reasonable to expect grantees to be skilled evaluators themselves, but ETA can help them become learning organizations and gain enough knowledge of evaluation principles to engage productively with evaluators. Further, the ETA provider may also help advise (at the margins) on the intervention itself, which can also help produce an “evaluable” intervention. The papers in this issue also suggest that ETA often produces useful resources that can efficiently be shared with both current and future grantees, which has the potential to help increase grantee capacity even more broadly.

Another goal of ETA is to build the capacity of small or local evaluators. This begs the question of why this support is necessary. An alternative model would have grantees only hire experienced evaluators who do not need ETA, or the national funder could select a single experienced national evaluator for all programs. These options are not always practical or desired for the type of grant programs documented in these papers. For one, the large evaluation contractors often come at a price point that is unaffordable to small grantees, or the size of the local evaluation may not be “worth it” for the large firms. But perhaps more importantly, the broad and increased focus on evidence-building in recent years means that demand for evaluation is continuing to increase. With increased evaluation activity comes the need to build capacity among smaller, local evaluators to expand the available pool of high-quality, skilled options. Presumably, a goal of building capacity in the evaluation field more generally is to increase competition and the ability of local evaluators to produce high-quality studies. This suggests a continued role for local evaluators to undertake small evaluations, often with the provision of ETA at least at first, whereas larger national firms will continue to undertake larger and often higher-profile evaluations. The papers in this issue suggest that many local evaluators seem to need the support and guidance that comes from ETA. This is not surprising. The big firm model includes such support and peer review within the firm. Small firms do not have the scale to provide these roles internally; instead, ETA may offer similar benefits through a cohort model and shared resources that address common evaluation challenges.

Finally, and importantly, ETA can facilitate relationships between local program staff and evaluators. These two groups often bring vastly different skillsets and knowledge to what is a complex joint venture. At the same time, ETA providers can be an important bridge between funders and grantees by serving as a trusted third party to both. Such a bridge is particularly important when evaluations encounter challenges, as they typically do. When evaluations face such challenges, the ETA provider can help all parties navigate needed changes to the original plans.

Recommendations for Success

Evaluation technical assistance clearly has potential, but it is not always the best use of limited funds. While it is true that funders—and the field at large—would prefer that all studies are high-quality, ETA may be particularly useful for programs attempting to do impact studies that meet clearinghouse standards. For example, the tiered evidence programs like SIF, TPP, and MIECHV were designed to have explicit evaluation requirements. These funding streams have a clear end goal for the evaluation, one designed to meet certain pre-determined standards. ETA can provide valuable support to help meet those benchmarks.

Nonetheless, many ETA efforts are focused on a more expansive view of evaluation, including support during the design or early implementation phases as illustrated by the fourth paper in this issue. This suggests a role for ETA focused not only on a particular high-quality impact evaluation but instead aimed at expansive capacity building to create a data-driven and evaluative culture among grantees. This is an important goal if we believe that organizations should be more evidence-based and data-driven. Such capacity building also serves as a precursor to preparing organizations for an eventual evaluation that can assess efficacy or effectiveness. Many grantees in these papers were implementing programs in the early stages in their evidence-building journeys; it would not yet have been appropriate for them to undertake an impact evaluation, but the ETA nonetheless laid necessary foundations for the future.

The papers in this special issue converge around a series of common recommendations for the type of ETA that works best. First, the timing of ETA is clearly critical so as to have maximum influence. ETA is likely most effective when begun as early as possible in the evaluation cycle since an early start allows the ETA provider to influence the evaluation design. The authors also suggest that ETA must be nimble, adaptive, and responsive to “on demand” needs as the evaluation proceeds. This is when challenges inevitably arise and ETA can guide evaluators and grantees toward necessary changes that will maximize the potential for useful results.

Second, the authors of these papers believe that effective ETA needs to be tailored to meet specific grantee and evaluator needs for their particular program, context, and evaluation design. Nonetheless, providing ETA to a cohort of grantees at the same time allows for some efficiencies to address areas of common concern among grantees. ETA providers do typically supplement individual coaching with group TA, peer support, and shared resources to address cross-cutting issues.

Third, the papers suggest that ETA usually works best when it is quite intensive. Such intensive ETA necessarily requires sufficient contracted resources, funder staff time, and participant time by both grantees and evaluators. ETA will likely produce more benefits when participation is either mandatory or when grantees and/or evaluators are required to do what the ETA provider recommends. At the same time, to be successful the recipients of ETA need to see its value and program leadership should prioritize and support participation.

Finally, and perhaps most importantly, the ETA will not be successful without building strong relationships among ETA providers and both the program being evaluated and their evaluator. All parties need to be clear about the desired outcomes from the outset, and the ETA provider must be a trusted “critical friend” who can help all parties navigate challenging, complex, and ever-changing conditions.