Challenges and Solutions in Evaluation Technical Assistance During Design and Early Implementation

Details on the Structure of the YARH Grant Program

The Children’s Bureau (CB) in the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services awarded grants to local communities to develop model interventions to prevent homelessness among youth and young adults with child welfare involvement. Simultaneously, the Office of Planning, Research and Evaluation (OPRE) awarded a contract to provide ETA and conduct a process study of the YARH grantees. ²

To date, there have been three phases in the YARH grant program. Awarded to 18 communities in September 2013, YARH-1 served as a planning period. ³ During the planning period, CB expected grantees to (1) identify the target population for intervention services, (2) propose plans for reaching that population, and (3) design or enhance a service array to improve outcomes in four areas: stable housing, permanent connections, education or employment, and social–emotional well-being. ⁴ YARH grantees were expected to work with three target populations: (1) youth currently in foster care between the ages of 14 and 17, (2) youth preparing to age out of foster care between the ages of 17 and 21, and (3) homeless youth and young adults with foster care histories up to age 21.

The 18 grantees represented state child welfare agencies, county or tribal child welfare agencies, and community-based organizations located throughout the continental United States in urban, rural, and suburban areas (Figure 1). Most grantees sought, and received, the maximum funding level of US$360,000 per year for 2 years. These funds supported any costs related to administration, planning, programming, and local evaluation. Nearly all YARH-1 grantees applied with an identified local evaluator, although a few state grantees needed to follow their procurement rules, which required bringing the local evaluator on board after the award of the YARH grant. CB provided guidance on expectations for YARH-1 work in preparation for a future evaluation. Applications needed to include and were rated on activities related to design work; a solid plan for developing and finalizing an evaluation design; demonstration that a qualified and appropriate team was in place related to activities of data mining and analysis, development and refinement of measures, and development of a rigorous evaluation plan; and documentation of the capacity to conduct a rigorous evaluation either with in-house or contracted staff. CB did not provide guidance or requirements about the local evaluator either in the funding opportunity announcement or subsequent to the awarding of the grants. The grantee was responsible for the relationship with the local evaluator; neither the funder nor the ETA provider had any control or sway over the relationship. The ETA provider worked with both the grantee and the evaluator.

OPEN IN VIEWER

Figure 1.

Location of Youth At-Risk of Homelessness and Performance Partnership Pilots for Disconnected Youth grantees.

Expectations of YARH grantees

CB expected YARH-2 grantees to (1) begin initial implementation of the intervention, services, and supports to the target populations; (2) test critical elements including intervention processes and data collection activities; and (3) modify components to improve and fine-tune the intervention model. As with YARH-1, the YARH-2 Funding Opportunity Announcement (FOA) did not include specific requirements related to a summative evaluation, although it included requirements that contributed to preparation for a summative evaluation, including formative evaluation activities. The YARH-2 FOA indicated that a future YARH-3 phase, if funded, would include a rigorous impact evaluation of some of the programs developed and tested in YARH-2.

Table 1 provides an overview of the characteristics of the YARH grantees.

OPEN IN VIEWER

Table 1.

Overview of YARH-1 and YARH-2 Grantees.

Site	Type of Lead Agency	Phase Participation
Alameda County Social Services Agency, Oakland, CA	County/tribal child welfare agency	YARH-1 and -2
BCFS Health and Human Services, San Antonio, TX	County/tribal child welfare agency	YARH-1
ChildNet, Inc., Plantation, FL	County/tribal child welfare agency	YARH-1
Clark County Department of Family Services, Las Vegas, NV	County/tribal child welfare agency	YARH-1
Colorado Department of Human Services, Denver, CO	State child welfare agency	YARH-1 and -2
Confederated Salish and Kootenia Tribes, Pablo, MT	County/tribal child welfare agency	YARH-1
Healing Place Serve, Baton Rouge, LA	Community-based organization	YARH-1
Humboldt County Health and Human Services, Eureka, CA	County/tribal child welfare agency	YARH-1
Lighthouse Youth Services, Cincinnati, OH	Community-based organization	YARH-1 and -2
Minnesota Department of Human Services, St. Paul, MN	State child welfare agency	YARH-1
New Jersey Department of Children and Families, Trenton, NJ	State child welfare agency	YARH-1 and -2
Oklahoma Department of Human Services, Oklahoma City, OK	State child welfare agency	YARH-1
Philadelphia City Department of Human Services, Philadelphia, PA	County/tribal child welfare agency	YARH-1
United Way of King County, Seattle, WA	Community-based organization	YARH-1
University of Maryland, Baltimore, MD	Community-based organization	YARH-1 and -2
Westchester County Department of Social Services, White Plains, NY	County/tribal child welfare agency	YARH-1 and -2
Wisconsin Department of Children and Families	State child welfare agency	YARH-1
Youth Network Council dba Illinois Collaboration on Youth, Chicago, IL	Community-based organization	YARH-1

Source: ETA records.

Note. YARH = Youth At-Risk of Homelessness.

Details on the Structure of the P3

The P3 emerged from federal and nonfederal concerns that the sheer number of legislative, regulatory, and administrative polices of disparate federal programs for disconnected youth was hampering efforts to serve this group effectively (The White House, 2011). The Consolidated Appropriations Act of 2014 included authorization to pilot test the hypothesis that awarding flexibilities to states, localities, and tribal agencies to pool funds across discretionary programs and to obtain waivers from federal programmatic requirements would result in more effective services for disconnected youth. The P3 program involves multiple federal agencies, as seen in Figure 2. Figure 2 includes additional federal agencies who joined the initiative in subsequent cohorts.

OPEN IN VIEWER

Figure 2.

Federal Performance Partnership Pilots for Disconnected Youth partners.

As the designated lead agency, the U.S. Department of Education issued the notice inviting applications (NIA) for the first round of P3 pilots in November 2014. The NIA identified the target population as “disconnected” youth and defined disconnected youth as “individuals between the ages of 14 and 24 who are low-income, and either homeless, in foster care, involved in the juvenile justice system, unemployed, or not enrolled in, or at risk of dropping out of, an educational institution.” (Federal Register, 74, p. 70034). The first cohort of nine pilots was announced the following October. ⁶ The pilots, including their granted flexibilities, expired in September 2018. Conducting an evaluation was not a condition for a P3 pilot award; however, applications that proposed an independent local RCT or QED evaluation received additional points. The NIA did not stipulate any particular standards for methodological rigor or statistical power.

Every application identified a local evaluator, although one pilot replaced their evaluator in the first 6 months. The NIA did not include any requirements or expectations of the local evaluator. Rather, the grantee was responsible for the relationship with the local evaluator. Neither the funder nor the ETA provider had any control or sway over the relationship. The ETA provider worked with both grantees and local evaluators. P3 pilots also worked with a separate programmatic technical assistance provider, provided under a separate contract. The ETA liaisons and programmatic coaches frequently provided TA jointly given the interrelated nature of programmatic and evaluation TA.

Table 2 presents an overview of the P3 Cohort 1 pilots. Pilots included a tribal agency (1), a state agency (1), and local agencies (7). Agency types include human services (3), economic or workforce development (4), a public safety department (1), and a mayoral office (1). The majority of pilot programs ranged in intended size from fewer than 80 youth to 210 youth, although one pilot expected to serve 1,000 youth and another pilot expected to serve 8,000 youth. Five pilots initially proposed QEDs and four pilots initially proposed RCTs. In practice, two pilots were unable to conduct the type of evaluation originally proposed. Instead, they sought federal approval to change their evaluation designs: one from an RCT to a QED and the other from a QED to an implementation study with no comparison group.

OPEN IN VIEWER

Table 2.

Overview of Performance Partnership Pilots for Disconnected Youth Cohort 1 Grantees.

Site	Type of Lead Agency	Program Recruitment Goal	Proposed Evaluation Design	Final Evaluation Design
Baton Rouge, LA	Local (mayor)	84	RCT	RCT
Broward County, FL	Local (human services)	420	RCT	RCT
Chicago, IL	Local (human services)	210	RCT	QED
Indianapolis, IN	Local (public safety)	80	QED	Implementation
Los Angeles (City), CA	Local (development)	8,000	QED	QED
Oklahoma County, OK	State (human services)	60	QED	QED
Seattle, King County, WA	Local (development)	100	QED	QED
Southeastern Kentucky	Local (development)	1,000	QED	QED
Ysleta del Pueblo Sur	Tribal (development)	85	RCT	RCT

Source: ETA records.

Note. QED = quasi-experimental design; RCT = randomized controlled trial.

Designing ETA

ETA was designed in response to each grantee’s progress. Any activities or supports were discussed with both the federal agencies responsible for the ETA and the agencies responsible for the grantees. Staff overlap enabled the P3 ETA and YARH ETA to learn from the other grant program’s ETA. Additionally, staff worked with the Office of Adolescent Health providing ETA to teen pregnancy prevention grantees. ETA staff shared materials across grant programs including evaluation plan and report templates, webinar ideas and materials, and tips for dealing with similar issues.

ETA Content and Format

ETA was provided to YARH and P3 grantees on a variety of topics, using different formats. One important consideration in determining the format was whether grantees would need the information at the same time. For some topics, grantees needed the ETA at different times because different grantees developed their intervention and evaluation plans at different paces. In these cases, one-on-one consultations or written supplements were appropriate. For issues that all grantees were encountering at a similar point in time, group ETA was more appropriate. Table 3 presents the various formats and content of the ETA provided to the three groups of grantees (P3, YARH-1, and YARH-2).

OPEN IN VIEWER

Table 3.

ETA Offerings by Grantee Group.

ETA Format	YARH-1	YARH-2	P3
One-on-one consultations		Monthly phone callsYearly in-person working meetingsFull range of topics, including data analysis, intervention definition, defining target populations, thinking about the comparison condition, potential outcomes, and potential evaluation designs	Monthly phone callsMostly fine-tuning of intervention, comparison, and evaluation design (including outcomes)
Group	Limited phone callsWebinars or meeting presentationsData analysisAdministrative dataTheory of changeLogic modelsImplementation scienceData collection techniquesResearch designs	Webinars or meeting presentationsVarious templatesCost analysisFormative evaluationMeasuresYouth engagement in servicesFidelity	Webinars or meeting presentationsEvaluation plan template Clearinghouse for Labor Evaluation and Research and What Works Clearinghouse standardsEquivalenceEvaluation report templateLogic models
Written	Memorandum on institutional review boardsMemorandum on cost analysis	Memoranda on involving youths in research and measurement concernsSummaries of interventions discussing critical components and findingsTemplates to be submitted to funder: theory of change; logic model; summary of population, intervention, comparison, and outcomes; usability testing; formative evaluation; summative evaluation	Templates to be submitted to funder: evaluation plan, evaluation abstract, and evaluation report

Source: ETA records.

Note. ETA = evaluation technical assistance.

The ETA was provided in three basic formats: (1) one-on-one consultations (phone calls or in-person meetings), (2) group interactions (webinars or larger conference calls), or (3) written materials (e.g., templates and memoranda). The YARH-1 grantees were not offered one-on-one consultations, although they were offered both group interactions and written supports. YARH-2 and P3 grantees received the ETA in all three formats. Grantees have told liaisons that one-on-one consultations were the most helpful because the focus was on their own program, evaluation, and challenges. Both YARH-2 and P3 grantees participated in monthly phone calls focused on the ETA, although their liaisons were available more frequently if needed. Initially, group interactions were offered roughly quarterly, depending upon what topics needed to be addressed. Templates were provided to help grantees systematically document information related to their work. The templates were designed to support grantees in documenting and testing their intervention in preparation for a summative evaluation. YARH-2 grantees were provided five templates to be completed for each unique comprehensive service model: (1) theory of change; (2) logic model; (3) a description of the target population, the intervention, the potential comparison group, and the outcomes of interest; (4) usability tests and findings; and (5) formative evaluation and findings. YARH-2 grantees also received written supports including summaries of interventions and memos on measuring specific outcomes such as homelessness. P3 pilots were provided an evaluation plan template, evaluation report abstract template, and an evaluation report template.

All ETA was designed and provided with a rigorous evaluation in mind. Given that YARH-1 grantees were not yet at the stage of conducting a rigorous evaluation, the ETA focused on what a rigorous evaluation needed to include and how that affected the comprehensive service model design work they were conducting. The ETA for YARH-2 grantees focused on (1) defining the comprehensive service model, including articulating a cohesive theory of change and logic model; (2) tracking initial implementation, including usability tests for newer elements and a formative evaluation; and (3) ensuring that a rigorous evaluation was feasible, including considering different designs and articulating the comparison condition. P3 pilots received ETA to strengthen their initial local evaluation designs, which included articulating research questions and outcomes rather than performance measures that would be the focus of the evaluation.

Did the ETA Improve the Local Evaluations?

Success cannot be defined in the same manner for these two ETA efforts. For example, one definition could be that each P3 pilot addressed identified issues in their evaluation as they arose and wrote local evaluation reports that could receive a “desirable” rating from a federal systematic review effort. (Desirable ratings generally mean the evaluation would have no fatal design or analytic flaw.) The first cohort of P3 pilots submitted evaluation reports at the end of June 2018, which was the first comprehensive review of the evaluations. The eight that used an RCT or QED would likely receive a desirable rating from a federal systematic review effort based on the ETA review of the reports.

Whether the federal government’s efforts to support rigorous evaluations by providing ETA to YARH grantees resulted in rigorous evaluations is still unknown. ACF awarded a contract in July 2019 to support a federally led rigorous evaluation of one or more YARH-2 interventions, but it remains to be seen how successful the YARH-3 summative evaluation will be. Determining whether the YARH grantees produce rigorous evaluation reports that meet the criteria of a federal systematic review effort will likely not be possible until 2024 or later.

Although we cannot definitively state that ETA to YARH and P3 grantees was “successful,” we can cite ways we believe the ETA improved grantee’s work. These findings are common to both the YARH and the P3.

How Did Grantees Use the ETA?

Even within a single grant program, not all grantees started at the same point, which required the ETA provider to be flexible. Because YARH included designated time for program planning and implementation, grantees were expected to use ETA to consider what was needed for a summative evaluation while completing planning and initial implementation work. In YARH-1, grantees varied in their access to administrative data, skills to analyze the administrative data, and ability to think innovatively with respect to their comprehensive service model. In YARH-2, grantees discussed comprehensive service models in their grant applications based on their YARH-1 work. However, some grantees included services that they or their partners were unwilling to withhold from some youth and young adults, so those components could not be tested in a rigorous summative evaluation. For example, some grantees included trauma-informed training as part of their intervention but felt it was important all staff were trained, not just those involved in YARH. Another grantee worked to change the intake process, which the grantee implemented in all offices and with all youth, not just those involved in YARH. Other grantees realized that their original comprehensive service model was not distinct enough from their existing services to have a strong contrast for testing in a summative evaluation. In these cases, the grantees needed to rethink their comprehensive service models.

The difference in time line—YARH having 6 years total and P3 having 3 years total—meant that P3’s ETA often began at the same time or shortly before the local evaluation. Because some P3 pilots were starting new interventions, program implementation was sometimes starting simultaneously and did not always go according to plan. Other P3 pilots were expanding pilot programs and ran into issues with expansion. While the staff from the programmatic TA contract had primary responsibility for helping to address implementation challenges, ETA was needed to address implementation challenges in ways that did not compromise the rigor of the evaluation.

Benefits of ETA

A primary role ETA providers played was working to establish a better understanding between members of the grantee or pilot team—that is, its’ program staff and evaluation staff. Each group has a particular focus—providing or evaluating services—that is intertwined. Focusing on one particular part of the work is important but can be problematic—for example, if the program staff forget to share service changes with the evaluation staff or the evaluation staff alters intake processes without confirming whether those changes are feasible for the program staff. Working together can be challenging when individuals use different terms for the same concept or are unfamiliar with a concept. One example is a grantee in which the program staff used both the term “case manager” and the term “case planner.” Neither the local evaluator nor the ETA provider realized the terms were not interchangeable until many months into the project. At times, ETA providers helped local teams improve communication with each other. Ideally, this ETA will have effects that last beyond these particular evaluations.

ETA was designed to improve local evaluators’ abilities to work with programmatic teams and vice versa. The local evaluators brought a variety of experience to the projects. Local evaluators were generally from either a research firm or a university. They came from a wide range of disciplines and used discipline-specific frameworks and language to describe their work. It is important for ETA providers to work with grantee teams to help them define their conditions, including information on the comparison condition. One-on-one consultations with grantee/pilot teams often focused on helping local evaluators explain their evaluation processes to programmatic staff and ensuring both parts of the team had the same understanding of the services being evaluated.

In both projects, ETA providers helped grantees/pilots lay the groundwork for rigorous evaluations. Conversations with YARH grantees and P3 pilots included discussions of how to ensure rigor in RCTs by fully articulating the randomization process, monitoring the implementation of randomization, tracking the sample and attrition, and checking equivalence between the intervention and comparison groups on critical baseline characteristics. Grantees/pilots planning QEDs needed support in thinking through how to define the comparison group. The ETA provider also discussed the need to collect data at the same time from both conditions.

Challenges Encountered During ETA

The two initiatives were distinct so the challenges were distinct as well. Notably, the YARH challenges focused on activities related to preparing for a rigorous evaluation. The P3 challenges arose from implementation of an intervention and conducting an evaluation simultaneously.

Balancing Program Requirements and Evaluation Needs—The Outcome Measures

Both YARH and P3 grantees/pilots needed the ETA to be focused on defining evaluation outcomes. YARH grantees were focused on the four outcomes mentioned in the U.S. Interagency Council on Homelessness framework: (1) stable housing, (2) employment or education, (3) permanent connections, and (4) social–emotional well-being. The ETA covered how to identify specific constructs and measures within those outcome domains. Most grantees also had conversations with the ETA provider about what measures may be affected by their comprehensive service model in different time frames. The conversations emphasized the need to focus on outcomes that could be measured for both intervention and comparison groups at similar points in time and that would likely be affected by the intervention and not by other services in the community.

P3 pilots and P3 federal agencies signed performance agreements that included performance measures. Unfortunately, those performance measures, while appropriate for program monitoring, were not always a good fit for an evaluation outcome. The performance measures needed to be collected only from intervention group members and were not necessarily easily available for comparison group members. In addition, the performance measures were so specific in some cases that the analytic sample could be fewer than 10 youths—for example, the number of students who took a particular end-of-year test in a particular subject. Conversations with the ETA provider helped pilots think about what data they could access for both the intervention and comparison groups that would be reasonable outcomes for their intervention.

Would Changing the Evaluators Change the ETA?

One obvious question is whether funders should include specific requirements for engaging local evaluators at the application stage, including requirements for the evaluator experience with the target population, particular program, and evaluation design. However, this could mean certain types of applicants are not able to meet be requirements—for example, state agencies that must conduct an open competition to identify an evaluator after a grant is awarded—and others would not feel they could afford this effort if they cannot be certain of winning funding.

How to Make ETA Successful

The relationship that an ETA provider develops with the grantee team is paramount to the success of the ETA. Building a strong relationship requires the ETA provider to be flexible—both in terms of where the ETA begins and in how the ETA is delivered. The ETA provider needs to remember and balance the different perspectives, priorities, and understandings of the parties involved, including program staff, local evaluation staff, and funders.

The ETA provider needs to be a “critical friend” and refer issues to the funder as needed. A critical friend helps grantees see problems—or potential problems—based on information from the particular grantee, similar grantees, or other rigorous evaluations with shared characteristics. More importantly, the critical friend helps the grantee and local evaluator understand and address the problem (or potential problem). For example, the ETA provider found that one pilot’s recruitment and randomization process threatened the validity of the proposed random assignment design. By raising this concern early, keeping feedback constructive, and helping focus discussion about the trade-offs involved, the ETA liaisons were able to help the pilot develop a process that satisfied programmatic staff, the local evaluator, and federal agencies.

Experience shows that sometimes the program and evaluation staff see the same problem differently and the solutions must be jointly identified. The ETA provider can support both the grantee and the funder in their conversations about changes to interventions, outcomes, and evaluations. Helping both grantees and funders understand the trade-offs for potential changes to service models or evaluation designs is a critical function of the ETA provider.

The ETA provider is required to inform the funder of the progress of grantees/pilots, raising concerns as appropriate. P3 consulting agencies approved evaluation plans written by the pilots. Pilots needed to discuss any deviation from these plans—in terms of design, number served, outcomes collected, or intervention delivered—with their consulting agency. The ETA provider frequently provided the consulting agency background information on the reason for and potential effects of the proposed change. Pilots were not required to follow ETA provider guidance, unless required by the consulting agency to do so. In both YARH and P3, funders and the ETA provider used discussions about progress and concerns to identify topics for group-based ETA.

Our experiences with YARH and P3 grantees highlight the need for ETA providers to remain on task, be flexible, and think innovatively. Both initiatives seek to add to the evidence base on improving outcomes for youth and young adults. Our work with both initiatives suggests that grantees need time to start implementation prior to starting the evaluation, particularly when the intervention is new and has not be implemented elsewhere. As a program evolves rapidly during initial implementation, ETA providers need to adjust both their understanding of the program and their recommendations for how best to evaluate the program. ETA providers must help grantees, local evaluators, and funders identify new approaches when changes necessitate them and may need to play an active role in helping these different stakeholders justify changes to each other. YARH addressed grantees’ delays in being ready for a rigorous evaluation by adding (and then expanding) an initial implementation period. P3 pilots received programmatic TA in collaboration with the ETA. However, pilots still struggled to start an evaluation in time to learn something meaningful from the evaluation.

ETA is likely to increase capacity for the recipients, no matter whether it is an intended focus of the ETA or not. The opportunity to interact with other evaluators, discuss challenges, or learn of interesting solutions can only enhance skills. CB intended that YARH would help build evaluation capacity among grantees—including the local evaluators. P3 did not explicitly identify capacity building as a goal; however, the P3 Leads supported ETA that increased capacity.

ETA—Even in Early Stages—Can Be Helpful

Our experience as the ETA provider for both the YARH and P3 grant programs highlights important lessons learned. First, developing relationships between the ETA provider, funders, other agencies with a vested interest, and the grantees is imperative. The relationships enable the ETA provider to support the various parties in achieving their related but distinct goals. Second, ETA needs to be flexible. To be successful, ETA needs to include individual TA, which must be tailored to each grantee’s specific needs. Additionally, the format of ETA should vary to support the varied progress of grantees. ETA liaisons need to be innovative to help grantees structure evaluations that respect their practical constraints while maximizing the amount and quality of findings to be produced. This balances the goals of both the funders and grantees and is the most promising approach to expanding the breadth of research on and determining the effectiveness of programs and policies of improving participant outcomes.