Abstract
The Maternal, Infant, and Early Childhood Home Visiting (MIECHV) program, administered by the Health Resources and Service Administration in collaboration with the Administration for Children and Families, provides evidence-based home visiting services across 50 states, the District of Columbia, and five U.S. territories. MIECHV invests in comprehensive technical assistance (TA) to support and build the capacity of awardees to conduct rigorous evaluations of their programs. Throughout the course of the evaluation process, awardees received TA from the Design Options for Home Visiting Evaluation project. Between 2011 and 2020, over 173 state-led evaluations have been conducted. Individual technical assistance (TA) modalities included conference calls, emails, interactive and individualized webinars, developing and sharing resources, and involvement of content experts. When issues and challenges were identified across multiple awardees, Design Options for Home Visiting Evaluation (DOHVE) delivered targeted group TA to awardees with common needs that may benefit from peer-to-peer learning. When cross-cutting issues and challenges were identified, DOHVE used universal approaches such as webinars and guidance documents that were made available to all awardees.Through the Maternal, Infant, and Early Childhood Home Visiting (MIECHV) program, efforts have been taken to promote awardee capacity by targeting all phases of the evaluation process, including planning, implementing, and disseminating findings and providing TA that is responsive and tailored to meet awardee-specific needs. This approach enabled DOHVE to support MIECHV awardees in expanding knowledge of their programs and the evidence base on home visiting. Lessons learned from TA provision highlight the importance of developing feasible plans and providing ongoing support during implementation.
Home visiting programs support healthy family functioning by helping new and expectant parents access comprehensive services that improve outcomes for children and families. Evidence-based home visiting programs have been shown to improve child development, school readiness, and maternal mental health, among several other significant outcomes (Filene et al., 2013; Lugo-Gil & Tamis-LeMonda, 2008; Sama-Miller et al., 2017). The MIECHV program is a two-generation approach to support healthy families through evidence-based home visits during pregnancy and early childhood. MIECHV-funded state programs provide pregnant women, new parents, and families with tailored supports and education through trained professionals who partner with families to achieve their goals. In 2018, Congress reauthorized the MIECHV program (42 U.S.C. § 711), continuing its investments in evidence-based home visiting programs for an additional 5 years.
The Health Resources and Service Administration’s (HRSA’s) Maternal and Child Health Bureau, in collaboration with the Administration for Children and Families (ACF), awards MIECHV grants to eligible entities, which include states, territories, and the District of Columbia, 1 to carry out these programs. These entities are eligible for formula funds per the MIECHV authorizing statute that includes a base amount plus additional funding based on the state’s proportion of children under five living in poverty as calculated by the Census Bureau’s Small Area Income and Poverty Estimates. Through specific opportunities, recipients of MIECHV formula grants may be eligible to receive competitive grants to focus on the development, implementation, and evaluation of innovations to strengthen and improve the delivery of MIECHV-funded home visiting services. MIECHV has supported over 150,000 children and parents in receiving home visiting services in fiscal year 2018 through nearly 1 million home visits across 50 states, the District of Columbia, and five U.S. territories.
The MIECHV program prioritizes funding to implement home visiting models that have a demonstrated evidence of effectiveness and requires awardees to spend at least 75% of their grant funding to implement home visiting models that meet standards of evidence, as defined by the Department of Health and Human Services (42 U.S.C. § 711 (d)(3)(A)(ii)) Home Visiting Evidence of Effectiveness review. 2 This provision also supports innovation by allowing awardees to spend up to 25% of funds to implement rigorously evaluated home visiting models that have not met the criterion for evidence of effectiveness and thus qualify as promising approaches.
Program requirements for MIECHV emphasize both the use and the development of evidence to increase knowledge about the implementation and effectiveness of home visiting programs through a continuous program of research and evaluation. Awardees implementing promising approach home visiting models are required to rigorously evaluate those models. The purpose of evaluating promising approaches is to help build the evidence of those models. For all other formula awardees, funding an evaluation with a MIECHV grant is optional. Federal agencies overseeing MIECHV have long utilized a learning agenda approach to program implementation and improvement. This approach aims to “continually improve program performance by applying existing evidence about what works, generating new knowledge, and using experimentation and innovation to test new approaches to program delivery” (Office of Management and Budget, 2013, p. 1) and use evidence to support policymaking (Foundations for Evidence-Based Policymaking Act of, 2018, Pub. L. No. 115-435). As part of this learning agenda approach, MIECHV has supported the design, implementation, and use of locally focused evaluations of evidence-based home visiting programs. Through guidance provided in Notices of Funding Opportunities, HRSA encourages all awardees to conduct evaluations of their program that contribute to the awardee’s own understanding of their program and improve data-informed decision making. Awardees that propose an evaluation must ensure that the evaluation answers questions of interest to the awardee, includes an appropriate evaluation design, and meets expectations of rigor based on the ACF Common Framework for Research and Evaluation (Office of Planning, Research & Evaluation, 2016). Many awardees have long-standing partnerships with academic researchers to develop and execute rigorous studies focused on implementation, participant outcomes, and systems-level outcomes. The federal standards of rigor are summarized below (see Figure 1). Betwenn 2011 and 2020, over 173 state-led evaluations have been conducted.
Criteria for Rigorous Evaluations.
The MIECHV program invests in comprehensive TA to support and build all awardees’ capacity to administer and conduct grant-funded activities. The DOHVE project, funded by HRSA and led by ACF’s Office of Planning, Research and Evaluation (OPRE), operated between 2010 and 2020 as one of these federal TA investments and provided TA to MIECHV awardees implementing state-led evaluations. All awardees implementing evaluations received evaluation TA at varying levels of frequency and at specific touchpoints in the evaluation process.
The DOHVE team was comprised of professionals with advanced degrees in public health, psychology, sociology, education, child or human development, or other related fields. Staff had a broad understanding of early childhood and experience in program evaluation and applied research. The project used regional assignments of TA liaisons to ensure each awardee received guidance from a DOHVE liaison with whom the awardee could build a collegial working relationship. These relationships enabled rapid, accurate identification of each awardee’s strengths and weaknesses. The staffing structure relied on collaboration and coordination with HRSA project officers, OPRE evaluation staff, and other TA providers to develop a TA plan to meet each state awardee’s individual needs and priorities. This article addresses the aspects of the evaluation process and approach that DOHVE TA supported. Examples of resources developed for TA include the following: developing a state learning agenda (Till & Zaid, 2019), ensuring quality evaluations (Zaid & Sparr, 2018), and working with an external evaluator (Sparr & Zaid, 2017).
Framework for DOHVE TA Developmental Approach
The overarching goals of DOHVE TA were to build the capacity of awardees, both project directors and evaluators, to conduct rigorous and relevant evaluations, make data-informed decisions about their programs, and disseminate information within their states and to the broader field of home visiting. Many factors may influence the success of home visiting programs, such as geographical area, populations served, home visiting workforce, and system partners. The evaluations helped show how models can be replicated in new settings. The DOHVE approach recognized that variations in awardee context and evaluation capacity influence evaluation implementation (see Figure 2) and therefore provided TA that was specific to an awardee’s needs. Using a developmental framework, DOHVE TA aimed to build the capacity of awardees by providing support throughout all phases of the evaluation process, including planning, implementing, and disseminating findings. The framework for TA includeds universal, group, and individualized TA to build capacity at each key phase of the evaluation.
Sources of variability across awardees.
DOHVE’s approach includes the following guiding principles (see Figure 3): Understanding awardee capacity. Assessing awardee strengths and opportunities for improvement was key to providing TA that was applicable. The DOHVE project assessed capacities and developed TA goals to fit individual contexts and capacities (Wandersman et al., 2012). DOHVE liaisons met with the awardee evaluation team, reviewed drafts of the evaluation plan, and discussed ways to strengthen the evaluation to align with federal expectations of rigor. Incorporation of adult learning principles. To ensure awardees were fully engaged in and benefitted from TA, the DOHVE project used principles of adult learning including the following: (a) TA that was applicable and connected to existing capacities and priorities, (b) collaborative relationships that emphasized awardee autonomy and skill-building, (c) clear identification and communication of the purposes/goals of TA, and (d) reliance on team members’ technical and content expertise to enhance and expand capacities and knowledge. Accessible, proactive, and responsive TA. The DOHVE team provided accessible TA by using multiple modes, with different levels of support (e.g., phone calls, web meetings, site visits, facilitated peer-learning opportunities) to awardees depending on the intensity of TA needs, and by hosting a central online space for awardees to easily access TA tools (e.g., briefs, checklists, guidelines, webinars, FAQs). Knowledge gained from working closely with each awardee allowed the DOHVE project to provide proactive TA that addressed common challenges faced by all awardees and developed universal support for needs awardees may have. Examples include the development of evaluation plan checklists awardees can use prior to submission and communities of practice for evaluators to discuss evaluation implementation successes and challenges.
Design Options for Home Visiting Evaluation 2 guiding principles to interacting with stakeholders.
Comprehensive Plan for Awardee Support
The DOHVE project used a tiered and phased approach to meet awardees’ TA needs (see Figure 4), starting with identifying TA needs. The team used formal and informal assessments of awardee needs to identify TA priorities and goals. Such assessments included informal semi-structured interviews, formal TA needs assessment interviews, reviews of awardee plans and reports, and assessments of awardee satisfaction and utilization. Once TA needs were identified, DOHVE liaisons were in close communication with federal partners and the awardee to understand expectations, prioritize TA needs, and select the appropriate modality for individual TA to address prioritized needs. Individual TA modalities included conference calls, emails, interactive and individualized webinars, development and sharing of resources, and involvement of content experts to discuss a range of topics including evaluation design considerations and best practices using specific methodologies. When issues and challenges were identified across multiple awardees, DOHVE used targeted group TA to awardees with common needs who may benefit from peer-to-peer learning. Topics such as utilizing a state learning agenda and considerations for participatory evaluations were featured in communities of practice for evaluators to promote peer dialogue and share practical lessons learned. When cross-cutting issues and challenges were identified, DOHVE used universal approaches such as webinars and guidance documents that were made available to all awardees. Examples of resources that addressed universal needs can be found on the DOHVE Resource Page [https://www.jbassoc.com/project/design-options-home-visiting-evaluation-dohve/].
Tiered and phased approach to technical assistance.
TA Provision Across Evaluation Phases
Evaluation Design and Planning Phase
Awardees proposing evaluations with MIECHV funds are required to submit a comprehensive evaluation plan for approval by HRSA prior to beginning evaluation activities. The DOHVE project used an evaluation plan review checklist to systematically review the study design, methodology, and analysis plan to ensure all studies met the federal standards of rigor outlined in the funding requirements. Liaisons provided detailed written feedback on each plan and worked with awardees individually to strengthen plans to the extent possible. Often, contextual factors, such as sample size limitations, affected evaluation decisions on the design and methodology. Understanding these contextual factors was important in balancing the desired rigor of the plan while ensuring the awardee evaluation addressed topics relevant to their state.
In addition to individual TA provided to awardees on their evaluation plans, DOHVE developed resources including checklists, tip sheets, briefs, and webinars to universally support awardees in developing strong evaluation plans.
Evaluation Implementation Phase
To support the implementation of evaluations as planned, DOHVE liaisons and federal project officers met with awardees and their contracted evaluators regularly during the implementation of the evaluation. While all awardees were provided universal resources, quarterly evaluation calls with awardee evaluation teams provided an opportunity to discuss the unique challenges faced during the implementation of the evaluation and often led to targeted individual TA to address unique awardee needs. This provided an opportunity to identify deviations to the approved plan and address challenges. By addressing challenges early on during implementation, awardees often had the opportunity to course correct or adjust to new circumstances. For example, a common challenge was recruiting enough participants to provide sufficient power for the planned analyses within the timeframe of the evaluation. When challenges could not be overcome and compromised the quality of the study, the DOHVE team worked with awardees and their evaluators to revise the evaluation in a way that could feasibly be completed while maintaining rigor to the extent possible.
Evaluation Final Report Phase
Awardees submit evaluation final reports upon the completion of the grant. Supports provided to awardees included individual TA to ensure reports adequately addressed federal expectations as well as universal supports such as webinars and checklists to ensure all awardees understood the final report expectations.
At the end of each grant funding cycle, DOHVE reviewed and synthesized key evaluation components from awardee-led evaluation final reports. This provided an opportunity to understand evaluation strengths and limitations and to identify opportunities for universal TA support. DOHVE summarized 160 awardee evaluation final reports from funding year 2011 through funding year 2017. By understanding common challenges faced by awardees in each cohort of completed studies, DOHVE proposed universal TA opportunities in coming years to specifically target these challenges, which included a series of tip sheets addressing common evaluation challenges and peer learning exchanges to leverage awardee strengths.
Dissemination Phase
DOHVE promoted a research-to-practice approach by identifying opportunities for awardees and their evaluators to share findings with other awardees and within the field of home visiting through webinars, TA resources, grantee meetings, and featuring awardee findings in journal special issues. This began with the review of awardee evaluation plans where DOHVE liaisons ensured awardees include a dissemination plan that promoted sharing of findings with stakeholders at multiple levels, ranging from program administrators and program staff to the home visiting field more broadly, and continued after the evaluation final report had been submitted.
Areas of Success
DOHVE has supported over 170 evaluations across 48 awardees. 3 These state-led evaluations have allowed awardees to (1) answer questions that are important to their program and solve local problems, (2) show how home-visiting implementation and outcomes may be replicated and scaled up in diverse contexts, and (3) contribute new knowledge, both individually and collectively, by identifying trends across similar studies (Zaid & Till, 2016). Findings from completed studies highlight factors contributing to participant engagement, the influence of professional development on job satisfaction, barriers to service utilization, effectiveness of a coordinated intake system, development of early childhood coalitions, and the benefits of continuous quality improvement and have been featured in the MIECHV Special Issue: Taking Home Visiting to Scale: Findings from the Maternal, Infant, Early Childhood Home Visiting Program State-Led Evaluations (Denmark et al., 2018).
Through the review of awardee evaluation final reports, DOHVE identified common strengths among state-led evaluations from funding year 2011 through funding year 2017. There were many strengths identified across these completed evaluations, three of which are highlighted below. Triangulation of data to enhance the methodological rigor of studies. Many awardees triangulated data in study components through both data source triangulation (e.g., program staff, MIECHV and non-MIECHV program participants, stakeholders, and home visiting model representatives) and methodological triangulation (e.g., observations, interviews, focus groups, document reviews, surveys, and validated tools) as a strategy to enhance methodological rigor and add insight to the topic of interest. Triangulation helped validate the findings by providing support from two or more sources. In addition, many studies complemented the triangulation of data with the use of a participatory evaluation approach to engage stakeholders in all phases of the evaluation process, which further strengthened the evaluation design. Awardees noted that the evaluation findings contributed to data-informed decision making about program operations. Specifically, many awardees noted findings informed the development of new ways to support local programs, refinement of data collection tools, and improvements in implementation approaches. Findings were also used to justify the need for specific service components, such as funding for infant mental health specialists. This suggests ongoing communication and data sharing among the evaluators, management, and program staff about the evaluation can improve program performance. Many awardees leveraged secondary administrative data sources to inform findings. For some awardees, secondary administrative data allowed for larger sample sizes than what they would have been able to gather from other data sources. Administrative data also helped awardees minimize data collection burden and potential biases than from other methods of data collection.
Discussion
Several lessons have been gleaned from 10 years of DOHVE TA. First, while universal TA can be a practical and effective way to disseminate information to all awardees, variations in awardee strengths and needs underscore the need to tailor TA to individual awardee capacities. Understanding local context and realities of implementing the evaluation is critical to providing TA that is accessible and meets specific awardee needs. This highlights the importance of offering TA with multiple modalities to reach awardees with varying needs. Individual TA has provided an opportunity to meet awardee needs by addressing contextual issues and problem-solving unique awardee challenges. Targeted group TA has provided an opportunity for awardees to learn from and share with their peers, which led to the communities of practice for evaluators and other networking opportunities to promote peer sharing. Universal TA provides an opportunity to disseminate technical support uniformly to all awardees.
During the first DOHVE contract (2010–2016), DOHVE support to awardees focused on the first and last phase of the evaluation process, developing evaluation plans and submitting final reports summarizing the evaluation. Unfortunately, many well-designed studies were not implemented as planned, due in part to unexpected external factors and time constraints. Many awardees encountered delays in procuring a contracted evaluator and obtaining federal approval on an evaluation plan that sufficiently met the federal expectations for rigor, which delayed the onset of evaluation activities and successful completion of the proposed evaluation within the original timeline. These delays may have prevented some awardees from fully answering the proposed research question but have still led to preliminary findings that informed program understanding and built evidence to frame a subsequent evaluation. To respond to these awardee challenges, HRSA and OPRE recognized the need for ongoing TA support throughout the evaluation process to proactively identify and address struggles and issues during implementation. This support to awardees was built into the work plan of the second contract of the DOHVE project (2016–2020) and allowed DOHVE and federal staff the opportunity to check in with all awardees implementing evaluations several times a year to identify awardees who may need additional support during implementation. A critical lesson learned was that in applied research, post hoc changes may need to be made to address real-world constraints, but these changes may be mitigated by offering ongoing support during implementation. Even though changes during evaluation implementation must often be made, those evaluations can still generate findings that are relevant and inform decision making.
To further respond to challenges in completing evaluations within the funding period, DOHVE liaisons increased support to awardees during the planning phase to help awardees develop feasible plans and identify any pre-evaluation work that might be needed to generate internal buy-in for the study. TA efforts have also increasingly emphasized using evaluations to build a continuum of evidence where evidence may be gathered through iterative studies. In many cases, programs are too complex to be able to study in one evaluation. TA has increasingly focused on creating state learning agendas to help awardees generate findings across studies that are useful and can inform decision making.
Conclusions
The MIECHV program has been successful in supporting knowledge building at the local level and to the broader field through supporting states’ development of evaluations. Notably, awardees implementing evidence-based models are encouraged but not required to engage in evaluation, yet nearly half of awardees chose to do so in the last 2 years. Through the DOHVE project, the MIECHV program has been able to promote awardee capacity building by targeting all phases of the evaluation process, including planning, implementing, and disseminating findings. TA was responsive and tailored to meet awardee-specific needs by using a developmental approach to support awardees throughout the evaluation and by process improvements made from lessons learned. This approach enabled DOHVE TA to support MIECHV awardees and their evaluators in expanding their knowledge of their programs and the evidence base on home visiting.
Supplemental Material
Supplemental Material, sj-pdf-1-erx-10.1177_0193841X21992196 - Using a Developmental Approach to Evaluation Capacity Building in Home Visiting
Supplemental Material, sj-pdf-1-erx-10.1177_0193841X21992196 for Using a Developmental Approach to Evaluation Capacity Building in Home Visiting by Susan Zaid, Mariel Sparr, Kyle Peplinski, Nicole Denmark and Pooja Gupta Curtin in Evaluation Review
Footnotes
Authors’ Note
The views expressed in the publication are solely the opinions of the authors and do not necessarily reflect the official policies of the U.S. Department of Health and Human Services, the Health Resources and Services Administration, or the Administration for Children and Families nor does mention of the department or agency names imply endorsement by the U.S. Government.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research and/or authorship of this article: Design Options for Home Visiting Evaluation is funded under Contract No. HHSP233201500133I.
Supplemental Material
The supplemental material for this article is available online.
Notes
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
