Implementing mental health interventions within a national nurse home visiting program: A mixed-methods evaluation

Abstract

Background

Up to half of low-income mothers experience symptoms of depression and anxiety that affect their well-being and increase their children's risk for behavioral and emotional problems. To address this problem, an engaged research/practice planning team designed the Mental Health Innovation (MHI), a multicomponent implementation strategy that integrates evidence-based mental health interventions within the national Nurse-Family Partnership (NFP). The MHI includes four implementation strategies: online training modules, clinical resources, team meeting modules, and virtual consultation.

Methods

A convergent, mixed methods observational design was applied to evaluate implementation outcomes, guided by the RE-AIM framework. We operationalized Reach as the number and demographics of women enrolled in NFP agencies exposed to MHI strategies. Adoption was operationalized as the number and proportion of nurses and supervisors who used MHI implementation strategies. For implementation, we assessed multilevel stakeholders’ perceptions of strategy acceptability and feasibility. Data were pulled from NFP's national data management systems and collected through focus groups and surveys. Quantitative data were analyzed using counts and summary statistics. Qualitative themes were generated through content analysis.

Results

The MHI reached agencies serving 51,534 low-income mothers (31.2% African American and 30.0% Latina). Adoption rates varied across implementation strategies. Between 60% and 76% of NFP nurses (N = 2,100) completed each online module. Between 27% and 51% of nurse supervisors (n = 125) reported using each team meeting module. Of 110 teams invited to participate in virtual consultation, 40.9% (n = 45) participated. Mothers served by agencies participating in virtual consultation differed significantly from those who did not, with lower percentages of African American and Latina. Qualitative themes suggest that MHI strategies were generally viewed as acceptable; perceptions of feasibility varied across strategies.

Conclusions

This study identified both strengths and opportunities for improvement. Further evaluation is needed to assess the MHI's effectiveness in improving mothers’ mental health.

Plain Language Summary

Up to half of low-income mothers experience symptoms of depression and anxiety. To reach these mothers, this study implemented evidence-based mental health interventions in a national nurse home visiting program that provided services to over 51,000 low-income mothers (31.2% African American and 30.0% Latina). The study aimed to build nurse- and agency-level capacity to identify, support, and refer mothers with mental health symptoms. Four strategies were used to implement mental health interventions in the nurse home visiting program: online training modules, team meeting modules, clinical resources, and virtual consultations. This paper reports findings from an evaluation of the reach, adoption, and perceptions of these strategies. Study findings indicate that nurses and their supervisors generally valued the implementation strategies but were more likely to adopt some strategies than others. The majority of nurses adopted online modules, which were perceived as easy to use and were required by the national nurse home visiting program. Less than half of nurse supervisors adopted team meeting modules or virtual consultation. Supervisors noted concerns about the feasibility of using both strategies. Findings from the evaluation were applied to further improve implementation strategies. Future evaluation is needed to determine how well the refined strategies work and whether they impact nurse home visitors’ capacity to address the mental health needs of low-income mothers.

Keywords

maternal and child health perinatal postpartum implementation strategies RE-AIM mental health

Up to half of low-income mothers experience symptoms of depression and anxiety (Beeber et al., 2014b; Pooler et al., 2013). These symptoms not only affect the well-being of mothers, but also affect the quality of mothers’ interactions with their infants and young children and increase their children's risk for behavioral and emotional issues across the lifespan (Ciciolla et al., 2014; Holditch-Davis et al., 2015; Park et al., 2018). Interventions to reduce these symptoms are essential to improve the well-being of low-income mothers and their children. In this paper, we report findings from a project that implemented evidence-based mental health interventions within the Nurse-Family Partnership (NFP), a nationwide nurse home visiting program for low-income mothers. The purpose of this project was to design, implement, and evaluate the Mental Health Innovation (MHI), a multicomponent strategy to implement evidence-based mental health interventions into the care provided by NFP nurses. In this paper, we report findings from an evaluation of the MHI's implementation outcomes.

For over four decades, NFP nurses have provided support to low-income women during their pregnancy and their child's first two years of life. During home visits, nurses provide education, referrals, and other support to promote healthy pregnancies and healthy mother–child attachment relationships. Multiple studies of the NFP model provide evidence of its effectiveness in improving the health, well-being, and development of both mothers and their children (Kitzman et al., 2010, 2019; Olds, Henderson, et al., 1998; Olds, Kitzman, et al., 2007). As of 2020, the NFP was providing services to mothers in 40 states, five tribal communities, and the US Virgin Islands.

The original NFP model of care included little guidance for nurses on how to address mothers’ mental health needs. NFP leadership identified this as a high-priority concern following a national survey of NFP nurses found that many felt unprepared to address the high prevalence of depressive and anxiety symptoms they were encountering in their clients (Beeber et al., 2022). To close this gap, NFP leadership partnered with experts in maternal and child mental health to create the MHI. The nine-year, stakeholder-engaged process used to design the MHI is described in a prior publication (Beeber et al., 2022). In brief, the NFP engaged a planning team that included experienced staff from NFP's National Service Office (educators and nurses), researchers with expertise in infant mental health and mental health interventions for low-income mothers, and researchers from the University of Colorado Denver Prevention Research Center for Family and Child Health, a long-time partner of the NFP. Starting in 2010, the team met weekly to develop and pilot test elements of the MHI. In 2018, an implementation scientist was added to the team.

The planning team was led by Linda Beeber who introduced the group to the EBIs she was using or had used in multiple, federally funded nurse-delivered, home-based mental health intervention studies with low-income mothers of young children (Beeber, Holditch-Davis, et al., 2010; Beeber, Perreira, et al., 2008; Beeber, Schwartz, et al., 2013, 2014a). In this prior research, Beeber drew on evidence-based approaches derived from mindfulness-based cognitive behavioral therapy (Hofmann & Gómez, 2017), social support/social network mobilization (Bryan et al., 2019; Li et al., 2017), behavioral activation (Knittle et al., 2019), and anticipatory guidance/advance directives (Substance Abuse and Mental Health Services Administration [SAMHSA], 2019). Building on Beeber's prior research and these four evidence-based approaches, the planning team identified a menu of EBIs and formatted them to align with nurses’ professional scope of care and the six dimensions promoted by the NFP model (individual health, maternal role, life course development, close relationships, environment and health, and service usage). For example, a set of interventions guide the nurse and client in an assessment of the client's available social support, followed by strengthening ties with viable supporters and developing new supports where there are deficits (Bryan et al., 2019; Li et al., 2017). Throughout the MHI, the team created crosswalks between the mental health EBIs and existing NFP interventions to identify a wider range of symptom-reducing interventions. In the example above of social support mobilization, the MHI social support interventions are expanded by crosswalks to appropriate existing NFP interventions that the client can use depending on the results of the social support assessment. These interventions include steps to communicate effectively with helpers, reducing conflict with close family members, using problem-solving to get resources, and evaluating relationship choices.

The planning team created four implementation strategies to integrate the identified mental health EBIs into practice. Over multiple years of formative work and pilot testing, the planning team embedded the EBIs within each of the strategies such that the MHI is a composite of both EBIs and implementation strategies. The team designed the MHI strategies to fit within National Service Office infrastructure (e.g., online educational management system), and to be used along with other NFP tools (e.g., Strength and Risk Framework for assessment [STAR]; Beeber et al., 2022). The four strategies are described in greater detail below and in Table 1, following reporting guidelines outlined by Proctor et al., (2013).

Table 1.
The four implementation strategies that comprise the MHI.

MHI strategy Named using ERIC terminology Actor Action Dose Timing

Online modules Conduct ongoing training NFP National Service Office Posted to national website, required completion Five 20–35 min modules Posted January 2018

Clinical resources Develop and distribute education materials NFP National Service Office Posted to national website NA Posted January 2018

Team meeting modules Promote network weaving NFP National Service Office
Agency-level supervisors Posted to national website
Facilitated team meetings Five 60–90 min interactive activities Posted January 2018

Virtual consultations Facilitation Mental Health Experts Facilitated interactive webinar discussions with multiple teams Six 60–90 min interactive activities Two waves, June–November 2018 and January–June 2019

Note. MHI = Mental Health Innovation; ERIC = expert recommendations for implementing change (Powell et al., 2015), NFP = Nurse-Family Partnership.

Implementation Strategy 1: Online modules. The planning team created five self-directed online modules to build the knowledge, skills, and confidence nurses need to address their clients’ mental health symptoms. The modules were designed to introduce didactic content followed by interactive unfolding case studies in which nurses chose responses and interventions for a client and received immediate feedback on the client's response. If the nurse chose a less optimal intervention, the module was programmed to provide immediate feedback about the reasons why the choice was not optimal and to direct the nurse back to the case to try another choice. Other pedagogical mechanisms embedded in the modules included deliberate repetition of key content to which more complex practice examples and practice-based examples of how to apply clinical resources were added (described below).

Implementation Strategy 2: Clinical resources. The team created 11 clinical resources for nurses to use with their clients to assess symptoms, develop care plans, manage symptoms, and refer them for mental health services. Examples of these resources include a crisis management decision aid, suicide risk assessment, and mindfulness activities for nurses to use with their clients (e.g., a one-page handout for mothers on how to “Breathe Those Thoughts Away”).

Implementation Strategy 3: Team Meeting Modules. The NFP requires all participating local agencies to convene weekly supervisor-led team meetings with nurses. The planning team created packages of materials for nurse supervisors to structure six team meetings to facilitate discussions. Team members discussed what they learned in the online modules, how they used clinical resources, and shared what had and had not worked in their efforts to integrate mental health into practice. The topics of the team meeting modules addressed the barriers nurses identified during pilot studies of the MHI's online modules and clinical resources. Examples of topics addressed by the modules include “Legal Liability: Crisis Prevention and Management” and “Screening and Sharing Screening Feedback.”

Implementation Strategy 4: Virtual Consultations. Two experts in mental health nursing care (LB and PZ) provided coaching to NFP teams in six interactive videoconferences. With guidance from nurses and supervisors in the field, the team designed the consultations to reinforce topics covered in the team meeting modules. Consultants met with cohorts of three to six geographically diverse teams monthly for 6 months and encouraged teams to bring client examples to each consultation. Every consultation included expert guidance and peer discussion to reinforce what was working well and to share approaches to overcoming barriers to using MHI interventions and clinical resources with clients. Each consultation also included nurse self-care strategies. Written summaries of each consultation were sent to participating teams.

Evaluation Framework

The RE-AIM framework guided our evaluation of MHI implementation (Figure 1; Glasgow, Harden, et al., 2019; Glasgow, Vogt, et al., 1999). RE-AIM specifies metrics for assessing an innovation's reach (R), effectiveness (E), adoption (A), implementation (I), and maintenance (M). In applying RE-AIM, attention was given to whether reach was equitable across population subgroups (e.g., across racial/ethnic subgroups; Shelton et al., 2020). The rationale for selecting RE-AIM is that intervention effectiveness is necessary but not sufficient. To improve health at the population level, an intervention also needs to reach the intended population. Furthermore, to reach the intended population, the providers and settings that work with that population need to adopt, implement, and maintain the intervention over time (Glasgow et al., 2019). As summarized in Figure 1, the goal of the MHI was to integrate evidence-based mental health interventions into the NFP model using four implementation strategies. The purpose of this evaluation was to assess the impact of those strategies on implementation outcomes (reach, adoption, implementation, and maintenance). A future paper will report the study's effectiveness outcomes.

Figure 1.
Evaluation framework.

Methods

Design

The engaged research/practice planning team that developed and implemented the MHI (Beeber et al., 2022) also oversaw its evaluation. The team applied a convergent, mixed-method observational design to evaluate implementation outcomes. The study was reviewed by the University of North Carolina Institutional Review Board and determined to be quality improvement and not human subjects and was therefore exempt entirely from human subjects research oversight.

Settings and Sample

In January 2018, the NFP National Service Office (NSO) rolled out three of the MHI's four implementation strategies to 264 NFP agencies nationwide: online modules, clinical resources, and team meeting modules (basic MHI). These agencies were home to 359 teams, each of which included between four and eight nurse home visitors and a supervisor. The NSO required all NFP nurse home visitors and supervisors (N = 2,100) to complete the online modules and recommended use of MHI clinical resources and team meeting modules. Nationwide delivery of the fourth strategy (virtual consultation) was not feasible within existing resources. We therefore delivered it to a subset of randomly selected NFP agency teams (enhanced MHI). The planning team promoted MHI strategies through brief reports in the NFP monthly online newsletter and updates at NSO Nursing and NFP Leadership team meetings.

Measures

Reach

The reach of the basic MHI was operationalized as the number and demographics of women enrolled in NFP during the implementation period (January 2018–February 2020). The reach of the enhanced MHI was assessed as the absolute number of women enrolled in agencies that received virtual consultation and the extent to which these women were representative of the overall NFP population (rural/urban, race/ethnicity). Data were pulled from the NFP's national data management system, which maintains detailed records on all women enrolled in the program.

Adoption

Adoption of the MHI was operationalized as the number and proportion of NFP nurses and supervisors who completed each of the online modules, number and proportion of supervisors who reported using each of the team meeting modules, and number and proportion of invited NFP teams who participated and level of participation in consultation. Data on online module completion were pulled from the NFP's national education management system. Data on supervisors’ use of team meeting education modules were collected through a Qualtrics survey of all NFP nurse supervisors (Spring 2019). To assess adoption of consultation, the study coordinator maintained a tracking log of teams that were invited and teams that declined.

Implementation

To assess implementation, we used focus groups and surveys to assess stakeholder perceptions of the MHI's acceptability and feasibility. Acceptability was defined as the perception among stakeholders that the MHI is “agreeable, palatable, or satisfactory” and feasibility as the extent to which the MHI “can be successfully used or carried out” (Proctor et al., 2011).

Focus Groups

We conducted two focus groups with a convenience sample of 26 regional NFP consultants, each of whom provides consultation to multiple NFP agencies (Spring 2020). Focus groups lasted 45 min during which NFP consultants were asked to respond to a series of questions related to their perceptions of NFP nurses’ use of the MHI, what worked well, what was challenging, and suggestions for improvement.

Surveys

We collected data through two open-ended questions in an online Qualtrics survey of all NFP supervisors (Spring 2019) asking them why they did or did not use MHI strategies.

Maintenance

To assess potential for maintenance of the MHI over time, seven members of the engaged planning team completed the Clinical Sustainability Assessment Tool (Fall 2020; Malone et al., 2021). The team used findings from the assessment to focus a team discussion on additional strategies needed to maintain the MHI over time. Planning teams use the tool to assess the presence of factors known to be associated with the sustainment of new interventions or practices, using a seven-point Likert Scale. We adapted the tool to address the six domains most relevant to MHI sustainment. The adapted tool included 27 items within the following six domains: engaged staff and leadership, organizational readiness, workflow integration, implementation and training, monitoring and evaluation, and outcomes and effectiveness.

Analysis

Quantitative data were analyzed using counts and summary statistics. For the qualitative data, focus group interviews were recorded, transcribed, and entered into Atlas.ti (version 8) for analysis. Two members of the study team (JL and EB) applied conventional content analysis to code both transcripts and open-ended responses to survey questions (Hsieh & Shannon, 2005). Coding focused on participants’ descriptions of how they used MHI strategies and perceptions of their acceptability and feasibility. Coders read through transcripts and survey responses and developed a code book. They independently applied codes and then met to reconcile coding and generate themes. The research/practice planning team reviewed themes to confirm coherence and credibility of the findings. For the sustainability assessment, findings were summarized as counts of the number of team members who gave the NFP a rating of 4 or lower in response to whether the organization has or does each item on the survey. The scale ranged from 1 (to little or no extent) to 7 (to a very great extent). The team reflected on findings and generated recommendations to address gaps. Recommendations were captured by a note taker and reported back to the team to confirm consensus.

Results

Reach

The national roll out of the basic MHI included agencies that served 51,534 low-income mothers of whom 31.2% were African American and 30.0% were Latina (Table 2). The enhanced MHI included 45 agencies that served 3,500 low-income mothers. The mothers served by agencies that agreed to participate in virtual consultation (i.e., enhanced MHI) were significantly different from those served by agencies that declined (p = .0001). As summarized in Table 2, agencies that agreed to participate in virtual consultation served a smaller percentage of African American and Latina women and a larger percentage of White women than those who declined to participate.

Table 2.
Mental Health Innovation (MHI) reach to first-time mothers through agency participation.

Race Basic MHI number of mothers (%) Enhanced MHI number of mothers (%) Declined enhanced number of mothers (%) P-value

Whole sample 51,534 3,500 6,344

American Indian/Alaskan Native 1,165 (2.3%) 120 (3.4%) 129 (2.0%)

Asian 1,213 (2.4%) 76 (2.2%) 113 (1.8%)

Native Hawaiian/Pacific Islander 232 (0.5%) 23 (0.7%) 19 (0.3%)

African American/Black 16,071 (31.2%) 661 (18.9%) 2,719 (42.9%)

Caucasian/White 23,746 (46.1%) 2,132 (60.9%) 2,329 (36.7%)

More than one race 3,408 (6.6%) 258 (7.4%) 405 (6.4%)

Unreported/unknown 5,699 (11.1%) 230 (6.6%) 630 (9.9%)

Enhanced MHI versus declined enhanced <.0001

Ethnicity

Hispanic or Latinx 15,473 (30.0%) 1,122 (32.1%) 1,613 (25.4%)

Enhanced MHI versus declined enhanced <.0001

Adoption

As summarized in Table 3, between 60% and 76% of NFP nurses (N = 2,100) completed each of the online modules; completion rates declined for each successive module and were lowest for the final module. For team meeting modules, 40% of supervisors completed the survey (n = 151 of 359). Between 27% and 51% of these nurse supervisors reported using each of the team meeting modules. Rates peaked in the second module, which addressed nurses’ legal liability and crisis prevention and management, and then declined for each successive module. Of the 110 teams invited to participate in virtual consultation, 45 (40.9%) participated in at least one meeting and 33 of those (73.3%) attended all six sessions.

Table 3.
Adoption of online modules, team meeting modules, and consultation.

Mental Health Innovation (MHI) strategy Number (%)

Online modules (N = 2,100 nurses)

Module 1: Mental health introduction 1,586 (76)

Module 2: Learning about depression and anxiety part I 1,606 (76)

Module 3: Learning about depression and anxiety part II 1,557 (74)

Module 4: Integrating mental health 1,394 (66)

Module 5: Practicing mental health nursing care 1,262 (60)

Team meeting modules (N = 125 supervisors)

Module 1. Role of the nurse home visitor 51 (40.8)

Module 2. Legal liability: crisis prevention and management 64 (51.3)

Module 3. Intervention planning: adapting visit content and materials 50 (40.0)

Module 4. Screening and sharing screening feedback 44 (35.2)

Module 5. Mental health case conference 36 (28.8)

Module 6. Beyond integration, sustaining nurse, agency, and community impact 34 (27.2)

Virtual consultation (N = 110 teams invited)

Agreed to participate and attended at least one videoconference 45 (40.9)

Attended all six videoconferences 33 (73.3)

Implementation

Below, we summarize findings on the acceptability and feasibility of the MHI overall and for each of the four implementation strategies.

Acceptability

Participants’ perceptions of the MHI were generally positive. Supervisors valued the way strategies built on and reinforced each other. Team meeting modules reinforced the training provided in the online modules and provided a structure for nurses to share their experiences using clinical resources. For example, one supervisor reported,
So doing the integration activities [team meeting modules] provided a good review of information, gave discussion opportunity about integrating what we learned into our practice, and helped us take the time to really go through the helpful items in the resource packages that were provided with the online mental health education modules.

Supervisors who participated in the virtual consultations valued the opportunity to learn from nurses with expertise in mental health and from their peers at other agencies. NFP regional consultants reported that teams who participated in the virtual consultations were more confident in their ability to use MHI interventions and resources. Of note, supervisors who participated in virtual consultations identified them as a primary motivator for completing the team meeting modules. One supervisor reported,
We are in the midst of the MH Consultation calls. Prior to each call we are completing the MH Team activity [team education modules]. We are finding them very helpful. We have 3 more calls and usually complete the activity prior to the call. We have not done the activities before due to time constraints.

In contrast to participants’ generally positive perceptions of the MHI, a few nurse supervisors reported the MHI did not fit their agency, either because the information was too basic, they needed resources for a range of mental health disorders beyond just depression and anxiety, or they had a mental health professional on staff.

Feasibility

Overall, supervisors found the MHI time-consuming and hard to prioritize given multiple competing demands, with one supervisor reporting they had “No time to do extras.” Supervisors and NFP consultants identified several other NFP initiatives that competed with the MHI for time and energy, including an intervention to address intimate partner violence and a change to NFP's data entry system, among others. Many agencies also were working to increase caseloads and taking on more high-risk mothers, further adding to the demands on nurses’ time. In addition, many nurses and their supervisors reported they were new to their jobs, or their agencies were in transition (e.g., reorganizing or dealing with a loss of funding).

Participants’ perceptions of feasibility varied across the four MHI strategies. The fact that the NFP national office required the online module supported nurses’ taking time to complete them. On the other hand, NFP consultants and supervisors reported difficulty locating clinical resources on the NFP website and nurses had difficulty accessing resources online during visits to rural communities with limited internet coverage. Both team meeting modules and virtual consultations were time-consuming. Participants appreciated that team meeting modules were designed to occur during already scheduled team meetings. However, modules competed with other topics they needed to cover during team meeting time. Furthermore, some supervisors could not remember hearing about team meeting modules or had forgotten they were available.

Maintenance

The engaged planning team gave consistently positive scores to two of the six domains of the Clinical Sustainability Assessment Tool. Within the “Outcomes and Effectiveness” domain, team members endorsed the MHI as effective and as having a clear advantage over alternatives. Within the “Engaged Staff and Leadership” domain they scored the MHI as having the support and engagement of multiple levels and types of staff within the NFP. Scores for the other four domains were either mixed or consistently low. The team gave low scores to items in the “Organizational Readiness” domain related to sufficient staffing and resources, with team members particularly concerned about the lack of national-level NFP staff with expertise in mental health. Within the “Implementation and Training” domain, the team gave a low score to an item related to ongoing coaching, feedback, and training on the MHI. Within the “Workflow Integration” domain, low scores were given to items pertaining to how well the MHI was built into the NFP workflow and ease of use. Workflow concerns included limited training and unclear roles for NFP regional consultants as well as the lack of MHI integration with training and resources for other, related NFP practice priorities (e.g., intimate partner violence, substance use, and parenting). Finally, within the “Monitoring” domain, the team gave moderately low scores to all items, including items addressing the presence, use, and reporting of process and outcomes metrics related to MHI implementation and effectiveness.

Discussion

The MHI provided training and resources to nurse home visiting programs that served over 51,000 low-income mothers nationwide, the majority of whom were African American and/or Latina. NFP consultants and supervisors valued the MHI for providing new knowledge, skills, and resources to address mothers’ symptoms of anxiety and depression. Building on extensive formative work (Beeber et al., 2022), an engaged research/practice planning team designed the MHI to include four implementation strategies, with the goal that they work synergistically. The team designed online modules and clinical resources to provide nurses with knowledge, skills, and resources needed to address mothers’ mental health symptoms. The team designed team meeting modules and virtual consultations to reinforce the newly acquired knowledge and skills by providing opportunities for application, discussion, peer learning, and expert guidance and reassurance. Survey findings suggest that nurses who engaged in all four strategies valued the way strategies built and reinforced each other. However, only a minority of nurses engaged in all strategies. In this evaluation, we found that although most nurses were introduced to new knowledge and skills, fewer participated in the team meeting and virtual consultation strategies designed to reinforce the new learning. Extensive research suggests that education alone is likely to achieve only small improvements in practice (Forsetlund et al., 2021). Education is more likely to be effective when coupled with interactive discussion and support for problem-solving, provided by both team meetings and virtual consultations (Baskerville et al., 2012). Of particular concern, adoption of virtual consultation was lower in agencies that served more African American and Latina women as compared to those serving more White women. Low rates of adoption among agencies that serve at-risk populations have the potential to exacerbate rather than reduce disparities in maternal and child outcomes (Shelton et al., 2020). Multiple factors may have lowered adoption rates among agencies serving these African American and Latina populations, including limited community and agency resources among others (Woodward et al., 2019).

Variations in MHI strategy adoption rates aligned with participants’ perceptions of each strategy's feasibility, such that concerns about feasibility were greatest for strategies with low adoption rates. Multiple implementation strategies often are required to integrate EBIs within a complex system. The risk in using multiple strategies is the burden it may place on adopters, which may result in lower than ideal adoption rates for some of the strategies and declining strategy use over time. Many public service agencies struggle with limited resources and competing demands, particularly those serving populations at greatest risk for health disparities (e.g., new mothers who are low-income and are African American or Latina). In designing implementation strategies, care needs to be taken to maximize feasibility. Given the number of competing demands most agencies face, care also needs to be taken to promote the prioritization and sustainment of an innovation over time. All four MHI implementation strategies targeted nurse and team-level capacity to integrate mental health EBIs into routine practice. To prioritize and sustain the MHI, strategies may need to target factors at other levels of the NFP system, for example, policies and incentives, communication of goals and feedback, and leadership engagement, among others (Damschroder et al., 2009). The need to target multilevel system factors was identified during the sustainability assessment, which occurred near the end of the project. Assessing sustainability earlier in the project would have provided the team with guidance on multiple levels of change that would be needed to sustain the MHI over time.

Future Directions

Table 4 summarizes evaluation findings related to adoption, feasibility, and potential sustainability of MHI strategies. Table 4 also provides recommendations for revising the MHI to improve adoption and promote sustainment over time and notes which of the recommendations have been completed. The team recommended that additional strategies be put in place to (a) provide additional, ongoing training in mental health, (b) create a communication plan to raise awareness of the MHI, (c) hire a national-level expert in mental health, (d) establish a clear role for regional NFP consultants in supporting MHI implementation, and (e) create a system for monitoring, reporting, and taking action to improve MHI processes and outcomes. Creating a system for monitoring and reporting is particularly important to providing the data the NFP will need to adapt the MHI to meet the changing needs of agencies and the mothers they serve (Malone et al., 2021).

Table 4.
Summary of findings and recommended revisions to the MHI.

MHI strategies Adoption Feasibility Sustainability Revisions (status)

Online modules 60%–76% of nurses Required Need for ongoing training Continue to require online modules for all existing nurses and new nursing hires (completed) Design periodic mental health learning opportunities (recommended)

Clinical resources NA Hard to locate and access Not built into workflow Create a graphic, mapping locations of MHI components and centralize content on NFP national office website (completed)

Team meeting modules 27%–53% of supervisors Difficult due to competing demandsForgot they were available Team meeting modules are difficult to use Develop and implement a communication plan to remind supervisors and nurses of meeting modules and other MHI resources (completed); continue and enhance communication (recommended)

Virtual consultation 41% of teams, with low reach to African Americans Time-consuming, attendance rates were high Lack of staff with expertise in mental health Lack of ongoing coaching, feedback, and training Hire a mental health nurse (recommended)

Other Lack of MHI process and outcome metrics Create a system to monitor and report data on MHI completion rates and nurses’ mental health screening and referral rates (recommended)

Note. MHI = Mental Health Innovation; NFP = Nurse-Family Partnership.

Additional evaluation now is needed to determine the impact of a revised version of the MHI that builds on these recommendations. Further evaluation work also is needed to understand why some agencies chose not to adopt virtual consultations, particularly, those serving African American and Latina clients. Findings from this evaluation suggest that virtual consultation played an important role in activating supervisors’ use of the team meeting modules. Understanding and addressing the factors limiting adoption of virtual consultation is essential to ensuring that the MHI reduces rather than exacerbates inequities in mental health care.

This evaluation focused on implementation of the MHI. This paper only reports on implementation outcomes, which is a limitation. Future evaluation is needed to establish the MHI's effectiveness at improving nurses’ knowledge, skills, and practice and at improving the health and well-being of mothers and children.

Conclusions

This paper illustrates how an engaged research/practice evaluation was able to identify opportunities to improve the feasibility and potential sustainability of a multicomponent implementation strategy. This study illustrates the value of a national service agency engaging with academic researchers to develop, roll out, and evaluate implementation strategies. Through this partnership, the NFP's NSO was able to roll out the basic MHI to agencies nationwide. Nationwide implementation of the MHI was facilitated by the NSO's existing relationships with NFP agencies and nurses, which allowed them to send surveys directly to nurses and to require nurse completion of online modules. The NSO's national data management systems supported a practical approach to evaluation by providing a rich source of data on agencies, mothers served by those agencies, and nurse completion of online modules. Data were more limited for strategies not tracked in the national system (i.e., clinical resources and team meeting modules). NSO leadership’s engagement in all stages of the project facilitated rapid response to several recommendations for MHI improvement.

MHI strategy	Named using ERIC terminology	Actor	Action	Dose	Timing
Online modules	Conduct ongoing training	NFP National Service Office	Posted to national website, required completion	Five 20–35 min modules	Posted January 2018
Clinical resources	Develop and distribute education materials	NFP National Service Office	Posted to national website	NA	Posted January 2018
Team meeting modules	Promote network weaving	NFP National Service Office Agency-level supervisors	Posted to national website Facilitated team meetings	Five 60–90 min interactive activities	Posted January 2018
Virtual consultations	Facilitation	Mental Health Experts	Facilitated interactive webinar discussions with multiple teams	Six 60–90 min interactive activities	Two waves, June–November 2018 and January–June 2019

Race	Basic MHI number of mothers (%)	Enhanced MHI number of mothers (%)	Declined enhanced number of mothers (%)	P-value
Whole sample	51,534	3,500	6,344
American Indian/Alaskan Native	1,165 (2.3%)	120 (3.4%)	129 (2.0%)
Asian	1,213 (2.4%)	76 (2.2%)	113 (1.8%)
Native Hawaiian/Pacific Islander	232 (0.5%)	23 (0.7%)	19 (0.3%)
African American/Black	16,071 (31.2%)	661 (18.9%)	2,719 (42.9%)
Caucasian/White	23,746 (46.1%)	2,132 (60.9%)	2,329 (36.7%)
More than one race	3,408 (6.6%)	258 (7.4%)	405 (6.4%)
Unreported/unknown	5,699 (11.1%)	230 (6.6%)	630 (9.9%)
Enhanced MHI versus declined enhanced				<.0001
Ethnicity
Hispanic or Latinx	15,473 (30.0%)	1,122 (32.1%)	1,613 (25.4%)
Enhanced MHI versus declined enhanced				<.0001

Mental Health Innovation (MHI) strategy	Number (%)
Online modules (N = 2,100 nurses)
Module 1: Mental health introduction	1,586 (76)
Module 2: Learning about depression and anxiety part I	1,606 (76)
Module 3: Learning about depression and anxiety part II	1,557 (74)
Module 4: Integrating mental health	1,394 (66)
Module 5: Practicing mental health nursing care	1,262 (60)
Team meeting modules (N = 125 supervisors)
Module 1. Role of the nurse home visitor	51 (40.8)
Module 2. Legal liability: crisis prevention and management	64 (51.3)
Module 3. Intervention planning: adapting visit content and materials	50 (40.0)
Module 4. Screening and sharing screening feedback	44 (35.2)
Module 5. Mental health case conference	36 (28.8)
Module 6. Beyond integration, sustaining nurse, agency, and community impact	34 (27.2)
Virtual consultation (N = 110 teams invited)
Agreed to participate and attended at least one videoconference	45 (40.9)
Attended all six videoconferences	33 (73.3)

MHI strategies	Adoption	Feasibility	Sustainability	Revisions (status)
Online modules	60%–76% of nurses	Required	Need for ongoing training	Continue to require online modules for all existing nurses and new nursing hires (completed) Design periodic mental health learning opportunities (recommended)
Clinical resources	NA	Hard to locate and access	Not built into workflow	Create a graphic, mapping locations of MHI components and centralize content on NFP national office website (completed)
Team meeting modules	27%–53% of supervisors	Difficult due to competing demandsForgot they were available	Team meeting modules are difficult to use	Develop and implement a communication plan to remind supervisors and nurses of meeting modules and other MHI resources (completed); continue and enhance communication (recommended)
Virtual consultation	41% of teams, with low reach to African Americans	Time-consuming, attendance rates were high	Lack of staff with expertise in mental health Lack of ongoing coaching, feedback, and training	Hire a mental health nurse (recommended)
Other			Lack of MHI process and outcome metrics	Create a system to monitor and report data on MHI completion rates and nurses’ mental health screening and referral rates (recommended)

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Rita and Alex Hillman Foundation.

References

Baskerville

N. B.

Liddy

Hogg

(2012). Systematic review and meta-analysis of practice facilitation within primary care settings. The Annals of Family Medicine, 10(1), 63–74. https://doi.org/10.1370/afm.1312

Beeber

Ledford

Gasbarro

Zeanah

Knudtson

Sprinkle

McMichael

Mosqueda

Leeman

(2022). Developing a multicomponent implementation strategy for mental health interventions within the nurse–family partnership: An application of the EPIS framework. Journal of Nursing Scholarship, 54(4), 445–452. https://doi.org/10.1111/jnu.12755

Beeber

L. S.

Holditch-Davis

Perreira

Schwartz

T. A.

Lewis

Blanchard

Canuso

Goldman

B. D.

(2010). Short-term in-home intervention reduces depressive symptoms in early head start Latina mothers of infants and toddlers. Research in Nursing & Health, 33(1), 60–76. https://doi.org/10.1002/nur.20363

Beeber

L. S.

Perreira

K. M.

Schwartz

(2008). Supporting the mental health of mothers raising children in poverty: How do we target them for intervention studies? Annals of the New York Academy of Sciences, 1136(1), 86–100. https://doi.org/10.1196/annals.1425.008

Beeber

L. S.

Schwartz

T. A.

Holditch-Davis

Canuso

Lewis

Hall

H. W.

(2013). Parenting enhancement, interpersonal psychotherapy to reduce depression in low-income mothers of infants and toddlers: A randomized trial. Nursing Research, 62(2), 82–90. https://doi.org/10.1097/NNR.0b013e31828324c2

Beeber

L. S.

Schwartz

T. A.

Holditch-Davis

Canuso

Lewis

Matsuda

(2014a). Interpersonal psychotherapy with a parenting enhancement adapted for in-home delivery in Early Head Start. Zero to Three, 34(5), 35–44.

Beeber

L. S.

Schwartz

T. A.

Martinez

M. I.

Holditch-Davis

Bledsoe

S. E.

Canuso

Lewis

V. S.

(2014b). Depressive symptoms and compromised parenting in low-income mothers of infants and toddlers: Distal and proximal risks. Research in Nursing & Health, 37(4), 276–291. https://doi.org/10.1002/nur.21604

Bryan

C. J.

Vujanovic

A. A.

Nock

M. K.

(2019). Innovations in the science of suicide. Behaviour Research and Therapy, 120, 103451. https://doi.org/10.1016/j.brat.2019.103451

Ciciolla

Gerstein

E. D.

Crnic

K. A.

(2014). Reciprocity among maternal distress, child behavior, and parenting: Transactional processes and early childhood risk. Journal of Clinical Child and Adolescent Psychology, 43(5), 751–764. https://doi.org/10.1080/15374416.2013.812038

10.

Damschroder

L. J.

Aron

D. C.

Keith

R. E.

Kirsh

S. R.

Alexander

J. A.

Lowery

J. C.

(2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4, 50. https://doi.org/10.1186/1748-5908-4-50

11.

Forsetlund

O’Brien

M. A.

Forsén

Reinar

L. M.

Okwen

M. P.

Horsley

Rose

C. J.

(2021). Continuing education meetings and workshops: Effects on professional practice and healthcare outcomes. The Cochrane Database of Systematic Reviews, 9(9), CD003030. https://doi.org/10.1002/14651858.CD003030.pub3

12.

Glasgow

R. E.

Harden

S. M.

Gaglio

Rabin

Smith

M. L.

Porter

G. C.

Ory

M. G.

Estabrooks

P. A.

(2019). RE-AIM planning and evaluation framework: Adapting to new science and practice with a 20-year review. Frontiers in Public Health, 7, 64. https://doi.org/10.3389/fpubh.2019.00064

13.

Glasgow

R. E.

Vogt

T. M.

Boles

S. M.

(1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89(9), 1322–1327. https://doi.org/10.2105/ajph.89.9.1322

14.

Hofmann

S. G.

Gómez

A. F.

(2017). Mindfulness-based interventions for anxiety and depression. Psychiatric Clinics of North America, 40(4), 739–749. https://doi.org/10.1016/j.psc.2017.08.008

15.

Holditch-Davis

Santos

Levy

White-Traut

O’Shea

T. M.

Geraldo

David

(2015). Patterns of psychological distress in mothers of preterm infants. Infant Behavior and Development, 41, 154–163. https://doi.org/10.1016/j.infbeh.2015.10.004

16.

Hsieh

H. F.

Shannon

S. E.

(2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), 1277–1288. https://doi.org/10.1177/1049732305276687

17.

Kitzman

H. J.

Olds

D. L.

Cole

R. E.

Hanks

C. A.

Anson

E. A.

Arcoleo

K. J.

Luckey

D. W.

Knudtson

M. D.

Henderson

C. R.

Jr Holmberg

J. R

. (2010). Enduring effects of prenatal and infancy home visiting by nurses on children: Follow-up of a randomized trial among children at age 12 years. Archives of Pediatrics & Adolescent Medicine, 164(5), 412–418. https://doi.org/10.1001/archpediatrics.2010.76

18.

Kitzman

Olds

D. L.

Knudtson

M. D.

Cole

Anson

Smith

J. A.

Fishbein

DiClemente

Wingood

Caliendo

A. M.

Hopfer

Miller

Conti

(2019). Prenatal and infancy nurse home visiting and 18-year outcomes of a randomized trial. Pediatrics, 144(6), e20183876. https://doi.org/10.1542/peds.2018-3876

19.

Knittle

Gellert

Moore

Bourke

Hull

(2019). Goal achievement and goal-related cognitions in behavioral activation treatment for depression. Behavior Therapy, 50(5), 898–909. https://doi.org/10.1016/j.beth.2019.01.005

20.

Long

Cao

(2017). Social support and depression across the perinatal period: A longitudinal study. Journal of Clinical Nursing, 26(17–18), 2776–2783. https://doi.org/10.1111/jocn.13817

21.

Malone

Prewitt

Hackett

Lin

J. C.

McKay

Walsh-Bailey

Luke

D. A.

(2021). The clinical sustainability assessment tool: Measuring organizational capacity to promote sustainability in healthcare. Implementation Science Communications, 2(1), 77. https://doi.org/10.1186/s43058-021-00181-2

22.

Olds

Henderson

C. J.

Kitzman

Eckenrode

Cole

Tatelbaum

(1998). The promise of home visitation: Results of two randomized trials. Journal of Community Psychology, 26(1), 5–21. https://doi.org/10.1002/(SICI)1520-6629(199801)26:1<5::AID-JCOP2>3.0.CO;2-Y

23.

Olds

D. L.

Kitzman

Hanks

Cole

Anson

Sidora-Arcoleo

Luckey

D. W.

Henderson

C. R.

Jr Holmberg

Tutt

R. A.

Stevenson

A. J.

Bondy

(2007). Effects of nurse home visiting on maternal and child functioning: Age-9 follow-up of a randomized trial. Pediatrics, 120(4), e832–e845. https://doi.org/10.1542/peds.2006-2111

24.

Park

Brain

Grunau

R. E.

Diamond

Oberlander

T. F.

(2018). Maternal depression trajectories from pregnancy to 3 years postpartum are associated with children’s behavior and executive functions at 3 and 6 years. Archives of Women's Mental Health, 21(3), 353–363. https://doi.org/10.1007/s00737-017-0803-0

25.

Pooler

Perry

D. F.

Ghandour

R. M.

(2013). Prevalence and risk factors for postpartum depressive symptoms among women enrolled in WIC. Maternal and Child Health Journal, 17(10), 1969–1980. https://doi.org/10.1007/s10995-013-1224-y

26.

Powell

B. J.

Waltz

T. J.

Chinman

M. J.

Damschroder

L. J.

Smith

J. L.

Matthieu

M. M.

Proctor

E. K.

Kirchner

J. E.

(2015). A refined compilation of implementation strategies: Results from the expert recommendations for implementing change (ERIC) project. Implementation Science, 10(1), 21. https://doi.org/10.1186/s13012-015-0209-1

27.

Proctor

E. K.

Powell

B. J.

McMillen

J. C.

(2013). Implementation strategies: Recommendations for specifying and reporting. Implementation Science, 8(1), 139. https://doi.org/10.1186/1748-5908-8-139

28.

Proctor

Silmere

Raghavan

Hovmand

Aarons

Bunger

Griffey

Hensley

(2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7

29.

Shelton

R. C.

Chambers

D. A.

Glasgow

R. E.

(2020). An extension of RE-AIM to enhance sustainability: Addressing dynamic context and promoting health equity over time. Frontiers in Public Health, 8, 134. https://doi.org/10.3389/fpubh.2020.00134

30.

Substance Abuse and Mental Health Services Administration. (2019). A practical guide to psychiatric advance directives. Center for Mental Health Services, Substance Abuse and Mental Health Services Administration. https://www.samhsa.gov/sites/default/files/a_practical_guide_to_psychiatric_advance_directives.pdf

31.

Woodward

E. N.

Matthieu

M. M.

Uchendu

U. S.

Rogal

Kirchner

J. E.

(2019). The health equity implementation framework: Proposal and preliminary study of hepatitis C virus treatment. Implementation Science, 14(1), 26. https://doi.org/10.1186/s13012-019-0861-y