A mixed-methods process evaluation of Family Navigation implementation for autism spectrum disorder

Abstract

There is growing interest in Family Navigation as an approach to improving access to care for children with autism spectrum disorder, yet little data exist on the implementation of Family Navigation. The aim of this study was to identify potential failures in implementing Family Navigation for children with autism spectrum disorder, using a failure modes and effects analysis. This mixed-methods study was set within a randomized controlled trial testing the effectiveness of Family Navigation in reducing the time from screening to diagnosis and treatment for autism spectrum disorder across three states. Using standard failure modes and effects analysis methodology, experts in Family Navigation for autism spectrum disorder (n = 9) rated potential failures in implementation on a 10-point scale in three categories: likelihood of the failure occurring, likelihood of not detecting the failure, and severity of failure. Ratings were then used to create a risk priority number for each failure. The failure modes and effects analysis detected five areas for potential “high priority” failures in implementation: (1) setting up community-based services, (2) initial family meeting, (3) training, (4) fidelity monitoring, and (5) attending testing appointments. Reasons for failure included families not receptive, scheduling, and insufficient training time. The process with the highest risk profile was “setting up community-based services.” Failure in “attending testing appointment” was rated as the most severe potential failure. A number of potential failures in Family Navigation implementation—along with strategies for mitigation—were identified. These data can guide those working to implement Family Navigation for children with autism spectrum disorder.

Keywords

autism spectrum disorders qualitative research

Background

Patient Navigation (PN) is an evidence-based approach to improving access to health services by eliminating barriers to care. PN is an intervention rooted in the Chronic Care Model and is designed as a means to reduce racial, ethnic, and socioeconomic disparities in time to diagnosis by shortening the interval between a positive screen (e.g. a mammogram for breast cancer) and definitive diagnosis. PN has been shown to reduce disparities by providing culturally tailored patient education, linkage to community resources, and facilitating appointment scheduling and reminders (Ali-Faisal et al., 2017; Blakely and Dziadosz, 2008; Braun et al., 2012; Freeman, 2006; Marshall et al., 2016; Myers et al., 2015; Rohan et al., 2016). PN has proven effectiveness for a variety of disorders including cancer, HIV, asthma, and mental health disorders (Freund et al., 2014; Gleason et al., 2016; Koester et al., 2014; Myers et al., 2015). Numerous studies and PN programs have recently arisen within a variety of healthcare settings (Association of Maternal & Child Health Programs, 2018; Boston Medical Center, 2016; Family Navigation Project, 2018; Family Navigator Program, 2018; Family Voices Colorado, 2013; Markoulakis et al., 2016, 2018; National Institutes of Health, 2017). While specific standards of care for PN (e.g. what are the core components of the intervention, how is fidelity defined and measured) are currently being developed (Parker et al., 2010), the general principles of PN are consistent across settings. These include the following: navigators share key attributes (culture, language, lived experience) with the population served, navigators are trained in motivational interviewing and/or problem-solving strategies, and navigators work to support engagement in care (Clark et al., 2014; Freund et al., 2014; Gunn et al., 2014; Rohan et al., 2016).

Family Navigation (FN) is a version of PN that involves a focus on the family unit as opposed to an individual patient (Feinberg et al., 2016). FN is a particularly promising intervention for improving access to care for children with autism spectrum disorder (ASD) who experience significant challenges in obtaining diagnoses and accessing both ASD and non-ASD health services (Benevides et al., 2016; Chiri and Warfield, 2012; Mandell et al., 2009; Shattuck and Grosse, 2007). Racial, ethnic, and socioeconomic disparities are also pronounced among children with ASD (Acharya et al., 2017; Angell and Solomon, 2017; Parikh et al., 2018; St Amant et al., 2018; Yingling et al., 2018). Given that FN is an intervention designed to both improve access to care and reduce disparities, it is no wonder that there is growing interest from both the research and clinical ASD community in this intervention (Association of Maternal & Child Health Programs, 2018; Boston Medical Center, 2016; Family Navigation Project, 2018; Family Navigator Program, 2018; Family Voices Colorado, 2013; Markoulakis et al., 2018; National Institutes of Health, 2017).

Despite this growing interest in spreading FN, few data exist on how it can be successfully implemented in clinical practice, and data-driven strategies for dissemination are virtually non-existent (Paskett et al., 2011; Valaitis et al., 2017). Moreover, the definition of FN as a whole, its individual components, and the differentiation between the intervention and strategies for implementation remain ill-defined. Implementation of health systems interventions can be complex, and understanding the details of implementation can be critical to an intervention’s future success (MacDonald et al., 2016). Multiple studies of FN in cancer and HIV demonstrate varying success or diminution of impact of navigation upon implementation in real-world practice (Donelan et al., 2011; Hedlund et al., 2014; Ramachandran et al., 2015). Details of and reasons for implementation challenges are generally unknown, but the lack of concrete data on barriers to implementation is a likely contributor (Hedlund et al., 2014).

Process mapping and analysis is a method of systems evaluation utilized frequently in engineering and business (Damelio, 2011), and utilization of such methods in health and medical services is growing (Abujudeh et al., 2016; Carter et al., 2015; Damelio, 2011; Lins et al., 2018). The goal of process analysis is to improve health outcomes by identifying and reducing system inefficiencies in complex, multi-step interventions. Previous research has found that engaging in process analysis can lead to systems that are more appropriate, effective, and sustainable, with improved health service delivery (Abujudeh et al., 2016; Carter et al., 2015; Lins et al., 2018; Pojasek, 2013).

Therefore, we undertook a process analysis of the implementation of FN in the context of a large, randomized effectiveness trial of FN to reduce time to diagnosis and treatment for children with ASD, using a novel method—a Failure Modes and Effects Analysis (FMEA)—derived from the fields of engineering and operations. Our version of FN was developed to reduce the time from initial screen to definitive diagnosis and access to treatment for children with ASD. While prior FN studies have focused on improving access to services, the FN study evaluated here targets both reduced time to diagnosis and improved access to services. The intervention was tested over multiple pilot phases in multiple settings (Broder-Fingert et al., 2018).

The current FMEA study is the first in a larger, mixed-methods approach based on the Consolidated Framework for Implementation Research (CFIR) to understanding FN implementation (Damschroder et al., 2009). Specifically, this study was designed to explore the “process” domain of CFIR. CFIR was chosen because it provides a comprehensive framework to systematically identify factors from multi-level contexts and is useful in studying processes prospectively (Keith et al., 2017). We focused on the “Reflecting and Evaluating” construct of the process domain, defined as “group and personal reflection … reflecting or debriefing before, during, and after implementation” (CFIR Research Team, 2018). The aims of this study were to (1) delineate steps in the process of FN implementation and (2) understand potential failures in the process and the relative impact of such failures. The goal of this work is to guide investigators, clinical leaders, and policymakers on implementing FN for ASD and ultimately ensure future dissemination to vulnerable populations.

Methods

Study design

We employed a sequential mixed-methods design, using qualitative semi-structured interviews to inform an FMEA that includes both quantitative and qualitative analyses (Johnson and Onwuegbuzie, 2004). Data from the quantitative analysis were given more prominence since it allows for comparison and prioritization of potential failures. The study was approved by the institutional review board at Boston University.

The intervention

In this study, we evaluated a specific version of PN, FN. FN is rooted in the principles of PN and differs only in that it focuses on the entire family unit, as opposed to the specific patient. FN was implemented in this study as a means to reduce the time from an initial positive ASD screen in primary care to diagnostic resolution (completion of a diagnostic assessment) and deployment of services. The intervention is being tested as part of an ongoing National Institute of Mental Health–funded multi-site randomized controlled trial to improve access to diagnostic and treatment services for children at risk of ASD (Broder-Fingert et al., 2018). The study is being carried out in 14 primary care clinics and community health centers across three academic developmental and behavioral clinics in three states: Massachusetts, Connecticut, and Pennsylvania. In this study, children with a confirmed risk for ASD—as determined by a positive ASD screen at an 18- or 24-month primary care well child visit—are recruited from primary care and randomized either to conventional care management or to the FN intervention. The study population includes 350 families from primarily racial and ethnic minority populations (52% Black, 32% Hispanic, 81% publicly insured). Families receiving FN work one-on-one with a navigator who provides off-site support (e.g. home visits, accompanied family to appointments) to overcome logistic barriers related to the diagnostic visit and set-up services following the diagnostic evaluation. Navigators undergo extensive training in motivational interviewing, collaborative decision-making, child development, and social service provision. In addition, they participate in weekly supervisory meetings and have sessions with families recorded and reviewed by a supervisor to monitor fidelity to the intervention. More details about the study population and FN intervention are published elsewhere (Broder-Fingert et al., 2018).

Process map development

Interview participants

We interviewed 12 key stakeholders in FN implementation to develop our process map. Our methods were based on those used in other healthcare settings (Yarmohammadian et al., 2014). Interview participants included Family Navigators (n = 3), study coordinators (n = 3), research assistants (n = 1), developmental and behavioral pediatricians (DBPs; n = 3), and Principals or Co-investigators (n = 2). Participants were all members of the above-noted multi-site FN study funded by the National Institute of Mental Health.

Semi-structured interviews informing process map development

We employed an inductive approach to interviews, using a set of open-ended questions about implementation in general, stages of implementation, and key participants in implementation.

Convenience sampling was used to recruit interview participants; incentives were not provided. All interviews were carried out by either a single research assistant (S.Q.) or the Principal Investigator (S.B-F.) and ranged from 20 to 60 min in length. Questions focused on systems and individual-level factors involved in implementation. Extensive field notes were taken during each interview and were entered into a semi-structured memo template immediately after the interview to allow for the synthesis of key aspects of the data; qualitative software was not used. The research assistant and Principal Investigator subsequently met to review notes and update a draft of the process map. Interviewees were then given an opportunity to comment on the process map. This process was repeated until no new process steps were identified and the process map was finalized (Figure 1).

Figure 1.

Process of implementing Patient Navigation.

FMEA

FMEA is a quality assurance process that is commonly used to address potential flaws and errors during the design stage of a product’s life (Willis, 1992). FMEA is frequently used by manufacturing and medical device investigators and is only now emerging as a tool to evaluate behavioral health interventions (Hosoya et al., 2015; Kawai et al., 1989; Manger et al., 2015; Rienzi et al., 2015; Sorrentino, 2016).

In this study, the FMEA was performed using a process map of FN implementation (methods for creation of the process map are described above) via the following seven steps:

The process map was presented at a meeting of nine investigators responsible for FN implementation across 14 clinics in three states. Investigators included health services researchers (n = 2), DBPs (n = 3), primary care pediatricians (n = 2), and study coordinators (n = 2).

A single investigator (S.B-F.) presented the process map and described each step identified in the map. Participants in the FMEA were then given 15 min to review the map in detail on their own. The group reconvened and was then given the opportunity to comment or ask questions.

FMEA methodology was explained in detail, and examples of other FMEA studies were presented (Ashley et al., 2017; Yousefinezhadi et al., 2016). Investigators were then given three hypothetical examples to practice the FMEA method.

Because the process map included 66 steps, it was not feasible (due to time limitations) to perform the FMEA on every step. Specifically, because the FMEA requires rating three elements of each step, if each of the 66 steps had been rated, it would have required each person to perform over 200 ratings across the entire map. Therefore, after the training, each expert was asked to identify potential “high-risk” failures in the process map. Experts were instructed that high-risk failures should be defined as those that would be most consequential to implementation if the failure occurred. It was further explained that consequential failures were those that were most likely to occur, would have the greatest negative impact on implementation if the failure occurred, or would be least likely to detect (i.e. go unnoticed and therefore difficult to find and address). Experts were given a copy of the process map and allowed to indicate confidentially every step in the process they deemed as “high risk.” The experts were allowed to identify as many steps as they wanted as “high risk.” Of the 66 steps in the process, 7 steps emerged as high risk based on the frequency of selection by experts during the identification process.

The investigators were then asked to complete a confidential online survey in which they assigned a numerical score for each of the seven steps identified as high risk. For each step, a space for free text was provided to allow investigators to describe and justify each rating. Three indexes were used for each high-risk failure mode per the FMEA protocol: the occurrence rating, the severity rating, and the detectability rating (Institute for Healthcare Improvement, 2017). The occurrence rating corresponded to the likelihood of the failure occurring, the severity rating to the likelihood of severe consequences were the failure to occur, and the detectability rating to the likelihood of not detecting the failure after it occurred. A 10-point scale was used to score each category, with 10 representing an extremely high likelihood.

Investigators were also given the opportunity to comment on potential causes and actions to reduce occurrence through free text.

Finally, the risk priority number (RPN) was calculated as the product of the three scores: RPN = Occurrence × Severity × Detection and subsequently used to rank the various steps in order of importance. The RPN represents the overall risk assigned to a step, and steps with a higher RPN were ranked above those with lower RPNs. These values are reported in Table 1.

Table 1.

Failure modes, descriptions, mean risk priority numbers, and mitigation actions.

Core process(n = number of stakeholder ratings)^a	Description	Potential failure	Occurrence (1–10)Mean (SE)	Detection (1–10)Mean (SE)	Severity (1–10)Mean (SE)	Risk priority numberMean (SE; Range)	Mitigation actions
Setting up community-based services, (n = 9)	Navigators work with community agencies including early intervention, public schools, insurance, and social security administration	Navigator not familiar with resources available Family does not follow-up on recommendations Difficult to access (wait lists, insurance restrictions, limited options)	6.5 (1.1)	5.4 (0.9)	7.6 (0.7)	279.1 (74.1; 45.0, 630.0)	Supervisor develops list of options Navigator counsels family on value of services Develop relationships with community partners
Initial meeting with family (n = 9)	Navigators meet with families after initial positive ASD screen, usually within 7 days of initial referral to navigation	Family no-shows, cancels, or never schedules meeting Family does not provide accurate contact information Child(ren) interrupt visit	6.3 (0.7)	3.3 (1.2)	7.9 (0.7)	180.3 (75.1; 20.0, 576.0)	Obtain multiple forms of contact information Call/text/email reminder Demonstrate value (e.g. solving a problem) Navigator presents barriers faced by others
Initial navigator training (n = 18)	3-day intensive training in motivational interviewing, problem-solving, community resources, and child development	Initial training not comprehensive Does not attend training Practice session ineffective model Navigator not qualified	4.2 (0.8)	5.2 (1.0)	7.4 (0.7)	164.4 (50.3; 8.0, 405.0)	Clear expectations Senior navigators mentor Interview candidates with critical thinking skills, home visiting experience, and cultural concordance, as well as relevant experience
Navigator fidelity monitoring and refresher training (n = 18)	Navigators have weekly meetings with their supervisor to review cases. Navigator/family sessions are audio-recorded and reviewed by supervisor	Data not collected Training rushed Navigators not committed Inaccurate records Supervision meetings not prioritized	5.1 (0.5)	4.2 (0.7)	5.8 (0.5)	146.1 (35.1; 6.0, 480.0)	Mandatory refresher trainings Teach back More time for training Online trainings Emphasize value of data Dedicated meeting
Attending intake and/or testing appointments with families (n = 9)	Navigators accompany families to their developmental and behavioral pediatrics appointments	Family forgets appointment Transportation fails Childcare falls through for family	6.2 (0.8)	1.7 (0.3)	8.1 (0.6)	86.4 (20.8; 16.0, 200.0)	Reminder calls and texts Assists family with transportation and childcare

ASD: autism spectrum disorder.

Because two groups of categories were collapsed for the final analyses, some processes had more ratings than others.

Quantitative data analysis of FMEA data

Data were analyzed using descriptive analysis in Microsoft Excel. For each failure mode, means and standard errors (SEs) were calculated for the RPN and for each of the three indexes. SAS^® 9.4 software was used to perform a one-way analysis of variance, comparing mean RPNs across the core processes.

Qualitative data analysis of FMEA data

Two investigators read through all comments that raters provided explaining reasons for potential failures in the free text portion of the FMEA (S.Q., S.B-F.). Emergent themes were identified through a content analysis approach (Vaismoradi et al., 2013). Because the number of comments was limited, it was unnecessary to formally develop code books. Instead, the team systematically reviewed responses together and discussed key concepts. All comments were then organized into themes by both investigators.

Results

Process map

We identified 66 total steps in the process of implementing FN (Figures 1 to 3). Five categories of individuals were identified as key to implementing FN: patients/families, navigators, DBPs, early intervention providers, and the research staff. The two elements of the process that encompassed the most steps were (1) navigator training (14 steps) (Figure 2) and (2) diagnostic assessment (25 steps) (Figure 3).

Figure 2.

Navigator training.

Figure 3.

Diagnostic assessment.

Quantitative analysis: failure ratings

Completion of the ratings of all seven steps took an average of 26 min (range 14–47 min). The majority of raters completed ratings in a single session (one login into the survey), although two raters completed the survey over multiple days/sessions.

Initially, our expert panel identified seven steps in the process of FN implementation as potential “high-risk” failures, which were then included in the full FMEA analysis (Table 1). Of these steps, four were considered to be conceptually overlapping and were therefore collapsed such that the final FMEA included five steps. These included the following: “setting up community-based services,” “attending intake and/or testing appointment with the family,” “initial meeting with family,” “initial navigator training,” and “navigator fidelity monitoring and refresher training.” Common reasons for failure included FN not being aware of appropriate services, families not being receptive to services, difficulties associated with setting up services, family scheduling conflicts, and insufficient time for training. The step in the process with the highest RPN, representing the overall highest risk assigned to a step, was “setting up community-based services” (mean, SE: 279.1, 74.1). Severity, occurrence, and detection were each rated using a 10-point scale, with 10 representing an extremely high likelihood of severe consequences, a failure occurring, and a failure not being detected, respectively. The step in the process with the highest severity rating was “attending intake and/or testing appointment with the family” (8.1, 0.6). The step in the process with the highest occurrence rating was “initial meeting with family” (6.3, 0.7). The step rated with the highest detection rating was “setting up community-based services” (5.4, 0.9). Differences were significant between the mean detection and severity ratings (ps = 0.02 and 0.02, respectively) across processes, but not the occurrence or total RPN ratings (ps = 0.16 and 0.13).

Qualitative analysis: themes related to failure causes and mitigation

Two specific themes emerged in stakeholder descriptions of potential causes for and actions to reduce the occurrence of each failure (Table 1). These included (1) insufficient monitoring and training and (2) challenges to engaging families.

Theme 1: monitoring and training

Regarding monitoring and training, stakeholders suggested more training and monitoring would be beneficial. For example, one stakeholder noted, “[Fidelity] recordings are great, but sites could also schedule observed navigation interactions, follow up calls with family by non-navigator staff, etc., to assess fidelity.” Another stakeholder suggested, “Mandatory refresher trainings at least yearly that are scheduled with enough advance scheduling to ensure full participation.”

Theme 2: family engagement

Regarding family engagement, many stakeholders noted challenges in engaging families with FN, as well as the need for support in engaging some families. One stakeholder noted, “Helping families understand the value of the navigator prior to introduction” would be useful in decreasing this failure. Another suggested, “Navigators could compile and update lists of barriers that many families in the community have experienced in the past and use it as a talking point for their initial interview.”

Discussion

This is the first study to use a novel tool—the FMEA—to evaluate the process of implementing FN for children with ASD. We found the process of implementation to be complex (i.e. 66 steps, five key categories of participants). When assessing potential failures in implementation, we found “setting up/recommending community-based services,” “initial meeting with the patient/family,” “initial FN training,” “fidelity monitoring and refresher training,” and “attending intake and/or testing appointment with the family” to be the most high risk for failure. The failure rated as most severe if it occurred was “attending intake and/or testing appointment with the family.” The failure rated most likely to happen was “initial meeting with the patient/family and the navigator.” The failure rated least likely to be detected was “setting up community-based services.”

The first important finding of this study is the proof of principle. The importance of creating process maps to increase efficiency and reduce error has long been established in industry and manufacturing (Martin and Osterling, 2013). The technique has recently started to gain prominence in the healthcare sector, although primarily in the medical device, procedural, and quality and safety setting (Faerber et al., 2015; Huynh et al., 2016; Terezakis et al., 2011; Walsh et al., 2013). Given the complexity of many implementation strategies, methods for rigorously analyzing each element of an implementation strategy are critical for both investigation and dissemination purposes. Understanding each step in the process of implementation, how they interact, and where they might fail can inform two critical elements of implementation. First, it is integral to understanding what the “active ingredients” of any implementation strategy are. As is true for intervention development, implementation strategies can often be separated into “core components” and an adaptable periphery (Demby et al., 2014). Given the rapid growth of the field of implementation science, new strategies for implementation are continuously being developed and tested (Powell et al., 2015). Establishing methodologies for evaluation are a high priority for the implementation science research community (Chambers and Norton, 2016; Neta et al., 2015; Proctor et al., 2012). This study takes on additional significance when considering that strategies for rigorously measuring or evaluating the “Process” domain of CFIR—the domain we are addressing in this study—are yet to be described (Lewis et al., 2015; Society for Implementation Research Collaboration (SIRC), 2018). Second, the FMEA can be useful in identifying potential failures in implementation. The method used in this study, in which an FMEA was performing in the context of an ongoing trial, is a particularly appealing technique for those looking to speed the pace of implementation from effectiveness trial to “real-world” practice.

The specific findings of our FMEA are also notable. First, implementation of FN is highly complex. Under-standing implementation complexity is critical to developing formal implementation strategies for dissemination (May et al., 2016; Pfadenhauer et al., 2017). Given the growing interest in FN—for ASD and other disorders—understanding the complexity of the implementation process is critical (Esparza, 2013; Feinberg et al., 2013, 2014; Freund et al., 2014; Markoulakis et al., 2018; Perkins, 2015). It is likely that implementing FN for other disorders would be equally complex, and understanding the many stages in the process would be valuable for the non-ASD community as well. Using FMEA, we have identified a set of risks to successful FN that may be helpful in the planning and implementation of FN for other disorders or in other settings.

Setting up community-based services was the step with the largest RPN. A well-established challenge in obtaining services for children with ASD is the complexity of the service systems that provide care (Bryson et al., 2008; Klin et al., 2015; Locke et al., 2016; Stahmer et al., 2015; Thomas et al., 2007). Families and children with ASD are required to navigate the health, educational, and community-based service sector. Clearly, the challenges families face in navigating such services also impact implementation of interventions that must also cross several systems (Locke et al., 2016; Wood et al., 2015). Understanding which systems are involved, and how one can best interact with them, may be valuable information for anyone planning to implement FN in a new setting.

Another important finding of this study is that failure of “attending intake and/or testing appointment with the family” was rated the most severe if occurred (mean, SE: 8.1, 0.6) and had a high likelihood of occurring (6.2, 0.8). On the other hand, the likelihood of detection was also rated as higher than other failures (1.7, 0.3), rendering the overall severity less than many other failure modes. It is particularly notable that within the process map as a whole, the diagnostic assessment was the process that included the most steps (25 in total). Whether the core processes (e.g. the number of steps in an intervention) predict the significance, severity, and/or likelihood of detecting failure in implementation is unclear but may be worth further investigation (Chamberlain et al., 2011; Saldana et al., 2012).

Finally, fidelity monitoring and training is a major challenge for many ASDs and behavioral health interventions, and new and improved methods to measure and maintain fidelity are currently under development (Peavy et al., 2014; Sineath et al., 2017; Stahmer et al., 2015; Yates et al., 2013). The results of this study reiterate the value of such efforts, as a failure in fidelity monitoring was rated critical. When analyzing suggestions to ensure that fidelity monitoring and refresher training did not fail, we identified a predominant theme: more continual monitoring and training was preferable. This is a very important finding for anyone proposing to implement FN or for those working to develop implementation strategies (Broder-Fingert et al., 2018).

Limitations

This study has several limitations. Given that this is a new application of the FMEA methodology, there are certain caveats to its interpretation. First, the processes being evaluated at each step are very different, and the upper and lower limits of the RPN vary widely. Although this is expected, it can make interpretation challenging, particularly because there is no published cut-off value for interpreting RPN. According to the Institute for Healthcare Improvement (2017) who developed the FMEA methodology for the healthcare setting, comparisons are made between different steps/core processes only for the purpose of prioritizing which steps are most important to address. Steps with higher RPN scores should be prioritized above processes with lower scores, but mean scores are not meant to be compared between different interventions or across different studies. Therefore, the RPN scores in this study may only be useful to FN and not generalizable to other implementation or intervention efforts.

Another limitation is a lack of data on the validity of the scoring system. Although no data exist on reliability or validity of the FMEA for implementation science, one prior study of medication administration found face validity to be high, but other forms of validity (content validity, criterion validity, construct validity) were low (Shebl et al., 2012). Therefore, we feel this tool is best used within the context of larger, mixed-methods studies, which is how we are using it. Moreover, we are working on a methods paper to further explore the strengths and weaknesses of this methodology.

Finally, FN is a highly complex process, and while this study proposes 66 steps, this may not accurately reflect the full scope of what is necessary for completing FN. However, with increased detail, the usefulness of the process map as an overall picture of FN decreases (Madison, 2005). Similarly, with 66 steps, it was not feasible to analyze each one through FMEA. It is, therefore, possible that there are steps that would have higher RPNs had we been able to fully analyze every step. Future work could delve more fully into specific pieces of the process. In addition, we conducted our FMEA through an online survey, rather than through in-person discussions. This could decrease the consensus around how and why specific steps are likely to fail. Future work could address this limitation by conducting FMEA in-person discussion as a group with all of the key stakeholders present. Another challenge is that temporality (timing of each step) is challenging to depict but may be important in identifying failures that are particularly costly with respect to time.

Finally, while navigators, clinicians, and research staff were involved in our process evaluation, families receiving the intervention were not included. This was because our analysis was focused on the implementation of the intervention rather than how it was received by families. In the future, we hope to work with families on a similar analysis to understand their perspective and assess for differences across the two evaluations.

Future directions

The FMEA identified five high-risk steps in the FN intervention that can be modified to prevent future failures. “Setting up community-based services” could be improved by developing inventories of community resources and agencies. Establishing contacts at key agencies to keep navigators updated on wait lists and availability would provide navigators with helpful information to support families as they navigate service systems. Also, tools (i.e. novel care coordination technology platforms) may be useful. With respect to reducing failures in “attending intake and/or testing appointment with the family,” it may be beneficial to establish a robust reminder system as well as have navigators help create backup transportation and childcare plans for families. Asking parents for at least two alternative contacts (which is now being implemented in our current work) may help in attendance of appointments as well as establishing an initial meeting with the navigator. With respect to navigation, navigator training could be improved by having more frequent training sessions or by employing a mentored navigation approach in which new navigators are paired with experienced navigators to attend visits with families as part of their training. Fidelity monitoring of navigation could be improved by increasing monitoring and incorporating in-person observations or other methods besides recording. Moving forward, the high-risk steps identified in this FMEA will be addressed to improve the FN intervention that was evaluated. In addition, it would be beneficial to consider how failures at these high-risk steps impact clinical, service, and implementation outcomes to inform future strategies for mitigation. Failures associated with poor outcomes may require additional investigation and process improvement.

Conclusion

As the evidence for FN and other evidence-based patient engagement programs continue to grow, delineating and evaluating the process of implementation is critical for future successful dissemination (Perkins, 2015). Moreover, understanding the process, as well as potential failures, is critical for investigators looking to study—and hospital systems looking to implement—this evidence-based intervention. This study provides a new method for evaluating implementation, with a focus on early detection and mitigation of failures in the process. We are currently using these data to refine our implementation strategy and adapt the intervention to meet the needs of the population as well as the context in which it is being carried out.

Supplemental Material

AUT808460_Lay_Abstract – Supplemental material for A mixed-methods process evaluation of Family Navigation implementation for autism spectrum disorder

Supplemental material, AUT808460_Lay_Abstract for A mixed-methods process evaluation of Family Navigation implementation for autism spectrum disorder by Sarabeth Broder-Fingert, Sarah Qin, Julia Goupil, Jessica Rosenberg, Marilyn Augustyn, Nate Blum, Amanda Bennett, Carol Weitzman, James P Guevara, Ada Fenick, Michael Silverstein and Emily Feinberg in Autism

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval

The study was approved by the institutional review board at Boston University

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: E.F. was supported by R01MH104355; S.B.F. was supported by K23 MH109673; M.S. was supported by K24HD081057; N.B., M.A., and C.W. were supported in part by cooperative agreement UA3MC20218 and by project (T77MC00012) from the Maternal and Child Health Bureau, Health Resources and Services Administration, US Department of Health and Human Services. This information or content and conclusions are those of the authors and should not be construed as the official position or policy of nor should any endorsements be inferred by HRSA, HHS, or the US Government.

References

Abujudeh

Danielson

Bruno

(2016) A patient-centered radiology quality process map: opportunities and solutions. American Journal of Roentgenology 207(5): 940–946.

Acharya

Meza

Msall

(2017) Disparities in life course outcomes for transition-aged youth with disabilities. Pediatric Annals 46(10): e371–e376.

Ali-Faisal

Colella

TJF

Medina-Jaudes

et al . (2017) The effectiveness of patient navigation to improve healthcare utilization outcomes: a meta-analysis of randomized controlled trials. Patient Education and Counseling 100(3): 436–448.

Angell

Solomon

(2017) “If I was a different ethnicity, would she treat me the same?”: Latino parents’ experiences obtaining autism services. Disability & Society 32(8): 1142–1164.

Ashley

Armitage

Taylor

(2017) Recognising and referring children exposed to domestic abuse: a multi-professional, proactive systems-based evaluation using a modified Failure Mode and Effects Analysis (FMEA). Health & Social Care in the Community 25(2): 690–699.

Association of Maternal & Child Health Programs (2018) Family Navigation Implementation Strategies: Improving Systems of Care (Issue Brief). Available at: http://www.amchp.org/AboutAMCHP/NewsRoom/Pages/MCHUpdates.aspx

Benevides

Carretta

Lane

(2016) Unmet need for therapy among children with autism spectrum disorder: results from the 2005–2006 and 2009–2010 national survey of children with special health care needs. Maternal and Child Health Journal 20(4): 878–888.

Blakely

Dziadosz

(2008) The chronic care model for behavioral health care. Population Health Management 11(6): 341–346.

Boston Medical Center (2016) Center for Family Navigation and Community Health Promotion, 7 July. Available at: https://www.bmc.org/programs/center-family-navigation-and-community-health-promotion (accessed 19 July 2018).

10.

Braun

Kagawa-Singer

Holden

AEC

et al . (2012) Cancer patient navigator tasks across the cancer care continuum. Journal of Health Care for the Poor and Underserved 23(1): 398–413.

11.

Broder-Fingert

Walls

Augustyn

et al . (2018) A hybrid type I randomized effectiveness-implementation trial of patient navigation to improve access to services for children with autism spectrum disorder. BMC Psychiatry 18(1): 79.

12.

Bryson

Corrigan

McDonald

et al . (2008) Characteristics of children with autism spectrum disorders who received services through community mental health centers. Autism: The International Journal of Research and Practice 12(1): 65–82.

13.

Carter

AJE

Gould

Vanberkel

et al . (2015) Offload zones to mitigate emergency medical services (EMS) offload delay in the emergency department: a process map and hazard analysis. Canadian Journal of Emergency Medicine 17(6): 670–678.

14.

CFIR Research Team (2018) Reflecting and evaluating, 18 July. Available at: https://cfirguide.org/constructs/reflecting-and-evaluating/ (accessed 26 September 2018).

15.

Chamberlain

Brown

Saldana

(2011) Observational measure of implementation progress in community based settings: the stages of implementation completion (SIC). Implementation Science 6: 116.

16.

Chambers

Norton

(2016) The adaptome: advancing the science of intervention adaptation. American Journal of Preventive Medicine 51(4 Suppl. 2): S124–S131.

17.

Chiri

Warfield

(2012) Unmet need and problems accessing core health care services for children with autism spectrum disorder. Maternal and Child Health Journal 16(5): 1081–1091.

18.

Clark

Parker

Battaglia

et al . (2014) Patterns of task and network actions performed by navigators to facilitate cancer care. Health Care Management Review 39(2): 90–101.

19.

Damelio

(2011) The Basics of Process Mapping. 2nd ed. Boca Raton, FL: CRC Press.

20.

Damschroder

Aron

Keith

et al . (2009) Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation Science 4(1): 50.

21.

Demby

Gregory

Broussard

et al . (2014) Implementation lessons: the importance of assessing organizational “fit” and external factors when implementing Evidence-Based Teen Pregnancy Prevention Programs. Journal of Adolescent Health 54(3 Suppl.): S37–S44.

22.

Donelan

Mailhot

Dutwin

et al . (2011) Patient perspectives of clinical care and patient navigation in follow-up of abnormal mammography. Journal of General Internal Medicine 26(2): 116–122.

23.

Esparza

(2013) Patient navigation and the American Cancer Society. Seminars in Oncology Nursing 29(2): 91–96.

24.

Faerber

Horvath

Stillman

et al . (2015) Development and pilot feasibility study of a health information technology tool to calculate mortality risk for patients with asymptomatic carotid stenosis: the Carotid Risk Assessment Tool (CARAT). BMC Medical Informatics and Decision Making 15: 20.

25.

Family Navigation Project (2018). Available at: https://sunnybrook.ca/content/?page=family-navigation-project (accessed 19 July 2018).

26.

Family Navigator Program (2018). Available at: http://pediatrics.med.miami.edu/mailman-center/family-navigator-program/ (accessed 19 July 2018).

27.

Family Voices Colorado (2013) Bridging Hospital and Commu-nity: family navigator program expands, 22 July. Available at: https://familyvoicesco.org/2013/07/22/family-navigator-program-expands/ (accessed 19 July 2018).

28.

Feinberg

Abufhele

Sandler

et al . (2016) Reducing disparities in timely autism diagnosis through family navigation: results from a randomized pilot trial. Psychiatric Services 67(8): 912–915.

29.

Feinberg

Augustyn

Fitzgerald

et al . (2014) Improving maternal mental health after a child’s diagnosis of autism spectrum disorder: results from a randomized clinical trial. JAMA Pediatrics 168(1): 40–46.

30.

Feinberg

Silverstein

Ferreira-Cesar

(2013) Integrating mental health services for mothers of children with autism. Psychiatric Services 64(9): 930–930.

31.

Freeman

(2006) Patient navigation: a community based strategy to reduce cancer disparities. Journal of Urban Health 83(2): 139–141.

32.

Freund

Battaglia

Calhoun

et al . (2014) Impact of patient navigation on timely cancer care: the Patient Navigation Research Program. Journal of the National Cancer Institute 106(6): dju115.

33.

Gleason

Cicutto

Haas-Howard

et al . (2016) Leveraging partnerships: families, schools, and providers working together to improve asthma management. Current Allergy and Asthma Reports 16(10): 74.

34.

Gunn

Clark

Battaglia

et al . (2014) An assessment of patient navigator activities in breast cancer patient navigation programs using a nine-principle framework. Health Services Research 49(5): 1555–1577.

35.

Hedlund

Risendal

Pauls

et al . (2014) Dissemination of patient navigation programs across the United States. Journal of Public Health Management and Practice 20(4): E15–E24.

36.

Hosoya

Mochinaga

Emoto

et al . (2015) Failure mode and effects analysis of medication adherence in patients with chronic myeloid leukemia. International Journal of Clinical Oncology 20(6): 1203–1210.

37.

Huynh

Snyder

Vidal

et al . (2016) Assessment of the nurse medication administration workflow process. Journal of Healthcare Engineering 2016: 6823185.

38.

Institute for Healthcare Improvement (2017) Failure Modes and Effects Analysis (FMEA) Tool. Available at: http://www.ihi.org:80/resources/Pages/Tools/FailureModesandEffectsAnalysisTool.aspx (accessed 18 July 2018).

39.

Johnson

Onwuegbuzie

(2004) Mixed methods research: a research paradigm whose time has come. Educational Researcher 33(7): 14–26.

40.

Kawai

Takahashi

Ikuta

et al . (1989) Bilateral hypoplastic internal carotid arteries with multiple cerebral aneurysms. Clinical Neuropathology 8(6): 272–275.

41.

Keith

Crosson

O’Malley

et al . (2017) Using the Consolidated Framework for Implementation Research (CFIR) to produce actionable findings: a rapid-cycle evaluation approach to improving implementation. Implementation Science 12(1): 15.

42.

Klin

Wetherby

Woods

et al . (2015) Toward innovative, cost-effective, and systemic solutions to improve outcomes and well-being of military families affected by autism spectrum disorder. The Yale Journal of Biology and Medicine 88(1): 73–79.

43.

Koester

Morewitz

Pearson

et al . (2014) Patient navigation facilitates medical and social services engagement among HIV-infected individuals leaving jail and returning to the community. AIDS Patient Care and STDs 28(2): 82–90.

44.

Lewis

Stanick

Martinez

et al . (2015) The Society for Implementation Research Collaboration Instrument Review Project: a methodology to promote rigorous evaluation. Implementation Science 10(1): 2.

45.

Lins

Gerka

Lüpkes

et al . (2018) Enhancing safety of artificially ventilated patients using ambient process analysis. Studies in Health Technology and Informatics 247: 316–320.

46.

Locke

Beidas

Marcus

et al . (2016) A mixed methods study of individual and organizational factors that affect implementation of interventions for children with autism in public schools. Implementation Science 11: 135.

47.

MacDonald

Pauly

Wong

et al . (2016) Supporting successful implementation of public health interventions: protocol for a realist synthesis. Systematic Reviews 5: 54.

48.

Madison

(2005) Process Mapping, Process Improvement, and Process Management: A Practical Guide for Enhancing Work and Information Flow. Chico, CA: Paton Professional.

49.

Mandell

Wiggins

Carpenter

et al . (2009) Racial/ethnic disparities in the identification of children with autism spectrum disorders. American Journal of Public Health 99(3): 493–498.

50.

Manger

Paxton

Pawlicki

et al . (2015) Failure mode and effects analysis and fault tree analysis of surface image guided cranial radiosurgery. Medical Physics 42(5): 2449–2461.

51.

Markoulakis

Turner

Wicik

et al . (2018) Exploring peer support needs of caregivers for youth with mental illness or addictions concerns in family navigation services. Community Mental Health Journal 54(5): 555–561.

52.

Markoulakis

Weingust

Foot

et al . (2016) The Family Navigation Project: an innovation in working with families to match mental health services with their youth’s needs. Canadian Journal of Community Mental Health 35(1): 63–66.

53.

Marshall

Mbah

Ford

et al . (2016) Effect of patient navigation on breast cancer screening among African American Medicare beneficiaries: a randomized controlled trial. Journal of General Internal Medicine 31(1): 68–76.

54.

Martin

Osterling

(2013) Value Stream Mapping: How to Visualize Work and Align Leadership for Organizational Transformation. New York: McGraw-Hill.

55.

May

Johnson

Finch

(2016) Implementation, context and complexity. Implementation Science 11: 141.

56.

Myers

NAL

Alolayan

Smith

et al . (2015) A potential role for family members in mental health care delivery: the family community navigation specialist. Psychiatric Services 66(6): 653–655.

57.

National Institutes of Health (2017) PAR-17-265: initiation of a mental health family navigator model to promote early access, engagement and coordination of needed mental health services for children and adolescents, May. Available at: https://grants.nih.gov/grants/guide/pa-files/PAR-17-265.html (accessed 19 July 2018).

58.

Neta

Glasgow

Carpenter

et al . (2015) A framework for enhancing the value of research for dissemination and implementation. American Journal of Public Health 105(1): 49–57.

59.

Parikh

Kurzius-Spencer

Mastergeorge

et al . (2018) Characterizing health disparities in the age of autism diagnosis in a study of 8-year-old children. Journal of Autism and Developmental Disorders 48(7): 2396–2407.

60.

Parker

Clark

Leyson

et al . (2010) Patient navigation: development of a protocol for describing what navigators do. Health Services Research 45(2): 514–531.

61.

Paskett

Harrop

Wells

(2011) Patient navigation: an update on the state of the science. CA: A Cancer Journal for Clinicians 61(4): 237–249.

62.

Peavy

Guydish

Manuel

et al . (2014) Treatment adherence and competency ratings among therapists, supervisors, study-related raters and external raters in a clinical trial of a 12-step facilitation for stimulant users. Journal of Substance Abuse Treatment 47(3): 222–228.

63.

Perkins

(2015) Understanding the impact of patient navigation. Journal of Women’s Health 24(7): 544–545.

64.

Pfadenhauer

Gerhardus

Mozygemba

et al . (2017) Making sense of complexity in context and implementation: the Context and Implementation of Complex Interventions (CICI) framework. Implementation Science 12: 21.

65.

Pojasek

(2013) Lean, six sigma, and the systems approach: management initiatives for process improvement. Environmental Quality Management 13(2): 85–92.

66.

Powell

Waltz

Chinman

et al . (2015) A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science 10: 21.

67.

Proctor

Powell

Baumann

et al . (2012) Writing implementation research grant proposals: ten key ingredients. Implementation Science 7: 96.

68.

Ramachandran

Freund

Bak

et al . (2015) Multiple barriers delay care among women with abnormal cancer screening despite patient navigation. Journal of Women’s Health 24(1): 30–36.

69.

Rienzi

Bariani

Dalla Zorza

et al . (2015) Failure mode and effects analysis of witnessing protocols for ensuring traceability during IVF. Reproductive BioMedicine Online 31(4): 516–522.

70.

Rohan

Slotman

DeGroff

et al . (2016) Refining the patient navigation role in a colorectal cancer screening program: results from an intervention study. Journal of the National Comprehensive Cancer Network 14(11): 1371–1378.

71.

Saldana

Chamberlain

Wang

et al . (2012) Predicting program start-up using the stages of implementation measure. Administration and Policy in Mental Health 39(6): 419–425.

72.

Shattuck

Grosse

(2007) Issues related to the diagnosis and treatment of autism spectrum disorders. Mental Retardation and Developmental Disabilities Research Reviews 13(2): 129–135.

73.

Shebl

Franklin

Barber

(2012) Failure mode and effects analysis outputs: are they valid? BMC Health Services Research 12(1): 150.

74.

Sineath

Lambert

Verga

et al . (2017) Monitoring intervention fidelity of a lifestyle behavioral intervention delivered through telehealth. MHealth 3: 35.

75.

Society for Implementation Research Collaboration (2018) Instrument repository. Available at: https://societyforimplementationresearchcollaboration.org/measures-collection/ (accessed 26 September 2018).

76.

Sorrentino

(2016) Use of failure mode and effects analysis to improve emergency department handoff processes. Clinical Nurse Specialist 30(1): 28–37.

77.

St Amant

Schrager

Peña-Ricardo

et al . (2018) Language barriers impact access to services for children with autism spectrum disorders. Journal of Autism and Developmental Disorders 48(2): 333–340.

78.

Stahmer

Reed

Lee

et al . (2015) Training teachers to use evidence-based practices for autism: examining procedural implementation fidelity. Psychology in the Schools 52(2): 181–195.

79.

Terezakis

Pronovost

Harris

et al . (2011) Safety strategies in an academic radiation oncology department and recommendations for action. Joint Commission Journal on Quality and Patient Safety 37(7): 291–299.

80.

Thomas

Ellis

McLaurin

et al . (2007) Access to care for autism-related services. Journal of Autism and Developmental Disorders 37(10): 1902–1912.

81.

Vaismoradi

Turunen

Bondas

(2013) Content analysis and thematic analysis: implications for conducting a qualitative descriptive study. Nursing & Health Sciences 15(3): 398–405.

82.

Valaitis

Carter

Lam

et al . (2017) Implementation and maintenance of patient navigation programs linking primary care with community-based health and social services: a scoping literature review. BMC Health Services Research 17(1): 116.

83.

Walsh

Mazor

Roblin

et al . (2013) Multisite parent-centered risk assessment to reduce pediatric oral chemotherapy errors. Journal of Oncology Practice 9(1): e1–e7.

84.

Willis

(1992) Failure modes and effects analysis in clinical engineering. Journal of Clinical Engineering 17(1): 59–63.

85.

Wood

McLeod

Klebanoff

et al . (2015) Toward the implementation of evidence-based interventions for youth with autism spectrum disorders in schools and community agencies. Behavior Therapy 46(1): 83–95.

86.

Yarmohammadian

Abadi

TNB

Tofighi

et al . (2014) Performance improvement through proactive risk assessment: using failure modes and effects analysis. Journal of Education and Health Promotion 3: 28.

87.

Yates

Schumacher

Norman

et al . (2013) Intervention fidelity in a translational study: lessons learned. Research and Theory for Nursing Practice 27(2): 131–148.

88.

Yingling

Hock

Bell

(2018) Time-lag between diagnosis of autism spectrum disorder and onset of publicly-funded early intensive behavioral intervention: do race-ethnicity and neighborhood matter? Journal of Autism and Developmental Disorders 48(2): 561–571.

89.

Yousefinezhadi

Jannesar Nobari

Behzadi Goodari

et al . (2016) A case study on improving intensive care unit (ICU) services reliability: by using process failure mode and effects analysis (PFMEA). Global Journal of Health Science 8(9): 52635.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.44 MB