Intervention fidelity in a large scale model demonstration project: Lessons from Maryland PROMISE

2.1 PROMISE participants

Maryland PROMISE recruited and enrolled 2,007 Maryland youth between the ages 14–16 receiving Supplemental Security Income (SSI) benefits and their families during April 2014 through February 2016. Eligible youth and their families who consented to participate were randomized into one of two groups: 1) enhanced services group (n = 997) and 2) usual services group (n = 1,010). The intervention services received by the enhanced services group are the subject of this article which covers services delivered through September 30, 2018.

2.2 The PROMISE intervention

Maryland PROMISE designed an intervention that was guided by both the original federal Request for Proposal (RFP) which required that services be delivered to both eligible youth and their families, and extant research that identifies optimal transition services for youth with disabilities. Emerging research points to family engagement in youth transition and employment pursuits and work experiences during secondary school as strong predictors of post-school employment outcomes (Wehman et al., 2014). In particular, there is a growing body of contemporary evidence that work experience and paid integrated employment during secondary school years predicts successful post-school employment (Carter, Austin & Trainor, 2011; Gold, Fabian & Luecking, 2013; Test et al., 2009; Wehman, et al., 2014). Additionally, there is documentation of the need for low income families to develop financial literacy as they begin to increase earnings (Amsbaugh, 2007). And finally, connections to, and navigation of, the multiple services that may benefit youth with disabilities and their families has been identified as a frequent need (Wehman, et al., 2014). The Maryland PROMISE intervention featured components that recognize the need for research based comprehensive services to both youth and to their families and includes the following:

Assertive case management: This entails proactive and ongoing coordination of services for the youth and their families so that they can navigate through the services and supports available through the project and through the larger service delivery system for which they are eligible and which may be required to achieve desired outcomes.

The Maryland PROMISE intervention was organized through a team of professionals including a case manager, family employment specialist, benefits specialist, and, when necessary, school personnel who work cooperatively and intensively with youth and families to deliver PROMISE services. Teams operated in each of five state regions: Baltimore, Eastern Shore, Northern Maryland, Southern Maryland, and Western Maryland. Statewide, a total of 27 teams delivered the intervention. In partnership with the Maryland Department of Disabilities, the PROMISE treatment intervention was delivered by Way Station, Inc., an organization with a statewide community and employment service program for individuals with disabilities. TransCen, Inc., an organization experienced in transition interventions and broad scale research projects, provided technical assistance to support program fidelity.

2.3 Intervention fidelity

In order to reinforce intervention fidelity, a key feature of Maryland PROMISE was a continuum of technical assistance (TA) designed to build the capacity of the intervention teams to effectively implement the PROMISE service components. TransCen assigned experienced professionals to provide regular in-the-field training and assistance in implementing all aspects of the intervention. Way Station staff attended regular training events and participated in other learning opportunities to gain the knowledge and skills necessary to deliver intervention components. Each region was assigned a TA liaison from TransCen. The liaison provided guidance and support to the project team to promote intervention fidelity and resolve implementation issues as they occurred including field-based TA and one-on-one troubleshooting by the TA liaison with individual staff members.

The delivery of project services are documented by Way Station staff in a management information system customized for this project. Thus, TA was informed by data from the project’s management information system which documented service delivery activity and youth/family participation in components of the intervention. For example, documentation of staff contact with employers to facilitate work experiences might show a need to increase staff efforts and skill in this area of project services.

2.4 Measuring fidelity

At the start of the Maryland PROMISE project, six program services or components were identified as essential fidelity indicators measuring the extent to which the intervention was accurately and reliably implemented in adherence to the proposed model. They include:

Family Plan which is a regularly updated plan for identifying and coordinating services to meet both short and long term goals for participating youth and family members, including employment and education;

Recognizing that various factors could inhibit perfect service intervention delivery, we derived a ratio positing that receipt of three to five of these intervention components constitutes partial fidelity. That is, participants received a moderately strong, but not the full, dose of the intervention. Receipt of all six intervention components constituted ideal fidelity. Maryland PROMISE established overall goals for delivering each of the six intervention components as both a measurement for calibration of the dose of intervention delivery and as a management tool for monitoring delivery of service components by project staff so that technical assistance and support could be targeted accordingly.

To determine if there were significant differences in the fidelity of the intervention by region and by disability categories, we used a chi-square test of goodness-of-fit along with effect sizes (Cramer’s V). A non-significant result on for this test suggests that any differences by subgroups are due to chance, while significant results suggest that there may be non-random differences in the proportion of youth who receive the fidelity services by region or disability.

3.1 Service fidelity by component and state region

From the initiation of Maryland PROMISE service delivery in April 2014 through the cessation of PROMISE service delivery on September 30, 2018, fidelity measures reveal an overall consistent level of service delivery. Family Plans were completed for 94% of all PROMISE youth and their families. A Positive Personal Profile was completed for 87% of all PROMISE youth, Job Development Plans were created for 87%, 81% received Benefits Counseling services, 81% participated in an Unpaid Work Experience, and 73% achieved at least one paid employment experience.

A chi-square test of goodness-of-fit was performed to determine whether the youths’ receipt of fidelity services was equally distributed by region for each service. The number of youths receiving Family Plans, Positive Personal Profiles, and Unpaid Work Experiences, was approximately equally distributed by region, X² (4) = 6.03, 7.85, and 7.04; all p > .05. Meanwhile, the number of youth receiving Job Development Plans, Benefits Counseling, and Paid Work Experiences was not evenly distributed by region, X² (4) = 13.37, 13.39, and 17.69; p < .05,.05,.01. However, the effect sizes for these differences were in the small range (Cramer’s V < .15), suggesting that any differences in fidelity were marginal. Table 1 below illustrates the descriptive results and the chi-square analysis. The target goals for each component, which were established for project management purposes, are provided at the bottom of the Table.

Given the number of disability categories with very low incidence, and for ease of comparison, we condensed the twenty-three impairment categories used by the Social Security Administration into six general disability categories. For example, participants with musculoskeletal disorders (total n = 3) circulatory disorders (n = 3), diseases of the blood (n = 11) and other medical disorders were combined into the single category of “medical disorders.” The six disability categories are reflected in Table 2 below. Across disability categories, the percentage that experienced each of the six service components was at or near the established target with two exceptions. Sixty-two (62) percent of participants with autism spectrum disorders and 63% of those with sensory disability have experienced paid work, compared to 73% of all participants. A chi-square test of goodness-of-fit was performed to determine whether the youths’ receipt of fidelity services was equally distributed across all disability categories for each service. The number of youth receiving services was unequally distributed only for Paid Work, X² (4) = 12.63, p < .05, and the effect size was in the small range (Cramer’s V < .13), suggesting that any differences in fidelity were marginal. Table 2 illustrates the results across each disability category.

Finally, the levels of fidelity achieved were consistent across state regions as illustrated in Table 3. Ideal fidelity, that is, meeting all six fidelity measures, was achieved for 66% of all participants. Eleven percent (11%) received five of the six service components. The percentage of participants who received four or three service components was 7% and 4% respectively. Thus, at least partial fidelity was achieved for 88% of the participants, that is, they received at least three of the six service components. A chi-square test of goodness-of-fit was performed to determine whether the number of service components delivered was equally distributed by region. Only the number of youth receiving six service components and one service component was unequally distributed by region, X² (4) = 14.12, 13.77, both p < .01, and the effect sizes were both in the small range (Cramer’s V < .15), suggesting that any differences in fidelity were minimal.

4.1 Limitations

This analysis utilized data provided by in-the-field staff who entered service information at various points. They also uploaded the actual documents related to service delivery, such as the Family Service Plan, the Positive Personal Profile, and the Job Development Plan which provided direct evidence of their completion. Additionally, they recorded the duration of the unpaid and paid work experiences. However, the data used here only counted the incidence, not the duration or any other aspect, of these two services. Similarly, participants received Benefits Counseling Services which ranged from basic information about the existence of work incentives to direct help in calculating and reporting earnings to the Social Security Administration. Thus, we can draw conclusions only from the occurrence of these services rather than their relative intensity. Subsequent analysis of participant outcomes relative to both specific and overall intervention fidelity will need to take this into consideration.

4.2 Implications

A well-structured intervention is critical to examining its effects in an RTC study. However, the intervention alone, without assurances of fidelity, will not be sufficient to gauge achievement of project intent and outcomes. Setting project targets and using quantitative data reports to monitor fidelity of the intervention can serve as powerful management tools to suitably conduct the delivery of services consistent with project purposes. It also might be inferred that any transition service, irrespective of association with a research initiative, would benefit from the use of performance management systems that would support similar quantitative reporting on staff activities, along with consequent professional development activities for transition and employment staff who are tasked with delivering program services.

Ultimately, since we will be able to state how closely the intervention in Maryland PROMISE was delivered as intended, we can determine how well the intervention has addressed the challenges faced by youth who, by virtue of SSI eligibility, simultaneously experience significant disability and low family income levels.