Early findings from the Wisconsin PROMISE project: Implications for policy and practice

2.1 Participants

Participants in this study were recruited from a list provided by SSA of all eligible participants (9,249 youth) from April 2014 to April 2016. Youth were eligible if they were 14, 15, or 16 years old, receiving SSI and submitted a signed consent to participate and intake form on the day of enrollment. Wisconsin PROMISE enrolled 2,024 teenagers receiving SSI. Once the Wisconsin Promise Formative Evaluation Team received the enrollment packet, they inputted the youth into the online Random Assignment System developed by Mathematica Policy Research (MPR), the national evaluator for PROMISE contracted through SSA. Participants were then randomly assigned to either the treatment or control group. Of those enrolled in the study, 1,018 were randomly assigned to the Wisconsin PROMISE Services group and 1,006 continued services as usual group. Youth randomly assigned to the treatment group were automatically enrolled in Wisconsin’s DVR services (presumed eligible because receiving SSI) where they received both DVR and Wisconsin PROMISE services. Family members who signed the consent form could receive educational, employment, and financial self-sufficiency services and supports through the Family Service Plan (FSP) connected to the youth’s PROMISE DVR case. Youth and family members randomly assigned to the control group received information about Wisconsin’s usual transition services. All participants received two gift cards to a local grocery or retail chain store, $15 for enrolling and $15 for completing the intake form. One youth from the control group and seven treatment youth withdrew from Wisconsin PROMISE.

2.2 Measurement

The University of Wisconsin Stout Institutional Review Board (IRB) reviewed the research study design and measurement tools and approved the study for Human Subjects participation. All participants in both the treatment (1,018) and control (1,006) completed the intake form at enrollment. The intake form included identifying and contact information, household related information including income and public assistance; youth information including age, race, ethnicity, gender, and education; youth health status, including self-reported disability type; and parent and other household member information.

Service data. Wisconsin PROMISE service providers reported on service delivery through a secured Data Collection site set up via SharePoint. Wisconsin PROMISE work incentive benefits specialists recorded service length, number of contacts for both consult and analysis, number of service hours, work incentives, overpayments, age 18 redetermination, and challenges. Family advocates recorded completion of family advocacy trainings and if they assisted youth with the completion of the self-advocacy training. In addition, family advocates case noted all their contacts and activities with PROMISE youth and family members, recording activity type and date, number of service and travel hours, topics addressed, and case notes. Financial coaches recorded orientation date, information on the youth and family members PROMISE Individual Development Accounts (IDAs), and case notes.

Administrative data. In addition to the project specific data collected, the PROMISE Management Information System (MIS) included administrative data from multiple administrative data through inter-agency data sharing agreements. The PROMISE consent form included permission to share administrative data from these agencies. Administrative data included PROMISE specific data from Wisconsin DVR’s Integrated Rehabilitation Information System (IRIS) which is the Wisconsin DVR Case Management system that includes all Rehabilitation Service Administration (RSA) 911 data. Additional PROMISE data collected included case note tags (categorizing the nature of the case note), list of the youth’s family members, list of the youth’s resource team, resource team meeting notes, Family Service Plan (services for family members), youth and family member paid work, service completion dates, educational attainment, and engagement ratings.

Tracking rapid engagement and service completion. An effort was made to engage PROMISE youth and family members as quickly as possible, and services were individualized, meeting youth and families where they were at, to help increase the likelihood youth and family members would engage in services. In addition, if youth and family members did not respond to contacts right away, PROMISE staff continued to try to reach out to connect with youth and families to start and continue to engage in services. Wisconsin DVR PROMISE staff effort to implement rapid engagement activity was measured from date of enrollment to date of message, contact or activity where the case note tag of message, consumer contact, counseling, resource team meeting, or authorization of DVR or PROMISE service. This measure includes messages left by voicemail, which may or may not lead to engagement. Therefore, time to consumer engagement was counted by counting the number of days from enrollment to date with a case note with a tag of consumer contact, counseling, resource team meeting, or authorization of DVR or PROMISE service. Wisconsin PROMISE DVR staff also provided a rating of how engaged PROMISE youth and family members were within the PROMISE DVR case management system.

In addition to the DVR data, the PROMISE MIS also included quarterly earnings from the Wisconsin Unemployment Insurance (UI), employment services provided through Wisconsin’s Job Centers, Workforce Innovation and Opportunities Act (WIOA), and Youth Apprenticeship, Medicaid data, including long term care and mental health services, secondary and post-secondary school data, information on food share participation, and family and child welfare data, including Wisconsin Works.

Employment data. Wisconsin UI wage data were used to compare differences in employment rates and wages. Participants were considered to be employed if they had UI wages reported in the time period being studied. Participants were considered to be earning at Substantial Gainful Activity (SGA) if their UI quarterly wages were at least three times the monthly SGA amount. It is important to note that Wisconsin UI wage data only includes wage data from Wisconsin employers, so participants who worked in another state (e.g., Minnesota or Illinois) or for the federal government would not have wages reported in the Wisconsin UI system. Therefore, employment rates and wages are slightly underreported, but this holds equally true for both treatment and control participants allowing for a direct comparison between groups.

Educational attainment. In addition to connecting PROMISE youth and family members to employment opportunities, PROMISE DVR Counselors helped connect youth and family members to post-secondary education opportunities, helping to pay for these opportunities through training grants funded by DVR or PROMISE. Through a data sharing agreement with the National Student Clearinghouse, post-secondary educational enrollments and attainments of all PROMISE participants were added to the PROMISE MIS. Of note, only educational enrollments and attainments from institutions who agree to send their data to the National Student Clearinghouse is included in this data set, although most post-secondary institutions participate in this data sharing effort. Enrollments and attainments included in this report cover spring 2014 to summer 2018 for attainments and fall 2018 for enrollments.

Data Analysis. Differences in employment rates were analyzed in SPSS using a chi-square analysis to detect differences between categorical data (employed yes or no). Visual analysis was used to observe differences over time.

3.1 Demographics of Wisconsin PROMISE youth

Demographics are summarized in Table 1. At enrollment, Wisconsin PROMISE youth self-reported a primary disability of mental health or behavioral (34 percent), developmental or intellectual (30 percent), and other (e.g., learning, speech, ADHD, etc.; 25 percent), and a smaller percentage reported other disabilities such as sensory, physical, and long-term illness. Other notable demographics included: 49 percent of youth were African American, 36 percent white, 10 percent Hispanic; 67 percent male; 66 percent living in a single parent/guardian household; 85 percent reported good overall health; 70 percent reported an annual household income at or below $25,000; 84 percent had an Individualized Education Plan at enrollment; and 77 percent were attending a traditional high school or middle school. All youth were receiving SSI and Medicaid at the time of enrollment. In addition, 60 percent of families reported someone in the household receiving Social Security Disability Insurance (SSDI), 67 percent reported receiving Food Share (Wisconsin’s version of the Supplemental Nutrition Assistance Program), 22 percent reported other household members receiving SSI, 12 percent reported housing assistance, and 4 percent were receiving Wisconsin Works (W2) or Temporary Assistance for Needy Families (TANF).

Wisconsin DVR PROMISE staff effort to engage in rapid engagement as recorded by counselor case notes was most frequently one day (mode) with a median of four days and a mean of eight days. Time to consumer engagement in services was slightly longer with the mode still at one day, a median of eight days, and an average of 26 days. The majority (89 percent) of PROMISE youth had at least one face to face meeting with their Wisconsin PROMISE DVR Counselor. An additional eight percent of PROMISE youth had at least one phone conversation with their counselor. Most PROMISE youth (88 percent) had a signed DVR Individual Plan for Employment (IPE). After the initial IPE was written, engagement ratings ranged from highly engaged in services (50 percent) to mixed engagement (23 percent) to low engagement (27 percent).

3.3 PROMISE service completion

Most PROMISE youth (90 percent) had at least one DVR or PROMISE funded service aimed to help them reach their employment goals, and 88 percent of PROMISE youth had an identified resource team. Almost half of the PROMISE youth (48 percent) also had a Family Service Plan serving 652 different family members, and 109 Wisconsin PROMISE treatment family members had their own DVR case compared to 82 control family members. PROMISE treatment youth completed a variety of DVR and PROMISE services including 713 PROMISE youth receiving employment services, 594 met with a PROMISE family advocate, 576 met with a financial coach, 535 met with a work incentive benefits specialist, 532 completed work-related social skills training, 476 completed self-advocacy modules, and 351 completed health promotion fact sheets. In addition, 284 youth and 230 family members had a PROMISE Individual Development Account (IDA).

3.4 Work experience

Youth who received Wisconsin PROMISE Services were more likely to have a DVR case, engage in employment services, and have paid work than were youth randomly assigned to services as usual. From April 2014 to September 2018, 100 percent of PROMISE youth in the treatment group had a DVR case, compared to 33 percent youth in the control group. DVR services were correlated with higher employment rates (Kaya et al., 2016; Rumrill et al., 2017), and youth receiving PROMISE services were more likely to be employed.

During PROMISE services, from April 2014 to September 2018, 67 percent of PROMISE treatment youth had reported UI wages compared to 57 percent in the control group. A chi-square test of independence was performed, and PROMISE Services had a significant impact on employment rates, X² (1, N = 2016) = 23.37, p < .001. About two thirds of control youth never connected to DVR as part of usual services. The employment rate was even lower for youth in the control group without a DVR case at 51 percent. Despite the higher employment rates observed with treatment youth, an equal number of treatment and control youth had SGA equivalent earnings reported in at least one quarter during the service period from April 2014 to September 2018, 78 (eight percent) treatment youth and 78 (eight percent) control youth.

Figure 1 illustrates quarterly employment rates for treatment and control youth over time from prior to enrollment in Wisconsin PROMISE services starting the first quarter of 2013 until the last quarter of Wisconsin PROMISE services the third quarter of 2018. Youth were 14 to 16 years old when they enrolled in Wisconsin PROMISE, so employment rates prior to enrolling in Wisconsin PROMISE were low, one to two percent for both youth in treatment or control groups. After enrollment started in April, the second quarter of 2014, employment rates started to increase for both groups but at a higher rate from treatment group youth compared to control group youth. By the end of the enrollment period in April the second quarter of 2016, 16 percent of treatment youth had UI wages compared 12 percent in the control group. After the enrollment period, differences between the two groups increased. Employment rates by quarter were higher and more variable for youth in the treatment group with higher employment rates observed during quarters that included summer months. June is in the second quarter of the calendar year and even higher employment rates were observed in quarter three, which includes both July and August. By the last quarter of Wisconsin PROMISE service delivery, 41 percent of treatment youth and 35 percent of control youth had reported wages. The percent of control youth with reported wages during this quarter drops to 31 percent when only including youth in the control group without a DVR case.

OPEN IN VIEWER

Fig.1

Percent Youth with UI Wages by Quarter and Treatment Group Figure Caption: Percent of youth who were 14 or older with wages reported to Wisconsin UI in calendar quarters from quarter one of 2013 to quarter three of 2018. Percent of treatment youth with reported wages is represented by the dashed line with closed squares, control group is represented by the solid line with closed triangles, and the subset of the control group without a VR case is represented by the solid line with open circles.

Not only were treatment group youth employment rates higher than observed for the control (usual) services group but PROMISE youth’s employment rate also increased to a rate comparable to youth without disabilities in Wisconsin. According to the Community Population Survey, 53 percent of Wisconsin youth ages 16 to 19 were employed from July 2017 to June 2018. In comparison, 53 percent youth who received Wisconsin PROMISE Services had UI reported wages during this same period compared to 43 percent in the full control group and 38 percent of youth in the control group without a DVR case. In other words, youth with SSI who received PROMISE Services increased their employment rates comparable to the employment rates observed for youth without disabilities.

Employment rates for Wisconsin PROMISE family members increased at similar rates for both treatment and control groups. In the second quarter of 2014, the quarter when Wisconsin PROMISE enrollment began, 39 percent of parents in both the treatment and control groups had Wisconsin reported wages. Employment rates increased for both groups over time, and by the last quarter of services, July through September of 2018, half of the parents had UI wages reported, slightly higher in the control group 50 percent (514 of 1036) than the treatment group 48 percent (512 of 1073). During the full PROMISE service period, from April 2014 to September 2018, 66 percent (685 of 1036) of parents in the control group had a quarterly wage reported in at least one quarter during this period compared to 68 percent (729 of 1073) of parents in the treatment group. A chi-square test of independence was performed, and PROMISE Services did not have a significant impact on parent employment rates, X² (1, N = 2109) = 0.79, p = .37. According to the Bureau of Labor Statistics, during this period Wisconsin unemployment rates decreased from 5.7 percent in April 2014 to 3.0 percent in September 2018, so increases in parent employment rates may have been related to the improved economy.

Household members experienced similar employment rate increases across treatment and control groups with a few notable differences. It is important to note that household members included both siblings and adult family members, because siblings were usually around high school age their employment rates were lower, so average household employment rates started lower than observed with the parents. In addition, more household members in the treatment group signed the consent form to participate in Wisconsin PROMISE presumably because those in the treatment group could receive services to help them reach their education or employment goals through the PROMISE Family Service Plan. At enrollment, 18 percent (52 of 287) of treatment household members age 14 or older reported UI wages compared to 10 percent (12 of 119) control household members. This increased to 46 percent (159 of 346) for treatment household members and 48 percent (60 of 126) for control household members by the third quarter of 2018. Throughout the PROMISE Services period, 62 percent (178 of 346) of treatment household members and 61 percent of control household members (77 of 126) had reported wages in at least one quarter between April 2014 and September 2018.

Wisconsin PROMISE is a package of services including wraparound case management, which included close collaboration with schools, employment services, financial coaching, work incentive benefits counseling and family and self-advocacy. Each of these services were positively correlated with employment outcomes.

Collaboration with schools. At least one person from school was identified on 614 of the 884 (69 percent) resource teams. Youth with school staff included on their resource team had higher employment outcomes with 70 percent with reported UI wages during the service period compared to 63 percent of youth with reported wages for those with no resource team or a resource team that did not include school staff [X² (1, N = 1011) = 5.33, p = .021].

Employment services. Wisconsin PROMISE youth had access to all employment services available through DVR, which allowed for the individualization and customization of employment services. Throughout PROMISE, 713 (71 percent) of PROMISE youth received employments services, which included employment assessment services, Skills to Pay the Bills (soft skills training), job exploration counseling, job search assistance, temporary work experience, short-term job supports, work-based learning, on the job training, and supported employment services. These employment services were positively correlated with employment outcomes. Of the 372 PROMISE youth who received employment assessment services 77 percent had reported wages in at least one quarter between April 2014 and September 2018. In comparison, 62 percent of youth who did not receive employment assessment services had wages reported during this period [X² (1, N = 1011) = 21.96, p < .001]. Even higher rates were observed for the 532 youth who received Skills to Pay the Bills training with 81 percent of youth who received the training with reported wages compared to 53 percent of youth who did not receive this training [X² (1, N = 1011) = 88.03, p < .001]. PROMISE youth who received job exploration counseling and job search assistance also had higher employment rates. Of the 137 with job exploration counseling, 81 percent with reported wages compared to 65 percent of youth who did not receive job exploration counseling [X² (1, N = 1011) = 12.26, p < .001], and 91 percent of the 308 youth who received job search assistance had UI wages [X² (1, N = 1011) = 109.5, p < .001]. Employment rates were even higher for youth who received on the job supports. More specifically, 96 percent of the 308 youth who received temporary work experience were employed after this work experience [X² (1, N = 1011) = 150.9, p < .001]; 95 percent of the 285 youth who received short-term job supports had a paid job after receiving these supports [X² (1, N = 1011) = 127.64, p < .001]; 93 percent of the 304 youth who participated in a work based learning experience had a paid job after this work experience [X² (1, N = 1011) = 126.03, p < .001]; 95 percent of the 22 youth who received on the job training were working after this training [X² (1, N = 1011) = 8.03, p = .005]; and 100 percent of the 20 youth who received supported employment services were working before PROMISE services ended in September 2018 [X² (1, N = 1011) = 9.84, p < .002]. The impact of these services on employment rates are illustrated in Figure 2.

OPEN IN VIEWER

Fig.2

Employment Service Impact on Employment Outcomes Figure Caption. Percent of PROMISE youth who had a job following the receipt of a DVR employment service. N is the count of PROMISE youth who received the service. The number of youth who did not receive the service can be calculated by subtracting 1011 – N.

Financial services. Meeting with a work incentive benefits specialist, financial coach, and having a savings account were all positively correlated with youth employment rates. Of the 535 PROMISE youth who met with a work incentive benefits specialist, 78 percent had reported wages compared to 55 percent of those who did not meet with a benefits specialist [X² (1, N = 1011) = 58.9, p < .001]. Furthermore, 81 percent of youth earning SGA met with a work incentives benefits specialist. Youth that met with a work incentive benefits specialist had three times as many jobs and three times the total weekly earnings as youth who did not meet with a work incentive benefits specialist. Similarly, 79 percent of the 576 youth who met with a financial coach had reported wages while enrolled in Wisconsin PROMISE compared to 52 percent of the youth who did not meet with a financial coach [X² (1, N = 1011) = 78.71, p < .001], and 87 percent of youth earning SGA met with a financial coach. Youth who met with a financial coach had five times as many jobs and weekly earnings compared to those who did not meet with a financial coach. Less youth reported having a savings account while in PROMISE, but those who did also reported higher employment rates, with 83 percent of the 377 youth with a savings account with reported wages [X² (1, N = 1011) = 68.63, p < .001].

Family and self-advocacy. Family and self-advocacy training was also related to increased employment outcomes. Of the 594 youth whose families met with a PROMISE family advocate, 71 percent worked during PROMISE [X² (1, N = 1011) = 8.43, p < .004]. This employment rate was even higher for youth whose families completed the family advocacy training on navigating transition services and supports and employment possibilities for youth with disabilities with 77 percent of the 414 youth with families who completed the training who had a job while enrolled in PROMISE. Of families who completed the Wisconsin PROMISE family advocacy training, 90 percent agreed the training elevated their expectations of what is possible for their child after high school, 84 percent reported knowing about transition supports for youth, 71 percent reported using more and different resources, and 50 percent reported increases in regular job opportunities for PROMISE youth.

Like family advocacy training, employment rates were correlated with self-advocacy training. Of the 476 PROMISE youth who completed self-advocacy training 79 percent were employed compared to an employment rate of 57 percent for youth who did not complete self-advocacy training [X² (1, N = 1011) = 52.54, p < .001]. After self-advocacy training, 84 percent of PROMISE youth reported that their abilities help them handle difficult events, and 97 percent reported they know what they need, what they like, and where they excel.

Wisconsin PROMISE youth post-secondary enrollment and attainments were similar across both research groups with around 16 percent of youth enrolling in at least one post-secondary program (165 of 1011 treatment youth and 157 of 1005 control youth). Only two treatment youth and one control completed their post-secondary training as of the Summer of 2018, but these low rates may be attributed to the fact that as of August 31, 2018, 34 percent of Wisconsin PROMISE youth were under age 18, 27 percent were 18 years old, 23 percent were age 19, 14 percent were 20, and only one percent were 21 years old. Given that youth with disabilities can remain in high school until they are 21, there simply was not enough time for most PROMISE youth to reach a post-secondary educational attainment during the study period.

Similar to the education enrollments observed for youth, there were slight observable differences in Wisconsin PROMISE parent and household educational enrollments and attainments although these differences were not statistically significant. From spring 2014 to fall 2018, 191 of 1073 (18 percent) treatment parents and 81 of 346 (23 percent) treatment household members had a post-secondary enrollment compared to 177 of 1036 (17 percent) control parents and 26 of 126 (21 percent) control household members. Parent and household members in the treatment group also had more educational attainments from spring 2014 to summer 2018 with 53 of 1073 (5 percent) of treatment parents with 79 educational attainments compared to 41 of 1036 (4 percent) of control parents with 51 educational attainments. There were a wide range of educational attainments ranging from certificates to diplomas to associate degrees to bachelor’s degrees to master’s degrees. Fewer household members had reached an educational attainment by Summer 2018, five treatment household members and two control household members. Again, household members tended to be younger (e.g., siblings), so few were old enough to reach a post-secondary attainment.

4.1 Limitations

WIOA implementation occurred while PROMISE enrollment and services were beginning. Because WIOA services have a focus on transition aged youth, the timing of WIOA role out likely had an impact on employment outcomes for PROMISE youth in both the control and treatment groups, thus potentially reducing the overall impact of Wisconsin PROMISE services had in comparison to the new reality of usual services, especially for the third of youth in the control group who received DVR services during the PROMISE service period. In addition, because PROMISE youth enrolled when they were as young as 14, DVR and services providers needed time and training to adjust their service models to better serve younger youth. Prior to PROMISE and WIOA, the service system was more accustomed to serving adults than youth under 18, so implementation of PROMISE services took longer than originally anticipated. For these reasons, the full impact of early work experiences is just starting to be realized. More data needs to be collected over a longer period of time to truly evaluate the long-term impacts PROMISE services had on education, employment, and financial self-sufficiency.

4.2 Recommendations for Future VR Practice

Recommendations moving forward include (a) continued data sharing across state agencies to ensure effective inter-agency service coordination, (b) connect youth receiving SSI to employment-focused services and supports through schools, state VR agencies, long-term care, mental health, and others, and (c) continued interagency commitment to supporting employment and career pathways for youth with disabilities, including those receiving or eligible for SSI benefits. Future efforts should include a focus on increasing credentials and measurable skills gains as both lead to better jobs and career development. While entry-level jobs are appropriate while in high school and early adulthood, it is important that youth with disabilities have the opportunity to grow in their careers by working more hours and increase their wages. The goal is to decrease reliance on public benefits and increase the likelihood of financial self-sufficiency in the future, and in order to do so, competitive integrated employment serves as the foundation.

Furthermore, service agencies can learn from PROMISE implementation by ensuring trainings and supports are provided where youth are at whether that be in their homes, in the community, at schools, or on the job, and providing training in an inter-active model that allows for practice and shares information over time as needed rather than trying to share all the information at once in a passive setting. Finally, it is recommended that agencies work together to ensure that family members, not just the youth, have the support needed to navigate services and systems and information on post-school possibilities for all youth with disabilities.

Wisconsin PROMISE has demonstrated that more youth receiving SSI can be connected to employment than typically have been through services as usual. Youth with disabilities continue to show their skills, strengths, and potential can help build a better workforce in all local communities. Models that empower youth and their families while at the same time provide the needed disability related supports and connection to paid work can increase employment outcomes and self-sufficiency.