The financial viability of evidence-based supported employment for people with mental illnesses in a blended funding system

2.1 Study design

Ethics approval was provided by the Newcastle University Human Research Ethics Committee. Written approval was also obtained to access relevant Australian Government and ORS records for the purposes of this investigation. The approved study was a three cohort observational conversion design where an existing DES service (cohort 1, n = 107) was evaluated over 18 months, then converted into an IPS enhanced service and a new cohort of clients was evaluated for a further 18 months (cohort 2, n = 68). Results for both cohorts were then benchmarked to the most relevant national DES cohort (DSS, 2014a) with a similar primary disability type and followed for the same period (cohort 3, n = 15,496).

2.2 Participants

Participants from three cohorts: (1) met national age and residency requirements for the DES program; (2) were diagnosed with a mental illness as the primary health condition; and (3) were not employed and not in full time education at program commencement. Cohorts one and two were mutually exclusive with no shared individuals. Cohorts two and three were also mutually exclusive due to non-overlapping observational periods. However, it is possible that some individuals in cohort one were also counted in cohort three due to the overlapping observational period, and because cohort three is a whole population cohort. This violation of the assumption of mutual exclusion was unlikely to contaminate these results due to cohort one representing 0.7% of cohort 3, with the maximum overlap less than 0.7%.

Cohort one included all ORS Employment Solution DES participants who were recorded as having a mental illness or psychiatric disorder, who commenced DES assistance at the Gosford, Woy Woy, and Lake Haven sites between 1 July 2011 and 30 June 2012. Health and disability information was obtained from an independent assessment conducted for each participant by an allied health professional employed by the Australian Government Department of Human Services. Each participant in this cohort (n = 107) was tracked for 18 months, matching the follow-up periods in the two other cohorts.

Cohort two (n = 68) received IPS enhanced employment services. All eligible participants from ORS DES sites (Gosford, Woy Woy and Lake Haven) were included who had received less than six months DES assistance at the time of selection to allow tracking of up to 18 months. Participants who were already employed or close to being employed (e.g. had recently attended a job interview and were awaiting an outcome) or were enrolled in full-time education or vocational training courses, were excluded. All participants in the trial of IPS-enhanced services commenced between 24 February 2013 and 24 August 2013, and were followed for 18 months, measured at an individual level. The intake and follow-up periods matched those used in the national DES evaluation (DSS, 2014a).

Cohort three consisted of all DES participants who commenced between 1 July 2010 and 31 December 2010 across Australia with the primary disability type classified as psychological or psychiatric (n = 15496; DSS, 2014a). This cohort provided the national performance benchmarks to compare the effectiveness of both pre-IPS and IPS enhanced employment services.

2.3 DES funding

The Australian Government utilises a unique blend of fee-for-service and results-based funding methods in the national DES program. Contracted DES providers are paid according to a standardised contract which combines these funding methods. All payments made to service providers for three sub-programs under this contract are shown in Table 1. Providers are paid quarterly service fees at commencement of participation, and quarterly for up to 18 months, or until the participant commences employment, when service fees are replaced by employment commencement and employment milestone payments. Not all employment qualifies for the employment commencement fee. The hours of employment must meet the minimum benchmark hours for that individual which were determined prior to program commencement by an independent assessor, an allied health professional employed by the Australian Government Department of Human Services. Two employment milestone fees were paid if employment was maintained at or near an individuals benchmark hours for 13 weeks, and 26 weeks respectively (DSS, 2013). Only one 13 week and one 26 week milestone fee could be paid for each participant.

Once 18 months of employment assistance elapsed, all DES participants who were not employed or not enrolled in formal education were referred to the Department of Human Services to assess whether they would benefit from a further six months of DES participation. If assessed as likely to benefit, the program was extended and the DES provider paid a further two service fees if the participant did not commence employment or education. Otherwise results based payments applied. If the participant was assessed as not benefiting from further assistance, the case was closed and the client exited from the caseload (DSS, 2013).

Fees paid to DES providers are shown in Table 1. Funding varied by two program sub-types: Disability Management Services (DMS) and Employment Support Services (ESS). The latter included two levels of case-based funding: ESS Level 1, and ESS Level 2 (DSS, 2014b). A participant was referred to one of these programs following an Employment Services Assessment where a Department of Human Services allied health professional assessed the participant and their disability (DSS, 2014c). Individuals with a disability, injury or illness not considered permanent (lasting two years or more) or with post employment support needs of six months or less were eligible for DMS. Individuals with a permanent disability and a need for ongoing support post employment were assessed as eligible for ESS (DSS, 2013, 2014c). This assessment utilised information from general medical practitioners, medical specialists and previous employment service providers. It also assessed participants’ future work capacity (known as benchmark hours) to be expected on completion of the employment services intervention.

There were two types of employment milestone payments: pathway and full. Pathway outcomes were paid when a participant worked in excess of 66% of their pre-identified benchmark hours, on average, across 13 or 26 weeks; or where a participant completed a semester of an education course of at least two semesters in duration, and is 22 years or over, or is between 15 and 21 years and has completed Year 12 (DSS, 2013). A full milestone payment was payable when a participant worked in excess of their benchmark hours, on average across 13 or 26 weeks; or when a participant completed a semester of an education course of at least two semesters in duration, and was a principal carer, or was under 21 years of age and had not completed Year 12, or identified as indigenous (DSS, 2013).

DES providers also received a bonus payment at 13 weeks and 26 weeks if the participant who met these requirements also (1) completed a Vocational Education and Training Certificate 2 course or above, and then obtained employment in a directly related area within a year of completing the course; or (2) by completing a traineeship or apprenticeship (DSS, 2013) (see Table 1).

2.4 The interventions

The pre-IPS cohort consisted of 107 DES participants receiving services from ORS Employment Solutions on the Central Coast of NSW at the Gosford, Woy Woy and Lake Haven sites. The case management method used by ORS was based on the vocational rehabilitation model where one employment consultant assists the participant from commencement to completion of six months or longer in employment. This staff member has access to in house General Medical Practitioners, Psychologists, Occupational Therapists, Physiotherapists, Exercise Therapists, Marketers, Industry Trainers, Job Search Trainers and Recruitment Consultants. ORS is also a registered training organisation and runs accredited training in hospitality, administration, disability services, aged care and retail. All of these courses and services are available to ORS participants. ORS employment consultants also access low cost community and government resources to address needs and overcome barriers where suitable. At ORS, typical caseload sizes were 55 which include participants in post employment support.

The IPS enhanced service (cohort two) commenced on 24 February 2013, and new clients were recruited directly into this program. Existing participants in the pre-IPS employment service who met eligibility criteria were moved into the enhanced program at the time of their next appointment. The IPS enhanced service involved the full implementation of six of eight IPS principles (Drake, Bond, & Becker, 2012, pp. 33-39): a focus on competitive employment; attention to participant preferences; personalised benefits counselling; systematic job development; rapid job search; and time-unlimited and individualized support. The two principles not fully implemented were (1) participation based on consumer choice; and (2) vocational services are integrated with mental health services as much as possible.

2.5 Measures

Fidelity to IPS principles in cohort two services was assessed using the 25-item Supported Employment Fidelity Scale (Bond, Peterson et al., 2012). A six-monthly internal IPS fidelity assessment was conducted at each site by each employment consultant in consultation with the local IPS co-ordinator. External fidelity assessments were conducted by an independent ORS staff member trained by a Dartmouth trained fidelity assessor. Fidelity assessments were conducted on 24 March 2014 at Gosford and Woy Woy and 31 March 2014 at Lake Haven. Pre-IPS (cohort one) fidelity scores were assessed retrospectively by internal assessment using the consensus method.

The focal dependent variable was net revenue per participant. This was calculated from the provider perspective using actual direct revenue less actual direct expenditure. Indirect expenditure such as the broader costs of running ORS Employment Solutions, including staff recruitment and training, were not estimated. Direct revenue and direct expenditure were measured for each participant. Direct expenditure was defined as any money spent on a participant to assist them to overcome barriers to employment. There were two types, internal and external. Internal expenditure was defined as money spent by ORS using internal resources such as other specialist staff employed by ORS. External expenditure was defined as money spent on participants’ bus fares, petrol cards, train fares, interview or work clothing, personal protection equipment, external training courses, ORS wage subsidies to employers, criminal history checks, and external medical services.

2.6 Staffing

Eleven employment consultants were employed by ORS at the three Central Coast sites during the pre-IPS evaluation period. Staff members had caseloads of up to 55 clients each, including employed participants receiving post employment support. Each staff member reported directly to their site manager and also received direct assistance from a DES performance improvement specialist whose job was to manage performance in the DES contracts on the Central Coast. Four staff members were involved in the IPS enhancement intervention; two employment consultants, an IPS coordinator, and a psychologist. All four staff members remained in place for the duration of the evaluation.

2.7 Staff training

ORS staff in both interventions completed a range of generic DES and ORS training programs, in addition to an intensive six-month face-to-face induction program. Internal training modules included the following: Disability awareness; Mental health first aid; Cultural diversity training; Resumes and client marketing; Preventing and managing behaviours of concern and workplace incidents; DES compliance; DES post-placement support and ongoing support; Disability service standards; Quality awareness (e-learning); and Workplace health and safety (e-learning). The completion of DSS e-learning modules was mandatory.

Prior to the commencement of the IPS enhanced intervention, all four staff involved participated in a one-day training program in which the aims of the IPS model were discussed, including the way it would be applied within an ORS setting. The fidelity checklist was discussed in detail. The Dartmouth training videos on YouTube provided practical training about vocational profiles, job development and dual diagnoses (Dartmouth, 2009). Two employment consultants, the IPS coordinator, and the researcher enrolled in and completed the online training program facilitated by Dartmouth IPS Supported Employment Centre (the developers of the IPS approach) in June 2013. This online program was conducted by IPS practitioners with substantial field experience. In addition, each IPS employment consultant met fortnightly, one to one, with the IPS coordinator to discuss fidelity issues and develop strategies for participants for whom they were having difficulty finding suitable employment. Training involved field excursions in which the IPS supervisor conducted face-to-face job development and other marketing activities, to assist in building confidence and skill levels in each employment consultant. Skill levels were assessed during a monthly review of employment outcomes.

The level of training for IPS employment consultants, compared to pre-IPS consultants, was more intensive with more one on one time with supervisors, consisting of one half day to one full day per fortnight of training for the duration of the study. The site manager and the researcher were also available to provide assistance when needed. They sometimes attended the review meetings and assisted with participant issues. All IPS enhanced intervention staff continued to participate in all other ORS and DES training to ensure they remained up to date with contract, compliance and company requirements.

2.8 Data quality

Data quality was actively managed. All records of activities resulting in service fees and outcome fees were audited by DSS in face to face monitoring visits, and by external desktop monitoring exercises. These often involved contact with employers and participants to verify information. ORS income and expenditure were audited annually by an independent auditor. Quality checks were conducted on 10 random pre-IPS participants and 10 IPS enhancement participants to further assess data accuracy. Financial information was checked by a second staff member.

2.9 Data analysis

The data analysis strategy utilised a range of descriptive statistics: frequencies, means, standard deviations and cross tabulations. Statistical significance of group differences was examined using Wald Chi-square, T-tests, and Fisher’s exact test. Multiple logistic regression was used, when data permitted, to assess multivariate relationships to the binary dependent variable. Odds ratios and 95% confidence intervals were reported. Analyses were conducted using STATA version 11 (Stata Corp, College Station, TX, US).

3.1 Participant characteristics

In the pre-IPS cohort (n = 107) 46.7% were male, the mean age was 34.5 years (SD = 13.0), and 18.7% had a severe mental illness (diagnosis of Schizophrenia or Bipolar Affective Disorder). In the IPS enhanced cohort (n = 68) 51.5% were male, the mean age was 30.8 years (SD = 12.8) and 19.1% had a severe mental illness. Other diagnoses included Major Depression, Anxiety Disorders, Posttraumatic Stress Disorder, Personality Disorder and Substance Abuse Disorder. Demographic characteristics were not available for the national DES cohort which consisted of 15,496 participants aged 15 to 64 years classified as having a primary psychiatric disability, who commenced in the program anywhere in Australia between 1 July 2010 and 31 December 2010.

Attrition was defined as those who exited the program before the 18 month follow-up period was completed without a vocational outcome. Exit reasons that counted toward attrition were: transferring to another provider, relocating to a new area, and choosing to exit early (see Table 2). Attrition marginally improved following the IPS enhancement (40.2% , or 2.2% per month; vs. 36.7% or 2.0% per month).

3.2 Fidelity to evidence-based supported employment

Fidelity of the pre-IPS service at all three sites was measured retrospectively and found to be not supported employment (score 63/125) for all sites on the IPS-25 scale (Bond, Peterson, Becker, & Drake, 2012; Drake, Bond, & Becker, 2012; Bond, Becker, & Drake, 2011). Six months after implementing IPS principles, the fidelity score increased to fair (85/125) at all sites, assessed internally by IPS coordinators. At 13 months all three sites reached a good level of IPS fidelity (Gosford and Woy Woy 108/125; Lake Haven 110/125). This final assessment was conducted by an external assessor who previously conducted joint fidelity assessments with a Dartmouth trained IPS fidelity assessor.

3.3 Evidence-based enhancements and employment outcomes

The IPS enhanced cohort (all contracts combined) obtained more employment commencements over 18 months (67.6% , 46/68) than the pre-IPS cohort (56.1% , 60/107). This was despite the client mix becoming more challenging, as indicated by fewer participants allocated to the less intensive DMS contract, and more participants classified at ESS funding level two, after enhancement by IPS principles. In terms of job retention, 26 week employment milestones favoured IPS enhanced practices over pre-IPS (25.0 vs. 18.7%) although the difference was not statistically significant. The IPS enhancement also exceeded national DES 26 week milestones (25.0 vs 20.9%). The pre-IPS program was also effective achieving significantly more employment commencements (56.1 vs. 39.9%) than the national DES cohort, and attained similar 26 week milestones (18.7 vs. 20.9%) where the differences were not statistically significant.

3.4 Financial implications of IPS enhanced services

The financial implications of enhancement by evidence-based practices in supported employment were examined from two perspectives: the provider perspective and the Australian Government perspective. Tables 3–5 show direct revenue and direct costs comparisons between Pre-IPS services and IPS enhanced services. Table 5 disaggregates the total revenue actually received into service fees and outcome fees, by contract and by program type. Direct costs to the Australian Government were obtained from the 2010–2013 evaluation of DES (DSS, 2014a) per participant, per employment commencement, and per 26 week employment milestone. These direct costs represent payments to providers and exclude the overall departmental administration costs borne by the Australian Government.

3.5 Provider net revenue

The IPS enhanced service did not achieve higher gross revenue overall due to the smaller capped caseloads in the enhanced program (see Table 3). However, the IPS enhanced service achieved higher gross revenue per participant under the same blended funding structure ($9062) than pre-IPS services ($7514). From this revenue ORS purchased goods and services to enhance employment prospects. Examples included short training courses, suitable work clothing, allied health professional assistance, wage subsidies to employers, and reimbursement of transport costs. These costs were higher in the IPS enhanced program ($2132 per person versus $1353). Despite increased expenditure, the IPS enhanced program generated more net revenue (gross revenue less direct costs) per participant compared to pre-IPS services ($6929 vs. $6161).

Differences in net revenue per participant also depended on contract type. IPS enhanced services generated more net revenue for those with the greatest assistance needs (ESS funding level two) ($10579 vs. $8080) and less net revenue for ESS funding level one clients compared to pre-IPS services ($3815 vs. $5786). There was also a slight advantage towards the IPS enhanced program for DMS clients ($5284 vs. $4853).

Table 4 shows that the pre-IPS service overall received more revenue from service fees than from vocational outcome fees (62.0% vs. 51.3% for IPS enhanced services). This is an important result showing that as the government moves towards results-based funding, the adoption of IPS practices, particularly for participants classified as funding level two, represents less financial risk to providers (see Table 4).

3.6 Australian Government perspective

The mean expenditure on disability employment services (DES) by the Australian Government per participant and per employment milestone was calculated using the same estimation method as the official evaluation (DSS, 2014a, p. 54). This involved dividing total direct expenditure per contract (service fees, outcome fees and government paid wage subsidies) by the number of participants who commenced receiving assistance in each contract to obtain mean cost per participant. Costs per employment commencement and per 26 week employment milestone were calculated by dividing the total direct expenditure per contract by the number of participants in each contract who had achieved each particular type of employment outcome at least once during the eighteen month period.

Table 5 shows that the IPS enhancement cost more per participant across all contracts compared to the mean direct costs across all DES providers (DSS, 2014a). However, the IPS enhancement program resulted in the lowest cost per employment commencement in all contracts, and in the lowest cost per 26 week employment milestone for ESS level two, and for DMS participants. Pre-IPS services achieved a lower cost per 26 week milestone for ESS Level 1 participants. Thus IPS enhanced services were consistently more cost effective from the government perspective, for those with more intensive assistance needs classified as ESS funding level two.

3.7 Correlates of commencing employment

Bivariate and multivariate logistic regression analyses were conducted to assess the simultaneous effects of independent variables sex, age, funding level, diagnostic category and service type on job commencements within 18 months of commencing supported employment. Both the pre-IPS and IPS enhancement cohorts were combined to form a single cohort of 175 participants for these analyses. Whilst no bivariate or multivariate effects were statistically significant, one result was promising. Participants in IPS enhanced services had 1.64 times greater odds (CI 0.87–3.09 unadjusted), and 1.88 times (CI 0.95–3.76 adjusted) greater odds respectively for commencing employment than participants in pre-IPS services.

4.1 Financial viability of IPS-enhanced services

Anecdotally, some DES providers expressed reluctance to adopt IPS principles through fears of reduced financial viability compared to DES services as usual. This is due to expectations that capped caseloads lead to less revenue in a system where fee for service payments are blended with results based funding (Waghorn et al., 2012; DEA, 2013). Other DES providers also perceive clients with SPMI as a more challenging subgroup for attaining employment milestones. IPS enhanced services require caseloads to be capped at 20 active clients per employment consultant. Whereas, ORS DES services as usual are typical of most providers by allowing larger caseloads of up to 55 clients per employment consultant. This can be an advantage when service fees provide the greatest contribution to revenue and when the diagnostic mix of clients allows for less intensive services. However, this study shows that IPS enhancements are financially viable on a per client basis, particularly when the service is appropriately targeted to ESS funding level two participants. DES providers who have standard caseloads of 40 clients or less per employment consultant are unlikely to experience an overall reduction in net revenue, even if the current emphasis on service fees is retained by the funding system. However, an important caution is that indirect costs such as staff training were not measured in this study. Training costs for IPS staff are likely to be higher than for non-IPS staff in the first year, meaning that any financial advantage of implementing IPS may not accrue until the second year of implementation.

The Australian Government has expressed an interest in moving towards a greater use of results-based funding to replace the current mix of service fees and outcome fees. In this context, adopting IPS practices targeted to those most in need of intensive services, promises to reduce the business risk by increasing financial viability as the government increases the proportion of results-based funding.

4.2 Limitations

One important limitation was the research design. Randomisation was not possible because an existing service was first evaluated then converted to a new program with a new cohort of clients. While the ecological validity of this approach was high, attributing improved outcomes to the program change was hindered by the emergence of systematic differences between cohorts. To counter this reduced internal validity, systematic between-group differences were identified and controlled where possible (De Veaux, Velleman, & Bock, 2011).

Retrospectively evaluating a pre-existing intervention entailed limits to the type of data that could be examined. Some of the variables collected in the IPS enhancement intervention were not previously collected for the pre-IPS services. This excluded potentially useful comparisons between interventions involving: employment benchmarks, intergenerational unemployment, homelessness, literacy and numeracy, means of transport, living alone, ex-offender status, indigenous status, comorbid intellectual disability, and other health condition comorbidity.

Another important constraint was the nature of staff training and supervision provided. This was delivered by ORS staff who had completed the online training modules through the Dartmouth Supported Employment Centre, but who had not personally received formal training in IPS practices. Two types of fidelity assessments were used: (1) an independent fidelity assessment at 13 months was undertaken by a staff member trained by a Dartmouth trained assessor; and (2) self-assessments of fidelity were conducted using the Dartmouth fidelity assessment guidelines applied by consensus between the employment consultant and their supervisor.

Different definitions of mental illness posed problems for this study. DSS (2014a) used a broad definition of mental illness as part of their definition of psychiatric disability that included anxiety and depressive disorders along with substance abuse and autistic spectrum disorders. In addition, the validity of the medical condition codes and diagnostic categories in the national DES cohort remains unclear. This is important, because IPS was specifically designed for adults with severe and persistent mental illnesses, which in practice usually means the psychotic disorders and other equivalently severe and complex cases of mental illness. However, sufficient diagnostic information was available through the official records to compare the client diagnostic mix across cohorts.

4.3 Implications for service providers

The key implication for Australian DES service providers is that it is financially viable for some providers to provide a more intensive IPS approach within the DES program to a subset of clients with a psychological or psychiatric disability who have the most psychiatric disability and employment assistance needs (ESS Funding level two). However, changing existing DES practices to attain good IPS fidelity can take 12 months or more. The costs of implementing new IPS practices can be reduced by using the online training program from the Dartmouth SE Centre, Youtube videos and other Dartmouth resources. Although more costly, it is also possible to engage external trainers and fidelity experts from within Australia and New Zealand to conduct face to face training. The financial advantage to service providers for adopting IPS principles was shown to increase if the government increases utilisation of outcome-based funding.

4.4 Implications for policy makers

The key message for program administrators and policy makers is that a successful shift to good fidelity IPS practices benefits the very clients who currently least benefit from the DES program, namely those independently assessed as ‘ESS Funding level two’. This shift also reduces program costs per 26 week employment outcome, the most valued and most challenging employment outcome examined. Standard DES services delivered as in the pre-IPS services examined here, or as reflected in the the national DES evaluation reports, are less effective and less cost effective per participant and per employment outcome attained.

Another implication is that the financial viability of delivering more intensive services increases when participants’ employment related disabilities and impairments are accurately classified. However, the sensitivity and specificity of current program allocations (DMS or ESS) and funding level assessments, remain unknown since to our knowledge, these properties have never been investigated or reported. However, anecdotal reports from ORS staff suggest that false negatives can occur, where participants with severe mental illnesses are allocated to programs and funding levels that imply low assistance needs. False positives are also reported where people with less severe forms of mental illness are classified as having high needs for assistance.

It is likely that more can be done to improve the accuracy of Employment Services assessments currently conducted by the Department of Human Services (DSS, 2014c). One way to do this would be to include a measure of relative severity of psychiatric disability, informed by variables known to be associated with employment status, such as: diagnostic category (Jonsdottir & Waghorn, 2015), course pattern of illness (Waghorn, Chant, & Whiteford, 2003), current psychosocial impairments (Waghorn, Saha, & McGrath, 2014), and prior medical suspensions due to being too unwell to continue (DSS, 2014a). Collaborations between program administrators and external researchers could jointly investigate ways to measure and improve the accuracy of program type and funding level assessments.

4.5 Implications for researchers

Further implementation studies are needed to identify and strengthen the evidence-based practices most challenging to implement in Australia in this complex service delivery context. Once high fidelity is achieved, various enhancements could be studied such as the characteristics of high performing employment specialists and how this knowledge could be used to improve staff recruitment and training. This is important, because the employment specialist role is critical to program success and staff turnover in the industry can be high, particularly when easier office based roles are available.

The impact of wage subsidies on job retention could also be studied as it is a form of support provided to employers that has become widespread in Australia. However, limited information is available about whether wage subsidies contribute to program effectiveness or not. More employer centred research is also needed to understand why some employers and not others engage with this program, and how employer interest can be sustained. Finally, further research into job retention is needed because this remains an outstanding issue for IPS as it does for all forms of vocational rehabilitation for people with SPMI.

Although the focus of this study has been on implementing evidence-based practices in Supported Employment for people with psychiatric disabilities in the Australian context, the issues identified may generalize to a wide range of contexts and systems. It is likely that similar issues also hinder the adoption of best practices identified in other disability populations, such as employment first principles (ODDS, 2008) and quality indicators for Supported Employment (Wehman, Revell, & Brooke, 2003). For instance, large caseloads can be inadvertently induced by funding systems that do not specifically encourage the intensive, client-centred and highly individualised type of service needed to be effective. Hence, this line of investigation may have useful applications to other disability populations in a range countries, settings and systems.