The utility of item level fidelity scores for developing evidence based practices in supported employment

Abstract

BACKGROUND:

Fidelity scales are an important way to control the quality of implementation of psychosocial interventions such as supported employment programs. However, little is known about the utility of item level scores for informing the implementation of supported employment programs.

OBJECTIVE:

To explore the utility of item level scores from the IPS-25 fidelity scale for developing evidence based practices in supported employment for people with severe mental illnesses in Sussex, United Kingdom (UK).

METHODS:

Fidelity was assessed in 18 sites representing 34 independent cohorts where IPS was implemented by one organisation. Item-level fidelity scores were examined descriptively. Nine item level correlations were selected a-priori and examined with respect to two employment milestones and program attrition.

RESULTS:

Mean total fidelity score was associated with commencing competitive employment (r = 0.36, p = 0.04). None of the three items selected as likely to be related to either commencing employment or to employment duration, were correlated as expected. Of the three items selected as likely to influence attrition, only one ‘Rapid commencement of job searching’ was correlated as expected (r = 0.38, p = 0.02).

CONCLUSIONS:

Individual fidelity score items appear useful for improving supported employment programs, in terms of item response characteristics and expected correlations with particular program outcomes.

Keywords

Supported employment evidence-based practices fidelity item level properties severe mental illness psychiatric disability

1 Introduction

New implementations of psycho-social rehabilitation interventions need to accurately replicate the original program, then demonstrate efficacy and effectiveness in specified conditions and circumstances. Fidelity refers to the extent of adherence to the original program. Measuring fidelity can improve the translation of evidence-based practices into real world settings (Mowbray, Holter, Teague, & Bybee, 2003). The individual placement and support (IPS) approach to supported employment is a specialised form of vocational rehabilitation for people with severe mental illnesses (Marshall et al., 2014). Two fidelity scales are available (Bond, Becker, Drake, & Vogler, 1997; Bond, Peterson, Becker, & Drake, 2012). The most recent consists of 25 items (IPS-25) and is supported by an instruction manual on how best to assess the program, administer, and code the scale (Swanson & Becker, 2013).

Each question in the IPS-25 uses the same five-point (1-5) rating scale with anchor descriptions for each rating. A score of five represents good implementation of that practice, and a score of one indicates poor implementation or a failure to implement. The total score range on the IPS-25 is 25–125. Several favourable properties of the IPS-25 have been reported including predictive validity (Bond, et al., 2012; Kim, Bond, Becker, Swanson, & Langfitt-Reese, 2015). The predictive validity of individual scale items has also been explored in the USA (Becker, Xie, McHugo, Halliday, & Martinez, 2006), and more recently through an analysis of 79 Johnson and Johnson learning collaborative implementation sites (Bond, et al., 2012). The authors found a modest positive relationship between total fidelity score (r = 0.24) and the quarterly employed proportions of active caseloads, following adjustment for background unemployment, and program longevity. Eight fidelity scale items were associated with quarterly employment status (Bond et al., 2012). Item level information has the potential to inform program development in ways that enable core practices to be prioritised and targeted separately. This is not possible at present due to the lack of published reports of item level properties outside the USA (Lockett, Waghorn, Kydd, & Chant, 2016).

This study explored item level fidelity scores in a large multi-site implementation of IPS in Sussex (van Veggel, Waghorn, & Dias, 2015). Three groups of three pre-selected fidelity items were examined with respect to three aspects of program performance, namely commencing employment; duration of employment, and program attrition.

2 Methods

2.1 Ethics approval and funding

Ethics and governance approval to conduct and publish a formal evaluation of this program was provided by the Governance Support Team of the Sussex Partnership Foundation Trust of the National Health Service, United Kingdom (UK). The IPS implementations were funded by the Brighton and Hove City Clinical Commissioning Group with the Brighton and Hove City Council; the Sussex Partnership Foundation Trust of the National Health Service with the East Sussex County Council and the West Sussex Clinical Commissioning Groups. The data analysis and preparation of this report was funded by the Queensland Centre for Mental Health Research.

2.2 Study design

This was a prospective observational design of a large scale IPS program implemented at 18 sites, involving 34 cohorts scored by the IPS-25 fidelity scale. Details of the sites and the employment outcomes attained were previously reported (van Veggel et al., 2015). This investigation explored all item level fidelity scores descriptively, and a limited number of individual items thought most likely to be related to each of the three outcome variables examined.

2.3 The intervention

The IPS approach to supported employment was made available in stages at 18 mixed urban and rural sites covering the catchments of the city of Brighton and Hove, the county of East Sussex, and the county of West Sussex (van Veggel et al., 2015). Each site received, on average, one hour per week of external technical support in terms of training, evaluation, and external fidelity assessments and feedback.

2.4 Participant recruitment

Volunteer participants were identified from among those who expressed a vocational goal to a mental health team member. Those who expressed a vocational goal were referred directly to the employment specialist at each site by members of the community mental health teams into which the employment specialists were co-located. Self referrals were also accepted. Records of referrals to all sites were kept centrally by author RV of Southdown Housing and Supported Employment. Mental health team members were discouraged from filtering clients, or from making their own judgments about whether vocational interventions were suitable for the person at that time. Vocational goals included: (1) obtaining new competitive employment; (2) obtaining voluntary work, or obtaining unpaid work experience; (3) commencing study in a formal education or vocational training program; or (4) retaining their current employment, if the mental health team member considered their job was at risk of termination.

2.5 Measuring IPS fidelity

The extent of adherence to IPS principles and practices was assessed both externally and internally for each cohort at each location. External fidelity reviews were conducted annually by trained external fidelity reviewers following the recommended method for fidelity assessment using the standardised 25-item IPS fidelity scale (Bond, et al., 2012). One of these external reviewers was an Occupational Therapist who had received formal training in conducting fidelity reviews by the program developers in the USA. Each fidelity review involved gathering information from multiple sources over one to two days. All locations had three external fidelity reviews recorded. Internal fidelity reviews were conducted by site managers, either directly or by delegation to selected staff. Internal reviews were not conducted as recommended in the IPS training manuals by a person external to the workgroup implementing the practices. Instead, internal reviews were conducted by staff nominated by the local site manager.

2.6 Managing data quality

Data quality was managed by author RV in her role as Performance Coordinator at Southdown Housing Association, the employment organisation being evaluated. Author RV manually checked and entered data into one aggregated MS Excel file for all 18 locations on all common variables. Data from each location were checked and cleaned prospectively. This involved examining internal consistency, missing information, out of range values, and accuracy from two sources (the mental health service and the employment service) on a monthly basis, as information was received from each location. Maintaining an accurate data file was also a contract requirement from the funder of the supported employment program.

Details of employment outcomes were sought from both employment services and from participants for verification. In addition, employment specialists were required to update program information on a weekly basis. The updated data were then transferred into a MS Excel file at the end of each month. This file was locked and protected by author RV. Biannual audits of other files held by employment specialists were also conducted to ensure accuracy and compliance with the three commissioning contracts.

2.7 Analysis methods

Analyses were conducted using R statistical software (R Core team, 2016). Individual fidelity item scores and the total fidelity score were correlated with the proportion commencing competitive employment, comparing two types of fidelity assessment. These were operationalized by the logarithm of the odds ratio among all respondents at a given cohort (called logOR), weighted by the meta-regression approach of DerSimonian and Laird (1986). Weighting is necessary because correlations involving means and other summary estimates such as proportions do not reflect the sample size used for each estimate. To adjust for this, weighting by the inverse of the standard errors of estimate was applied in this investigation. A limited number of correlations were calculated with respect to three dependent variables: (1) the proportions (expressed as logOR) commencing competitive employment; (2) the duration of employment in the first job measured in days of continuous employment; and (3) attrition defined as the proportion ceasing participation within the first six months provided no vocational benefit was gained.

3 Results

3.1 Total fidelity scores

The externally assessed fidelity score mean for cohort one (pre-IPS) was 99.7/125, while for cohort two, following IPS implementation, this increased to 105.7/125. The mean total fidelity scores are shown in Table 1. Total fidelity scores and mean fidelity scores assessed externally were more conservative by around one standard deviation (5.4 points). This represents a significant inflation for internally assessed total fidelity scores overall at the 98% confidence level (see Table 1). Mean fidelity scores after combining both assessment methods, were significantly associated with commencing employment (r = 0.36, p = 0.04).

Table 1
Item level fidelity scores by method of fidelity assessment

Fidelity item External Internal Difference Overall

mean sd mean sd t pval.t mean sd

F1. Caseload size 4.77 0.69 4.58 0.51 0.91 0.37 4.71 0.63

F2. Employment services staff roles 4.36 1.00 4.50 0.52 – 0.52 0.61 4.41 0.86

F3. Vocational generalists 4.86 0.47 4.58 0.67 1.29 0.21 4.76 0.55

F4. Integration through Mental Health Team assignment 4.68 0.57 4.33 0.89 1.23 0.24 4.56 0.70

F5. Integration through frequent contacts with team members 4.32 0.84 4.42 0.51 – 0.42 0.67 4.35 0.73

F6. Collaboration with related programs 3.45 0.60 4.17 1.11 – 2.06 0.06 3.71 0.87

F7. Employment staff form a vocational unit 4.05 0.84 4.00 – – – 4.03 0.67

F8. Role of employment supervisor 4.09 0.53 3.00 – – – 3.71 0.68

F9. Zero exclusion criteria 4.95 0.21 4.08 0.90 3.30 0.007 4.65 0.69

F10. MHS focus on competitive employment 3.77 0.87 4.75 0.45 – 4.31 <0.001 4.12 0.88

F11. Executive team support for the program 3.18 0.50 5.00 – – – 3.82 0.97

F12. Work incentives planning 4.77 0.61 5.00 – – – 4.85 0.50

F13. Disclosure strategies 5.00 – 5.00 – – – 5.00 –

F14. Ongoing, work-based vocational assessment 4.73 0.46 4.92 0.29 – 1.48 0.15 4.79 0.41

F15. Rapid search for competitive job 3.95 0.49 4.17 0.39 – 1.39 0.18 4.03 0.46

F16. Individualized job search 4.18 0.73 4.50 0.67 – 1.27 0.21 4.29 0.72

F17. Job development-Frequency of employer contacts 2.23 0.69 1.75 0.45 2.44 0.02 2.06 0.65

F18. Job development-Quality of employer contacts 3.27 1.20 2.50 1.31 1.69 0.11 3.00 1.28

F19. Diversity of job types 4.09 1.34 5.00 – – – 4.41 1.16

F20. Diversity of employers 4.64 0.58 5.00 – – – 4.76 0.50

F21. Proportion of competitive jobs obtained 3.23 1.74 5.00 – – – 3.85 1.64

F22. Individualized follow-on supports 3.91 1.38 4.58 0.51 – 2.05 0.05 4.15 1.18

F23. Time unlimited follow-on supports 3.68 0.72 3.00 0.85 2.35 0.03 3.44 0.82

F24. Community-based services 3.55 1.01 4.00 1.13 – 1.16 0.26 3.71 1.06

F25. Assertive engagement and outreach 2.55 1.01 3.83 1.03 – 3.51 0.002 3.00 1.18

Total Fidelity Score 100.3 5.54 105.7 6.29 – 2.49 0.02 102.18 6.29

Mean Fidelity Score 4.01 0.22 4.23 0.25 – 2.49 0.02 4.09 0.25

Fidelity item	External	Internal	Difference	Overall
F1. Caseload size	4.77	0.69	4.58	0.51	0.91	0.37	4.71	0.63
F2. Employment services staff roles	4.36	1.00	4.50	0.52	– 0.52	0.61	4.41	0.86
F3. Vocational generalists	4.86	0.47	4.58	0.67	1.29	0.21	4.76	0.55
F4. Integration through Mental Health Team assignment	4.68	0.57	4.33	0.89	1.23	0.24	4.56	0.70
F5. Integration through frequent contacts with team members	4.32	0.84	4.42	0.51	– 0.42	0.67	4.35	0.73
F6. Collaboration with related programs	3.45	0.60	4.17	1.11	– 2.06	0.06	3.71	0.87
F7. Employment staff form a vocational unit	4.05	0.84	4.00	–	–	–	4.03	0.67
F8. Role of employment supervisor	4.09	0.53	3.00	–	–	–	3.71	0.68
F9. Zero exclusion criteria	4.95	0.21	4.08	0.90	3.30	0.007	4.65	0.69
F10. MHS focus on competitive employment	3.77	0.87	4.75	0.45	– 4.31	<0.001	4.12	0.88
F11. Executive team support for the program	3.18	0.50	5.00	–	–	–	3.82	0.97
F12. Work incentives planning	4.77	0.61	5.00	–	–	–	4.85	0.50
F13. Disclosure strategies	5.00	–	5.00	–	–	–	5.00	–
F14. Ongoing, work-based vocational assessment	4.73	0.46	4.92	0.29	– 1.48	0.15	4.79	0.41
F15. Rapid search for competitive job	3.95	0.49	4.17	0.39	– 1.39	0.18	4.03	0.46
F16. Individualized job search	4.18	0.73	4.50	0.67	– 1.27	0.21	4.29	0.72
F17. Job development-Frequency of employer contacts	2.23	0.69	1.75	0.45	2.44	0.02	2.06	0.65
F18. Job development-Quality of employer contacts	3.27	1.20	2.50	1.31	1.69	0.11	3.00	1.28
F19. Diversity of job types	4.09	1.34	5.00	–	–	–	4.41	1.16
F20. Diversity of employers	4.64	0.58	5.00	–	–	–	4.76	0.50
F21. Proportion of competitive jobs obtained	3.23	1.74	5.00	–	–	–	3.85	1.64
F22. Individualized follow-on supports	3.91	1.38	4.58	0.51	– 2.05	0.05	4.15	1.18
F23. Time unlimited follow-on supports	3.68	0.72	3.00	0.85	2.35	0.03	3.44	0.82
F24. Community-based services	3.55	1.01	4.00	1.13	– 1.16	0.26	3.71	1.06
F25. Assertive engagement and outreach	2.55	1.01	3.83	1.03	– 3.51	0.002	3.00	1.18
Total Fidelity Score	100.3	5.54	105.7	6.29	– 2.49	0.02	102.18	6.29
Mean Fidelity Score	4.01	0.22	4.23	0.25	– 2.49	0.02	4.09	0.25

3.2 Item-level fidelity scores and correlations

Items with no scoring variation (SD = 0) could not be analysed further. To minimise the total number of correlations required, external and internal fidelity assessments were pooled. None of the three items preselected as most likely to be correlated with commencing employment (items 1, 17 & 18) were statistically significant. Item 1 (Caseload size) was promising (r = 0.29, p = 0.10) but was not significant. Of the three items considered most likely to be associated with duration of employment (Items 14, 22 &24), Item 22 (Individualised follow-on supports) showed promise (r = 0.29, p = 0.09) but again was not significant. Of the three items considered most likely to be associated with attrition (items 15, 23 & 25), only item 15 (Rapid job search) was associated (r = 0.38, p = 0.02). Mean scores for the full scale were not associated with either employment duration or attrition.

4 Discussion

4.1 Summary

Mean total fidelity score on the IPS-25 fidelity scale predicted new employment commencements in the Sussex trial, accounting for 13% (r = 0.36, p = 0.04) of the variance in commencing employment. This is a stronger relationship than previously reported in the USA (r = 0.24, Bond, et al., 2012). In addition to improving overall fidelity, an item level analysis revealed how targeting individual item scores can further improve program implementation.

4.2 Internal versus external fidelity assessments

Internal fidelity assessments had a restricted scoring range and were inflated by 5.4 points overall compared to external assessments. Five of 25 items (items 9, 10, 17, 23, 25) were scored significantly differently by external compared to internal assessors. External scoring was usually but not always more conservative (see Table 1). Site managers responsible for internal assessments may have felt pressure to attain good fidelity, and if so, may have assessed prescribed practices rather than observing practices-in-use at each site.

Not all internally assessed fidelity items with a restricted range had higher mean scores than externally assessed items. For instance, items 7 and 8 suggest a negative response bias, suggesting that more careful assessment based on direct observation could lead to higher scores. Eight internally assessed items were scored identically across all sites, compared to one externally assessed item (Item 13 Disclosure).

4.3 Correlations with employment and attrition

As expected, the practice of rapid commencement of job searching was negatively associated with early attrition. However, other practices not covered in the fidelity scale may also help reduce early attrition. For example: ceasing the use of waitlists, decreasing the time for a person referred to the program to be accepted; and increasing the proportion of referrals correctly accepted (currently 59.4% ); could also help to reduce early attrition by increasing early engagement with the program.

4.4 Comparison with previous studies

Previous studies of IPS implementations found overall fidelity score to be positively associated (r = 0.36) with employment outcomes (Bond, et al., 2012; Kim et al., 2015). A recent systematic review and meta-analysis found a stronger correlation of r = 0.41 for the IPS-15 scale (Lockett et al., 2016). Differences between internally and externally assessed fidelity scores, although not reported previously, support the current recommendation (Swanson & Becker, 2013) that fidelity assessments are conducted externally.

5 Conclusions

A positive relationship was found between mean fidelity score and commencing employment when methods of fidelity assessment were combined. This is consistent in terms of strength and direction with correlations for total fidelity score found in two previous USA reports. The distributions of fidelity scores at an item level are also informative, particularly when fidelity is assessed as recommended, by trained external assessors. When fidelity items appear optimally scored across all sites with no variation in scoring, the focus can shift to checking the method, quality, and consistency of the fidelity assessment to ensure that actual behaviours are observed at a practitioner level, rather than being inferred from prescribed policies and practices.

Conflict of interest

None to report.

References

Becker

D. R.

, Xie

, McHugo

G. J.

, Halliday

, Martinez

R. A.

(2006). What predicts supported employment program outcomes?Community Mental Health Journal, 42(3), 303–313.

Bond

G. R.

, Becker

D. R.

, Drake

R. E.

, Vogler

K. M.

(1997). A fidelity scale for the Individual Placement and Support model of supported employment, Rehabilitation Counseling Bulletin, 40(4), 265–284.

Bond

G. R.

, Peterson

A. E.

, Becker

D. R.

, Drake

R. E.

(2012). Validation of the revised Individual Placement and Support Fidelity Scale (IPS-25). Psychiatric Services, 63(8), 758–763.

DerSimonian

, & Laird

(1986). Meta-analysis in Clinical Trials. Controlled Clinical Trial, 7, 177–188.

Kim

S. J.

, Bond

G. R.

, Becker

D. R.

, Swanson

S. J.

, & Langfitt-Reese

(2015). Predictive validity of the individual placement and support fidelity scale (IPS-25): A replication study. Journal of Vocational Rehabilitation, 43, 209–216.

Lockett

, Waghorn

, Kydd

, & Chant

(2016). Predictive validity of evidence-based practices in supported employment: A systematic review and meta-analysis. Mental Health Review, 21(4), 261–281.

Marshall

, Goldberg

R. W.

, Braude

, Dougherty

R. H.

, Daniels

A. S.

, Ghose

S. S.

,... & Delphin-Rittmon

M. E.

(2014). Supported employment: Assessing the evidence. Psychiatric Services, 65(1), 16–23.

Mowbray

C. T.

, Holter

M. C.

, Teague

G. B.

, & Bybee

(2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation. 24(3), 315–340.

R Core Team. (2016). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. >https://www.R-project.org/.

10.

Swanson

S. J.

, & Becker

D. R.

(2013). IPS Supported Employment: A practical guide. Dartmouth Psychiatric Research Center. Lebanon, NH, USA.

11.

van Veggel

, Waghorn

, & Dias

(2015). Implementing evidence-based supported employment in Sussex. British Journal of Occupational Therapy, 78(5), 286–294.