Abstract
PURPOSE:
To develop a fidelity score for the Stratified Vocational Advice Intervention (SVAI), and to evaluate associations between level of fidelity to the SVAI and number of sick leave days and work ability at six months follow-up in workers on sick leave due to musculoskeletal disorders (MSDs).
METHODS:
The SVAI was provided by physiotherapists (PTs) who documented delivery in individual participant logs. A fidelity score was developed based on the number of follow-up sessions held, core questions documented, and a written action plan. Data were extracted from the logs and fidelity to the SVAI was categorised as “fully delivered", “partially delivered, or “not delivered” based on predetermined criteria. The number of sick leave days was calculated from registry data, and work ability was assessed using a question from the Work Ability Index on current work ability compared with the lifetime best (0–10).
RESULTS:
148 SVAI logs were available for fidelity evaluation (87%). The SVAI was fully delivered to 87 (56%) participants and partially delivered to 61 (39%) participants. There was no statistically significant association between level of fidelity and sick leave days (B = 5.8, 95% confidence interval (CI) –6.2, 17.7, p = 0.34) or work ability (B = –0.2, 95% CI –1.4, 1.1, p = 0.77).
CONCLUSION:
A three-category fidelity score was developed and the SVAI was generally delivered according to protocol. There were no statistically significant associations between level of fidelity and number of sick leave days or work ability in participants at six months follow-up.
Introduction
In Norway, musculoskeletal disorders (MSDs) are the leading cause of sick leave and disability benefits [1]. Return to work (RTW) in workers on sick leave due to MSDs depends upon multiple factors such as individual and workplace factors, healthcare services, and compensation systems [2–4]. The Norwegian Labour and Welfare Administration (NAV) aims to help people on sick leave to participate in work and social life by different economical compensations and other stimulating interventions. In the Motivational Interviewing –NAV (MI-NAV) randomised trial [5], we evaluated the effectiveness of adding two vocational interventions to usual case management on RTW compared to usual case management alone. One of the interventions, a Stratified Vocational Advice Intervention (SVAI), was developed with the aim of facilitating RTW among workers on sick leave due to MSDs. The SVAI was offered to the participants through phone calls by trained physiotherapists (PTs) with the aim to assess and resolve modifiable health and work-related obstacles to return to work. The SVAI is a complex intervention and can be challenging to implement [6]. The United Kingdom (UK) Medical Research Council (MRC) guidance on developing and evaluating complex interventions [7] recommends performing process evaluations as a complement to outcomes evaluations to “better understand the functioning of an intervention, by examining factors such as implementation, mechanisms of impact, and contextual factors”. This study is part of a process evaluation of the SVAI [8] in the MI-NAV trial in which the PTs’ level of fidelity to the SVAI was assessed.
Fidelity, commonly referred to as adherence or treatment integrity, can be defined as the extent to which the intervention was delivered according to protocol [6]. Without assessment of fidelity, it is not possible to document whether a lack of effect is due to poor implementation or inadequacies in the intervention [9]. Numerous studies have demonstrated higher employment outcomes in programs that were implemented with better fidelity [10–14]. However, the studies have mostly been conducted on people with mental illness, and there is a lack of studies on the associations between levels of fidelity to vocational interventions and work participation outcomes in workers on sick leave due to MSDs. Furthermore, tools to assess fidelity to vocational interventions such as the Supported Employment Fidelity Scale have been developed [11], but there exists no tool for assessment of fidelity to the SVAI.
The purpose of this study was to develop a fidelity score, to assess level of fidelity to the SVAI in the MI-NAV study, and to assess associations between level of fidelity to the SVAI and number of sick leave days and self-reported work ability at six months follow-up. The following specific objectives will be addressed: To assess the extent to which the SVAI was delivered fully, partially, or not at all to workers on sick leave due to musculoskeletal disorders. To examine the association between level of fidelity to the SVAI and number of sick leave days in workers on sick leave due to musculoskeletal disorders. To examine the association between level of fidelity to the SVAI and self-reported work ability in workers on sick leave due to musculoskeletal disorders.
Methods
The MI-NAV study
This study is part of a process evaluation of the SVAI intervention delivered in the MI-NAV trial. The MI-NAV trial is a multi-arm randomised controlled trial (Clinical trials identifier: NCT03871712) conducted as a collaboration between Oslo Metropolitan University and the NAV in Norway [5]. Employed workers living in two counties South-West of Oslo aged 18 to 67 years on 50 to 100% sick leave for≥7 weeks due to MSDs were recruited and randomised to a usual case management group or to one of two intervention groups. In addition to usual case management offered to people on sick leave in Norway, participants received either motivational interviewing (MI) delivered by NAV caseworkers or the SVAI delivered by PTs. This study evaluates the fidelity of the SVAI. The fidelity of the MI intervention has been reported elsewhere [15]. The MI-NAV study has been evaluated by the Regional Committee for Medical and Health Research Ethics to not need approval (2018/1326/REK sør-øst A) and by the Norwegian Centre for Research Data (ref 861249) to fulfil the data management requirements. The MI-NAV study is described in detail in the study protocol [5].
Usual case management in NAV
In Norway, workers on sick leave are entitled to full salary compensation for up to one year. The first 16 days of sick leave are covered by the employer, after which the salary compensation is covered by the national insurance scheme through NAV. The employer should prepare a follow-up plan in cooperation with the worker within four weeks of sick leave and a dialogue meeting within seven weeks. Within 26 weeks a caseworker at the local NAV office should summon the worker and the employer to a second dialogue meeting. The meeting is mandatory for the worker and the employer. If there is a need for further dialogue towards the end of the sick leave period, all the parties may request that NAV convenes a third dialogue meeting. NAV caseworkers can offer advice and suggest various interventions to facilitate RTW during the sick leave period.
The stratified vocational advice intervention (SVAI)
The SVAI is described in detail in our process evaluation [8], however a brief description is provided here. The SVAI was adapted to fit a Norwegian context from the vocational advice intervention developed for the UK Study of Work and Pain (the SWAP trial) [16]. Rather than the stepped care model used in the SWAP trial, the SVAI was adapted for the purposes of the MI-NAV study with a stratified care model. Prior to randomisation, the participants’ baseline scores on the Keele STarT MSK tool [17] and the Örebro MSK Pain Screening Questionnaire Short Form (ÖMPSQ-SF) [18] were used to stratify the participants into two groups according to their risk for long-term sick leave; i.e. a low/medium-risk group, or a high-risk group.
The training of the PTs consisted of a three + two-day course led by one of the authors who was involved in the SWAP trial [8]. The course included presentations, discussions, and role plays. The PTs received a detailed manual on how to deliver the SVAI. Topics covered during the course included follow-up of workers on sick leave in Norway, the relationship between work and health, and communication skills. The course also trained the PTs in identifying and addressing obstacles to RTW through case management, problem solving, goal setting, action planning, and provision of information and advice.
The SVAI was delivered using a semi-structured conversation guide that included 15 core questions to assess the participant’s work and health situation. According to the protocol the participants in the low/medium-risk group should receive one or two SVAI conversation sessions by phone lasting up to one hour. Participants in the high-risk group should receive three or four conversation sessions lasting up to one hour; the first session by phone and the second, third, and potentially fourth by phone, as a face-to-face meeting, or as a workplace visit. Decisions on where to hold the second to fourth sessions were made by the PT in cooperation with the participant. During the sessions, topics such as assessment and resolution of modifiable health and work-related obstacles to RTW were emphasised. According to the protocol the PTs should document the content of the conversations during the sessions in a log, one log for each participant. The log contained information on the number of sessions held, the participant’s health and work situation, the PT’s notes on the participant’s responses to the 15 core questions (Table 1), as well as an action plan and an activity log. In the log the PT also listed all communication with relevant stakeholders. The follow-up should have ended if the participant had returned to work in their contracted working hours for four consecutive weeks, and at the latest 26 weeks after the start of the participant’s sick leave period.
Core questions in the log
Core questions in the log
NAV, Norwegian Labour and Welfare Administration.
We developed a structured Microsoft Excel sheet to extract information from the logs. The sheet was piloted by two of the authors on four logs, one from each of the PTs delivering the SVAI. Discrepancies in the extracted data were discussed, and some minor changes were made. The piloted Excel data sheet was also validated against audio recordings of the first SVAI sessions documented in each of the four logs. To determine level of fidelity the following information was extracted from the logs; (i) whether the correct number of sessions was delivered; (ii) how many of the core questions 3 to 6 and 8 to 15 in the log were completed; and (iii) whether a written action plan was devised, followed up and documented. Each criterion was assessed as follows: The correct number of sessions was rated as delivered according to the protocol if the PT delivered one or two sessions to low/moderate-risk participants and three or four sessions to high-risk participants. The core questions were deemed fully covered if the PT entered the participant’s responses to 12 core questions, i.e., questions 3 to 6 and 8 to 15, in the SVAI log; partially covered if responses to 8 to 11 questions were documented; and not covered if responses to 7 or fewer questions were documented. If responses to 7 or fewer questions were documented, then at least one question was missing from at least two topics (Table 1). Core questions 1 and 2 were excluded from fidelity assessment because knowledge of the first date of the participant’s current sick leave and the percentage of sick leave at the first consultation were expected to not influence intervention effectiveness. Core question 7, regarding follow-up from NAV, was excluded from the fidelity assessment as NAV caseworkers are not required to contact workers before they have reached six months of sick leave, which is the time the SVAI should have been completed, according to the protocol. We judged the action plan to have been completed if the log contained written information on the following three components: (i) one or more obstacles to RTW; (ii) one or more planned actions to facilitate RTW; and (iii) the participant’s progress on the planned actions (if applicable). The plan was judged as partially completed if the PT entered written information on one or more obstacles to RTW and one or more planned actions to promote RTW, but did not follow up on the participant’s progress, provided that two or more sessions were held. Otherwise, the action plan was judged as not completed.
A final sum score ranging from 0 to 6 points was given for each log based on criteria described in Table 2. Four logs were erased by one of the PTs after the SVAI was delivered but before the logs were made available for data extraction. For these four logs no fidelity judgement could be made, and they were not included in the statistical analyses. Participants who did not receive the SVAI because they did not answer attempts at establishing contact or were not contacted by PTs received the ‘not delivered’ fidelity score as they should have received follow-up according to the protocol. Participants who did not receive the SVAI because they returned to work≥50% prior to the first follow-up session or were on sick leave longer than 26 weeks at baseline were excluded (because they should not have received the SVAI according to the protocol).
Assessment criteria for fidelity to the SVAI
Assessment criteria for fidelity to the SVAI
Sum score: 6 points = fully delivered; 3–5 points = partially delivered;<3 points = not delivered.
At baseline demographic variables were collected including age, sex, BMI (weight (kg)/height (m)2), smoking status, higher education (at college or university level), marital status, first language, type of job (white or blue collar), degree of sick leave, and employment percentage in the work contract. Health literacy was assessed with the Health Literacy Survey Norwegian Questionnaire 12 (HLS-N-Q12) [19]. Work satisfaction was assessed by asking the participant “if you take into consideration your work routines, management, salary, promotion possibilities and work mates, how satisfied are you with your job?”, with a numerical scale from 0 to 10, where 0 refers to “not satisfied at all” and 10 refers to “completely satisfied”. The number of days on sick leave the year prior to inclusion in the study was received from NAV registry data.
Number of sick leave days
The number of sick leave days was collected using NAV registry data and was measured from baseline to six months follow-up, adjusted for degree of sick leave and for employment percentage at baseline.
Work ability
Work ability was assessed with one item from the Finnish Work Ability Index (WAI) [20]. At six-month follow-up, the participant was asked to register “current work ability compared with the lifetime best” on a 0–10 numerical rating scale, where 0 refers to “you are completely unable to work” and 10 refers to “your work ability is at its best” [20].
Data analysis
Data on baseline characteristics of the participants and the PTs’ background variables were analysed with descriptive statistics including frequencies, percentages, means, and median values using Stata version 16. Associations between level of fidelity to the SVAI and number of sick leave days and self-reported work ability at six-month follow-up were analysed using multiple linear regressions. We considered our study exploratory so no correction for multiple testing was performed. The alpha level was set to 5%, and 95% confidence intervals are reported on point estimates. Crude estimates and estimates adjusted for participants’ risk group and degree of sick leave at baseline, are presented. Sensitivity analyses excluding 18 participants who were erroneously stratified to the wrong risk group were conducted.
Results
Characteristics of PTs and study participants
Eight PTs completed the SVAI training course. Four PTs withdrew early in the intervention period due to other work commitments. The remaining four PTs delivered the SVAI to 95% of the participants and completed all SVAI logs included in the regression analyses. The PTs were women aged 28 to 45 years with four to 21 years of work experience as physiotherapists in primary care. Two PTs had completed relevant master’s degrees and three PTs had completed relevant continuing education. Each PT delivered SVAI to 30 to 40 participants.
Out of 170 participants randomised to SVAI, 152 (89%) received the SVAI. Among the 18 participants who did not receive the SVAI five did not answer the PT’s attempts at establishing contact, two were not contacted by the PT and one withdrew from the study. A total of 148 logs were available for analysis. Seven participants who according to the protocol should have received the SVAI did not and were assigned “not delivered” fidelity ratings, resulting in a total of 155 fidelity ratings. Out of the 155 participants who received fidelity ratings, 120 were in the low/moderate-risk group and 35 were in the high-risk group (Table 3). The mean age was 48 (SD 9.9) years and 59% were women, with a median of 36 (min-max 4–181) days of sick leave during the 12 months prior to study inclusion, and a median self-reported baseline work ability of 3 (0–10) on a scale from 0 to 10.
Baseline characteristics of participants
Baseline characteristics of participants
aBody Mass Index: kg/m2. bAssessed with HLS-Q12 (12–72). cNumber of days on sick leave during the year prior to inclusion in study. dSingle question (0–10). eCollege or university. fEmployment % in work contract.
Table 4 shows the extent to which the three intervention elements were delivered. As for the number of follow-up sessions held, too many sessions were held in six (4%) and too few in 18 (12%) cases. The number of cases for each fidelity sum score were: 0 points: n = 7 (5%), 3 points: n = 2 (1%), 4 points: n = 12 (8%), 5 points: n = 47 (30%), and 6 points: n = 87 (56%). According to the fidelity cut-off values SVAI was not delivered in 7 (5%) cases, partially delivered in 61 (39%) cases, and fully delivered in 87 (56%) cases.
The number (%) of SVAI intervention elements delivered (n = 155)
The number (%) of SVAI intervention elements delivered (n = 155)
At six months follow-up the median (min-max) number of sick leave days was 75 (32–131) in the “not delivered” group, 43 (4–131) in the “partially delivered” group, and 54 (2–131) in the “fully delivered” group. The median (min-max) work ability was 2.5 (0–5) in the “not delivered” group, 6 (0–10) in the “partially delivered” group, and 6.5 (0–10) in the “fully delivered” group. When adjusting for participants’ risk group and degree of sick leave at baseline, there were no statistically significant associations between level of fidelity (fully versus partially delivered) and number of sick leave days or self-reported work ability (Table 5).
Associations between fidelity to the SVAI and number of sick leave days and work ability at six months follow-up
Associations between fidelity to the SVAI and number of sick leave days and work ability at six months follow-up
Adjusted estimates are adjusted for participants’ risk group and degree of sick leave at baseline. B: Regression coefficient. CI: Confidence interval. p: p value. part. del.: Partially delivered.
Among the 18 participants who were stratified to the wrong risk group at baseline, one did not receive the SVAI and was not included in the regression analyses. The 17 participants included in the regression analyses had the SVAI partially (n = 13) and fully delivered (n = 4) and had a median of 66 (4–131) sick leave days compared to a median of 48 (2–131) sick leave days in the participants who were stratified to the correct risk group. Compared to correctly stratified participants, a higher proportion of erroneously stratified participants had high risk for long-term sick leave (65% vs. 21%) and blue-collar job (82% vs. 63%), and a lower proportion had a higher educational level (18% vs. 40%). The sensitivity analysis excluding participants who were erroneously stratified made no difference to the associations between level of fidelity to the SVAI and self-reported work ability. However, when excluding the erroneously stratified participants, those who had the SVAI fully delivered had, on average, 13 sick leave days more than participants who had the SVAI partially delivered (B = 13.1, 95% CI 0.87, 25.3, p = 0.04).
In this study we developed a three-point ordinal fidelity score, which distinguished between a fully, partially, and not delivered SVAI in the MI-NAV trial [5]. The SVAI was fully delivered in 87 (56%) cases, partially delivered in 61 (39%), and not delivered in seven cases (5%). There were no statistically significant associations between level of fidelity to the SVAI and number of sick leave days or self-reported work ability at six months follow-up. A sensitivity analysis including only the correctly stratified participants showed that participants who had the SVAI fully delivered had 13 sick leave days more than participants who had the SVAI partially delivered.
These results indicate that the three + two-day training course along with the SVAI manual and monthly mentoring sessions generally prepared the PTs to adequately deliver the SVAI in accordance with the protocol. When validated against audio recordings from four intervention sessions, it became clear that the logs were conscientiously completed by the PTs. Additionally, the SVAI process evaluation, in which a thematic analysis of 18 SVAI session recordings was conducted, showed that the core intervention elements were delivered [8]. However, some of the topics covered during the SVAI sessions were not documented in the logs. This may be because the PTs did not have time to document all the conversational content, and because the PTs, initially, were not informed that the logs were meant for fidelity assessment.
As for the number of follow-up sessions held, too many sessions were held in six (4%) and too few in 18 (12%) of cases. The main reason that too many or too few sessions were held was that 18 participants were erroneously stratified to the wrong risk group at baseline. The sensitivity analysis excluding the erroneously stratified participants showed an increased difference in the number of sick leave days between the partially delivered and fully delivered group (difference of 13 sick leave days) compared to the main analysis (difference of 6 sick leave days). This finding is due to the erroneously stratified participants having a median of 66 sick leave days compared to a median of 48 sick leave days among the participants who were correctly stratified and included in the sensitivity analysis. Given that 13 out of the erroneously stratified participants received the SVAI partially, the exclusion of these participants reduced the median number of sick leave days in the partially delivered group. The higher number of sick leave days in the erroneously stratified participants compared to correctly stratified participants may be due to a higher proportion of participants with high risk for long-term sick leave (65% vs. 21 % among correctly stratified participants), blue-collar jobs (82% vs. 63% among correctly stratified participants), and a lower proportion of participants with higher education (18% vs. 40% among correctly stratified participants). The ‘not delivered’ fidelity group was excluded from regression analyses due to too few participants in the group (n = 7). Further, three participants did not receive the correct number of follow-up sessions due to having returned to their contracted working hours for a period of four consecutive weeks before the end of their follow-up. One of these participants was among the seven who did not receive the SVAI and was excluded from all regression analyses. The two remaining participants did receive the SVAI but were among the erroneously stratified and were thus included in the main analyses but excluded from sensitivity analyses. However, data from these two were unlikely to have impacted the associations between level of fidelity to the SVAI and number of sick leave days and work ability in the main analyses.
Other studies evaluating the associations between levels of fidelity to vocational interventions and work participation outcomes have demonstrated higher employment outcomes in programs that were implemented with better fidelity [10–14]. Lockett et al.s’ systematic review found that Individual Placement and Support (IPS) programs implemented with higher fidelity were associated with higher employment rates [13]. A prospective cohort study by Yamaguchi et al. reported that supported employment programs delivered with high fidelity were superior to low-fidelity programs in terms of vocational outcomes including higher competitive job acquisition [14]. However, these studies were conducted with people having moderate to severe mental illness. Additionally, IPS is more comprehensive than the SVAI in that IPS focuses on securing paid work and then supporting the worker to become competent in their role while working [21]. Studies evaluating the associations between physiotherapists’ level of fidelity to vocational interventions and work participation outcomes in study participants who were on sick leave due to MSDs are to us unknown, potentially indicating that not much research has been conducted on this topic.
Strengths and limitations
The main strength of this study was that we had access to logs from 148 out of the 152 participants who received the SVAI. Another strength is that the number of sick leave days was collected from registry data from the NAV, and that information on participants’ work ability was collected using the work ability score from the validated WAI [20]. Additionally, this study was performed prior to the outcomes evaluation from the RCT and has therefore not been influenced by the results from the main trial.
A limitation in this study is the lack of a validated fidelity score. Furthermore, we do not know whether some of the intervention components have a higher impact on the effectiveness of the SVAI than others. Additionally, it might be a limitation that the present fidelity score was based solely on the logs from the physiotherapists, and do not reflect the quality of the conversations held during the follow-up sessions [8]. However, a previous process evaluation of the SVAI, in which qualitative analyses of 18 audio recordings of the SVAI sessions were conducted, showed that the PTs delivered the intervention elements in accordance with the SVAI logic model [8].
Conclusions
A fidelity score was developed to distinguish between a fully delivered, partially delivered, and a not delivered SVAI intervention. The results of this fidelity evaluation indicate that the SVAI was delivered according to protocol and that the participants received most of the intervention elements. There were no statistically significant associations between fidelity to the SVAI and number of sick leave days or work ability in workers on sick leave due to MSDs at six months follow-up. A sensitivity analysis including correctly stratified participants showed that participants who had the SVAI fully delivered had a statistically significant 13 sick leave days more compared to participants who had the SVAI partially delivered.
Author contributions
MG was responsible for acquisition of funding for the study. MG, GS, GWJ, BEØ and FA were involved in the development of the SVAI. GS, FA, and MG were responsible for the training and mentoring of the physiotherapists providing the SVAI. FA, BEØ, MG, GS and GWJ contributed to the planning of the process evaluation of the SVAI. RS transcribed the audio recordings, extracted data from the SVAI logs, and drafted the first version of the manuscript. All authors critically revised and commented on the previous versions of the manuscript and read and approved the final manuscript.
Footnotes
Acknowledgments
The authors would like to thank the participants, physiotherapists, the NAV, and the MI-NAV project group for contributing to the study.
Conflict of interest
The authors have no conflict of interest to report.
Ethical considerations
All participants provided written informed consent prior to inclusion in the study.
Funding
This work was funded by The Norwegian Fund for Post-Graduate Training in Physiotherapy; The Research Council of Norway; The Norwegian Labour and Welfare Administration (NAV); and Oslo Metropolitan University.