Development and psychometric evaluation of a theory-based questionnaire measuring women’s return-to-work beliefs after long-term sick leave for common mental disorders

2.1.1 Development of items for direct measures

The items used to create scales for direct measures were worded based on recommendations in the manual [29]. The items from the manual were reworded for RTW. The scales were: intention, attitude, subjective norms and perceived behavioural control. Some symptoms of CMDs are common, e.g., tiredness, and we wanted to keep the number of items as low as possible, without losing sight of the purpose of the questionnaire. Therefore, three of the four suggested items for the subjective norms scale were included, as recommended by the manual [29] (see Supplementary material Table A). This resulted in 14 items for direct measures, shown in Table 1.

The interview guide (Table 2) was developed through consulting the manual [29] and was tested on five individuals, before the elicitation study (an interview study to generate items), without any changes being made. Women met the criteria for the elicitation study if they were≥18 years, on full- or part-time sick leave for CMDs (ICD-10-SE codes: F30– F48 [36]) for≥2 months. They also had to be able to read, write and speak Swedish. Criteria for exclusion were severe mental illness, unemployment or on sick leave > 2 years. In December 2017, the Swedish Social Insurance Agency (SIA) invited 150 women on long-term sick leave for CMDs, who lived in a county in central Sweden, to participate in the elicitation study. A total of 25 women expressed interest in doing so. However, five changed their minds for various reasons, e.g., they did not feel they had the energy to participate in an interview after all. Hence, a total of 20 women were included in the elicitation study.

Interviews were conducted by the first author in January– May 2018. The women were asked about their behavioural, normative and control beliefs regarding RTW, as well as about background information, such as age, diagnosis and extent of sick leave. They listed their beliefs in writing and talked about them. A manifest qualitative content analysis [37] was conducted by the authors to determine beliefs that should become items for indirect measures. Behavioural, normative and control beliefs were analysed separately. The beliefs stated in the interviews were grouped based on their similarities and differences and named based on their content. The 75% rule recommended by Francis et al. [29] was used, i.e., choosing the most frequently mentioned groups that together constituted around 75% of the stated behavioural, normative and control beliefs, respectively. For example, there were totally 142 stated behavioural beliefs and 24 (16.9%) of those were about belonging to a social context and 13 (9.2%) were about routines. The shares of the most frequently mentioned groups were added together until the sum was about 75%. The names of these groups were retained, to become items for indirect measures. The analysis resulted in 23 items divided across the three scales for indirect measures: behavioural beliefs (n = 9), normative beliefs (n = 7) and control beliefs (n = 7). Within each scale, there were two subscales in line with the TPB: advantages and disadvantages (behavioural beliefs), supporters and non-supporters (normative beliefs) and facilitators and barriers (control beliefs). The items were worded with guidance from the manual [29]. When the manual offered multiple options for wording or number of items, the authors discussed and agreed on what was best suited for the target population and target behaviour (RTW). For each item, a corresponding item was created to assess the outcome evaluation (behavioural beliefs), the motivation to comply (normative beliefs) and the impact on the behaviour (control beliefs). For example: ‘If I return to work/stay at work, life will feel more meaningful: strongly disagree – strongly agree’ (behavioural belief, item range 1– 7) and ‘Perceiving meaningfulness in life is, for me: not important – very important’ (evaluation of outcome, item range – 3 to +3). Hence, a total of 46 items for indirect measures were included, see Table 1.

The last step was to let women test the questionnaire. In the spring of 2019, interviews were carried out with five women from the elicitation study, as recommended by Francis et al. [29]. The women were purposively chosen to achieve differences in age and diagnoses. Four women participated in face-to-face interviews and one responded by e-mail. The interviews were based on the following questions recommended by the manual [29]: Are any items ambiguous or difficult to answer? Does the questionnaire feel too repetitive? Does it feel too long? Does it feel too superficial? Are there any annoying features in the wording or formatting (including endpoints)? Furthermore, inconsistent responses to the questionnaire were looked for, as they might indicate that changes in the response endpoints were problematic (this could not be done for the woman who answered via e-mail). The authors then discussed the findings.

2.1.4 Description of the final questionnaire

Table 1 provides an overview of the scales and items. The questionnaire consisted of 60 items divided into seven scales and six subscales. The items were worded to be suitable for women on either full-time or part-time sick leave, e.g. “People who are important to me want me to return to work/stay at work”. Women who were on full-time sick leave focused on RTW and women that worked to some extent focused on staying at work when responding. Responses were rated on 7-point scales, unipolar (1– 7) or bipolar (– 3– +3). The items from the different scales were all mixed together except those for direct measures of attitude, which were grouped for theoretical reasons, i.e. the items belonged to the same main item. Background questions were included as well. The questionnaire in its original form is published in a previous study [3].

2.2.1 Participants, setting and procedure

The women met the criteria for the psychometric evaluation if they were≥18 years and on full- or part-time sick leave for CMDs for at least the preceding two months (ICD-10-SE codes: F32– F33, F35– F48 [36]). They also had to be able to understand written Swedish. Criteria for exclusion were severe mental illness, unemployment or sick leave > 2 years. The sample size needed was determined to be around 300 based on recommendations from Polit and Beck [38] regarding the required sample size for an estimated effect size of 0.35, a power of 0.8, and α= 0.05. These estimations are common within nursing research [38]. We did also take into account the number of participants needed for a factor analysis, where 300 is considered as good [39]. Based on previous research among a similar sample [40], we did expect a response rate of around 30%. The researchers prepared invitations, each with a unique code. In October 2019 and January 2020, SIA identified a total of 1,196 potential participants in two counties in central Sweden and sent them the invitations. The women could choose to complete the questionnaire on paper or electronically. Two reminders were sent out from SIA at 2-week intervals. A total of 371 questionnaires were returned to the researchers (367 on paper and four electronically), of which 89 were excluded. Reasons for exclusion were, no longer being on sick leave (e.g. have RTW full time or retired), unemployment, or an incomplete questionnaire. Hence, 282 women were included (response rate 23.5%). Subsequently, a test-retest procedure was carried out with an interval of around three weeks, in conjunction with a 1-year follow-up. Test-retest does generally not require more than 50 participants [35], why we considered it reasonable to invite the first 50 women who responded to the 1-year follow-up to the test-retest. The retest followed the same procedure as the former test. In total, 35 women participated. It is desirable that the conditions, on our case RTW-beliefs, has not changed between the test and the retest [35]. Although there are no guarantees, the conditions were considered to be approximately the same on the two test occasions. For example, it is reasonable to assume that the women still had the same supporting people around them after three weeks.

2.2.2 Preparation of the scales for data analyses

In preparing the scales for direct measures, the items with negatively worded endpoints on the right-hand side were recoded, so that high scores consistently reflected the same direction (i.e., stronger intention, more positive attitude, stronger social pressure, stronger perceived behavioural control). Then, the mean of each scale was calculated, producing overall scores. When calculating the indirect measures, each item was multiplied by its paired item, as illustrated in Fig. 2. Thus, among the 46 items for indirect measures, 23 products were created with the range – 21 to +21 (1– 7×– 3– +3), representing the perceived importance of the belief about RTW (a higher value indicated a higher perceived importance). The products were added together to create an overall score for each scale and subscale for indirect measures. Descriptive statistics for the scales such as mean and standard deviation is presented in the previous study [3].

OPEN IN VIEWER

Fig. 2

Associations between indirect and direct measures (n = 268– 271). Model fit regarding indirect and direct measures for attitude: F(2,268) = 93.86, p < 0.001, for subjective norms: F(2,265) = 41.88, p < 0.001, and for perceived behavioural control: F(2,266) = 33.52, p < 0.001. Bold numbers represent significant values (p < 0.001).

Descriptive statistics: Frequency and percent were calculated for the demographic data for the total sample and the retest sample, respectively.

Reliability: The scales for direct measures are reflective, which means that they represent the same underlying construct, i.e. that the items are assumed to be correlated with each other [35]. Internal consistency (Cronbach’s alpha) was therefore calculated for each of the scales for direct measures. This is partly presented in a previous study [3]. Desirable score for alpha is according to Francis manual [29] 0.6 or above. Internal consistency was not relevant to calculate for the scales of indirect measures because of their formative nature [35], i.e. that the items are not expected to correlate with each other. Instead, test-retest stability was calculated for these scales using intraclass correlation coefficients (ICCs) with a 2-way mixed model and absolute agreement type [41]. According to Terry and Mae [41], ICC values less than 0.50, between 0.50 and 0.75, 0.75 and 0.90, and greater than 0.90 means poor, moderate, good and excellent reliability, respectively.

Validity: To assess content validity, data from the pilot interviews (n = 5) were structured into a table based on the questions proposed by Francis et al. [29] and then discussed between the authors to agree on any changes to the questionnaire. Standard multiple regression analysis was used between the indirect and direct scales. In each of the three regression models, the direct scale was entered as the dependent variable and the two subscales within each indirect scale were entered as independent variables. Assumptions for regression analysis were met, i.e., variance inflation factor values were≤1.06, indicating no presence of multicollinearity [42]. The residual plots showed a linear pattern. There were a few outliers in the subjective norms scale. However, eliminating them did not yield any change in the result.

To study the construct validity of the scales for indirect measures, exploratory factor analyses (EFA) were conducted using principal axis factoring and oblimin rotation with Kaiser normalisation. EFA are useful in an initial stage of creating a questionnaire, to get a first overview of its construction [43]. Principal axis factoring is preferable when the questionnaire is primarily intended to be used on similar populations as in the testing [43]. The oblimin rotation method allows factors to be correlated, which often produces more accurate results in research of human behaviour [39]. The number of factors was determined with eigenvalue > 1. Assumptions for EFA [39, 42] were met, i.e., all Kaiser-Meyer-Olkin Measures of Sampling Adequacy were > 0.6, and Bartlett’s tests showed significance (p < 0.001). Moreover, several correlation coefficients (Pearson’s r) between items were > 0.3, indicating that items were correlated enough for a factor analysis to be meaningful [39]. In accordance with the literature [42], linearity was tested by creating a series of scatterplots between some of the items; these did not show any pronounced deviation from linearity.

Missing values within the scales did not exceed 2.5%, and overall, 1% of the values were missing. A few of the missing values were due to that items about the employer and/or colleagues were unsuitable for self-employed women, and items about friends were unsuitable for women that felt they did not have any friends. To deal with missing data, ‘exclude cases pairwise’ was applied in the regression analyses and the EFA. The significance level was set to p < 0.05. All analyses were conducted using IBM SPSS Statistics version 27.0 [44].

3.1.1 Elicitation study

The 20 women included in the elicitation study were aged 28– 63 years (median 45 years). At the time of the interviews, three of the women were working full time, while three were participating in work-oriented rehabilitation or combining work with parental leave/leave of absence. Fourteen were still on long-term sick leave. The most common diagnosis was a stress-related disorder (n = 13), and the most common professions were in the health/social care or school sectors (n = 12). Having a poor working environment was a frequently reported cause of sick leave (n = 14). Other reasons were traumatic events in private life or high demands on oneself. Most women (n = 14) had previous experience ofCMDs.

Forty-one potential items for indirect measures were derived from the interviews, equally divided between the three scales for indirect measures. Regarding behavioural beliefs, i.e., the advantages and disadvantages of RTW, the women mentioned almost as many advantages as disadvantages. However, when applying the 75% rule, most advantages were retained and most disadvantages were excluded. The retained advantages concerned different areas in life, while the retained disadvantages only concerned symptom burden. For normative beliefs, i.e., supporters and non-supporters of RTW, the women mentioned more supporters of RTW than non-supporters, and when the 75% rule was applied, more supporters than non-supporters were retained. A general pattern for subjective norms was that the closest associates (e.g., family/friends) were retained and more distant associates (i.e., superficial acquaintances or society as a whole) were excluded. Regarding control beliefs, i.e., barriers to and facilitators of RTW, the women mentioned more facilitators of RTW than barriers to it, but after the analysis according to the 75% rule [29], more barriers than facilitators were retained. Overall, perceived behavioural control concerned (reasonable) demands. Retained items for indirect measures are presented in Table 1, and the excluded beliefs for potential items are provided in the supplementary material in Table B.

3.1.2 Pilot interviews to determine content validity

On the whole, women were of the opinion that the items were clear and easy to answer, and most of their opinions were about ambiguities in the background questions. However, one woman felt that she had to concentrate because the endpoints sometimes changed. Another woman felt that some items were difficult to respond to if one thought about them from multiple perspectives, but easy to respond to when scored based on what first came to mind. Three of the women felt that the questionnaire was repetitive, but this was described as positive, because this meant it required less energy to complete. Another opinion was that ‘meaningfulness’ and ‘routines’ felt like clichés. The structure was perceived as confusing by two women because the items were presented in groups of three even though they had nothing in common. When reviewing the questionnaires afterwards, it seemed that one woman might not have noticed the change in endpoint direction in the middle of the questionnaire. None of the women felt that the questionnaire was too long; it took about 15 minutes to complete. In summary, the pilot interviews resulted in some changes in structure and additional clarification of the background questions.

Results of the reliability tests are shown in Table 4. Internal consistency for the scales for direct measures varied in the range 0.43– 0.92. The intention and attitude scales had the highest internal consistency. The subjective norms and perceived behavioural control scales showed low internal consistency. Inter-item correlations did not support removal of any items in the subjective norms scale, but showed that the item ‘Whether I return to work/stay at work is entirely up to me’ in the perceived behavioural control scale was weakly correlated with the other items in the scale as well as with the total scale score (Spearman’s rho = 0.08). Elimination of that item increased the alpha value from 0.48 to 0.60. Test-retest stability for the scales for indirect measures showed moderate to excellent stability (ICC 0.70– 0.92), with attitude showing the strongest stability and subjective norms the weakest.

Standard multiple regression analyses showed that the direct and most of the indirect scales were significantly associated with each other, which confirm the construct validity of the model. The analyses demonstrated that advantages (indirect subscale for attitude) and supporters (indirect subscale for subjective norms) were significantly associated with attitude and subjective norms, respectively, but that disadvantages and non-supporters were not. However, barriers and facilitators were both significantly associated with the direct scale of perceived behavioural control. Associations between direct and indirect scales are shown in Fig. 2.

The EFA on the scales for indirect measures showed that factors were generally in line with the theory, with one deviating pattern: Advantages (subscale within indirect measures for attitude) and supporters (subscale within indirect measures for subjective norms) were divided into two factors each, one related to life as a whole/personal life and one related to work. Therefore, three-factor solutions were shown for these scales. Also, the item about improved economy (advantage within indirect measures for attitude) loaded strongest together with disadvantages. The results of the EFA are presented in Table 5.