Evaluating the validity of the Work Role Functioning Questionnaire (Canadian French version) using classical test theory and item response theory

1 Introduction

In the field of work disability prevention research, little attention has been given to presenteeism (i.e., “attending work while ill” [1]) compared to work absenteeism. Yet, it has been found that presenteeism is a prevalent problem and accounts for higher productivity losses and costs compared toabsenteeism [1]. For example, prevalence rates between 45% and 64% were reported among the working population [2, 3]. The cost of lost productivity due to common pain problems was estimated at US$61.2 billion per year [4]. The largest part of indirect costs made on employees’ claims was found to be associated with presenteeism (63%; $311.8M)) compared to absenteeism (6%; $27M) [5].

Although presenteeism is an important problem for organizations, few validated measurement instruments have been developed as yet. A systematic review analyzed 16 articles assessing 7 of these instruments and concluded that none demonstrated satisfactory results and that there is a lack of evidence to recommend one over another [6]. To better understand presenteeism and eventually develop effective strategies, researchers and organizations need to rely on valid instruments.

Currently, the Work Role Functioning Questionnaire (WRFQ) and Work Limitation Questionnaire (WLQ) are among the most studied presenteeism instruments [6]. These self-administered questionnaires assess the perceived impact of a health problem on workers’ ability to perform their job [7]. Aside from turning up at work despite ill health (presenteeism), there is evidence that workers’ limitations to perform work demands have a negative impact on their work productivity [8]. Consequently as presenteeism is related to work ability/productivity, the WRFQ and WLQ have been used as a proxy for assessing presenteeism. The items in these questionnaires describe a number of work demands chosen because of their frequent occurrence in a variety of jobs and their importance as identified from the workers’ perspective [7, 9]. Both questionnaires are grounded in the same conceptual framework and their items were all drawn from the same pool [9]. They differ in the number of items, recall period, and response set [9]. These questionnaires are popular in several countries, which is demonstrated by the number of published cross-cultural adaptations: Canadian French [10], Dutch [11], Brazilian Portuguese [12], Turkish [13], and Spanish [14].

To our knowledge, very few studies on the factorial composition of the WRFQ and WLQ can be found in the literature. Factor analysis is a statistical method to identify clusters of related variables and is important for the validity of an instrument [15]. It is essential for construct validity by assessing the internal structure and the cross-structure of items of an assessment instrument [15]. Factor analysis is also important for content validity by providing valuable information to revise an instrument (e.g., identify items that need rewording) [15]. Moreover, it can be useful for choosing instruments that can be used as predictors (predictive validity) [15]. Among existing studies on the WRFQ and the WLQ, contrasting results have been found for their dimensional structure [6]. The objective of this study was to test the validity of one version of the WRFQ, the Canadian French version (WRFQ-CF), by examining its factorial composition and items.

In addition, up to now, the WRFQ has been studied using a Classical Test Theory (CTT) approach. CTT is the most popular measurement approach and has existed for more than a century [16]. A more recent approach has been proposed, specifically Item Response Theory (IRT). These two approaches differ in their basic assumptions, orientation, information provided, and sample size [17, 18]. It has been recommended that these two approaches can be used to provide a quantitative assessment of items/scales and maximize the content validity of patient-reported outcome measures [19]. In this study, we thus tested how IRT could contribute to the content validity of WRFQ-CF by comparing the results with those from the CTT approach.

In summary, the main objective of this study was to test the validity of the WRFQ-CF and more specifically the following: (a) to determine the internal structure of the WRFQ-CF, (b) to identify problematic items using two measurement theory approaches (CTT and IRT), and (c) to compare the results obtained from the two measurement theory approaches.

2 Method

3 Results

To our knowledge, the measurement properties of the WRFQ-CF were only tested in one study [10]. To contribute to its improvement, this study aimed to test the validity of the WRFQ-CF by assessing its items and factorial composition. It used two different measurement theory approaches (CTT and IRT), which gave similar and complementary results with nine weak items identified in both approaches. Three-factor and four-factor models were tested, which generated different items on work scheduling and output demands. The three-factor model showed good model fit with fewer problematic items.

To date, few studies on the factorial structure of the WRFQ or its alternate forms can be found in the literature. In previous studies, the number of factors studied ranged from three to five. The original article on the WLQ-25 identified four distinct factors: time, physical, mental-interpersonal, and output demands [9]. Tang et al. [36] compared four- and five-factor models and found that both models showed acceptable goodness-of-fit indices, with the five-factor model performing better [36]. Also, Walker, Michaud, and Wolfe [37] carried out a factorial analysis of the WLQ-25 and extracted three factors (eigenvalue more than 1.0) with one predominant factor explaining 77% of the variance. Several papers have combined the social and mental demands dimensions [9, 37]. Yet, in this study, all items in the social demands factor were found problematic using both measurement theory approaches, which prevented us from testing a five-factor model. The social dimension plays a salient role in presenteeism. For example, studies have found that workers who have a high level of support and integration at work will tend to show up ill at work compared to those with poor social support and integration [1]. We recommend that the social demands items be revised and further tested to provide better discrimination.

In this study, the work scheduling and output demands subscales were correlated. When using a four-factor model, three items in the output demands factor (O1, O2, and O6) loaded more on the work scheduling demands factor. Also, all except one item of the output demands were found weak by either measurement theory approach. This result suggests that these two factors could be combined. One study on the Dutch version of the WRFQ-27 also combined the work scheduling and output demands factors and suggested a new version composed of four subscales: (1) work scheduling and output demands, (2) physical demands, (3) mental and social demands, and (4) flexibility demands [45]. These two factors share some similar characteristics. Work scheduling demands were defined as “worker’s needs to manage the workday from beginning to end” and output demands as “activities related to completing work on time, with high quality and to everyone’s (including the worker’s) satisfaction” [38]. Both subscales involve a time component, which might explain the correlations between the factors. Also, these items might refer to control over tasks, which has been found to be an important risk factor for presenteeism [39].

The purification of items was conducted on the basis of exploratory and confirmatory factor analysis as well as of IRT. For the confirmatory factor analysis, we used four fit indices. Some may argue that one or two indices are enough, while others suggest combining different indices. Since there is no consensus in the literature and several criteria may influence the performance of each fit index [40], we preferred to use a combination of several types of indices to provide a better picture of the model fit. We used those proposed by Kline [28]. Also, in this study, the chi-square estimate did not support model fit. The chi-square estimate usually provides a reasonable measure of fit when the sample size ranges between 75 and 200 [41]. This estimate will often be statistically significant with a larger sample size [41]. Hence, the poor chi-square value might be explained by the sample size used in this study (n = 352).

In this study, the number of problematic items ranged from 10 (3-factor model EFA/CFA analysis) to 13 (4-factor model EFA/CFA analysis and IRT). If we consider all items that were identified as problematic by either approach, most factors will have fewer than three items. It is not recommended using fewer than three items per factor for patient-reported outcomes since the interpretation of some factors will be weak and will not properly assess the construct [24]. Revision of the weak items should be considered, especially the nine items found problematic by both approaches (Table 4). The results of this study could also be used to develop a short version of the WRFQ-CF. In the literature, some studies have reported short versions of the WRFQ and WLQ, such as WRFQ-15, WLQ-16, and WLQ-8 [6, 42].

This study used IRT to test items. To our knowledge, no other study on the WRFQ has used this approach. Also, in the field of work disability, IRT is less frequently used compared to CTT (e.g., [43–45]). Yet, IRT provides some advantages over CTT. For example, IRT can provide information on the relationship between the item and the latent trait. Thus, each item can be examined independently to determine their contribution in a test [18]. This allows for the comparison of different tests measuring the same ability [18]. This offers an advantage over CTT that requires test forms to be parallel before their scores can be compared [17]. In IRT, the items from different tests can be placed on a common scale, which enables the comparison of the level of difficulty of the tests as well as the development of item banks [18]. In this study, non-parametric IRT was useful to corroborate results obtained from CTT. Nine items were found to be problematic in both approaches. Also, using IRT gave complementary results with four additional items. This difference can be explained by the level of analysis of both approaches: CTT’s focus is on test-level information (comparing the items to the entire scale) whereas IRT focuses on the item level [18]. The results found in IRT can guide the choice of items and identify those that need revision.

Differences can be identified between studies on the WRFQ (and its alternate forms) and this study, which might explain the discrepancies between the number of factors identified and the items retained. First, studies on factorial validation of the WRFQ have mainly been conducted with workers having musculoskeletal disorders whereas in this study the workers were not recruited based on a specific diagnosed health problem. However, in this organization, distress was found in 62% of the sample, including 41% with high levels of psychological distress [21]. Other studies have also used the WRFQ with the general working population, which supports the need for validation studies on this population [46–48]. Our study participants were from the service sector, which is a sector at risk of presenteeism [2]. Also, presenteeism is not limited to workers having musculoskeletal disorders and affects a large proportion of the workforce [49]. Other health conditions such as allergies, diabetes, and headaches were found among the top conditions associated with productivity loss [50]. Documenting presenteeism in the general working population can help to put in place appropriate approaches to reduce its impact. Second, the statistical analysis methods were not the same. Among the few factorial validation studies identified, EFA (mainly PCA and orthogonal rotation [37, 47]) and CFA [36] were used. In this study, ML with oblique rotation was used. We did not use PCA because it only accounts for variance in the observed variables and does not differentiate between common and unique variance [24]. Also, orthogonal rotation is usually not recommended when the constructs are often correlated with one another [24]. In this study, oblique rotation was used and was justified by the presence of correlations between factors (ranging from 0.202 to 0.684).

This study has some limitations that need to be considered. First, the study population was mainly office workers from one government agency. Hence, it is not possible to generalize the results of this study to the general working population. Second, we did not perform cross-validation, i.e. randomly split the sample into two groups to check if the factor solutions can be replicated across groups [34]. In this study, since the WRFQ-CF has 27 items and the communalities were moderate, a sample size of at least 200 was targeted [24]. The sample size was large enough (n = 352) for factor analysis based on a subjects-to-variable ratio of 10 : 1 [34]. Third, we used TestGraf, a software allowing non-parametric IRT for item purification. One main limitation of TestGraf is the lack of pre-assigned cut-off values [51]. Hence, the interpretation of graphs generated in TestGraf might be based on the subjective judgment of the researchers. However, in this study, there were no uncertainties and disagreement among our team during the visual inspection of the graphs since all the identified items clearly showed some weaknesses (e.g., flat curves, dominating options) (Appendix 1).

5 Conclusion

The WRFQ and WLQ questionnaires are widely used to assess presenteeism. These questionnaires can be used as part of a broader work disability prevention process, for tailored intervention development, for establishing the prevalence of presenteeism, and for identifying risk factors. Testing the validity of a questionnaire is crucial to ensure that it serves its intended purpose. Factor analysis is invaluable for testing the validity of an instrument and provides information on the internal structure, and the items that need to be revised. To this end, this paper provides an example of the application and comparison of two measurement theory approaches (CTT and IRT). IRT is more and more recommended but it requires a large sample size to yield reliable results [19]. Non-parametric IRT is an interesting alternative method for item purification and could be further explored for the development of assessment instruments in work disability. Based on our sample, these two approaches provided similar and complementary results (9 common items and 11 items found in one approach). Also, in the literature, the number of factors determined for the WRFQ (and alternate forms) varies from three to five. In this study, we found fewer problematic items when using a three-factor model. Furtherstudies are needed on this questionnaire to refine the items and define its internal structure. Finally, studies on other measurement properties (e.g., reliability, convergent and discriminant validity) of the WRFQ-CF are required.

Conflict of interest

None to report.