Abstract
Background:
This study investigates work-related musculoskeletal disorders risk estimation by frequently as used as ergonomic methods in the field.
Objective:
To identify the difference in risk estimation by an in-house observational method and a self-reported questionnaire, and to evaluate the complementary aspects of these methods.
Methods:
A sample of 15 operators who worked on the assembly workstations was selected from a truck manufacturing plant. The risk assessment of these workstations (28 scenarios) was performed by the observational method and the self-reported questionnaire. The agreement between both methods to identify risk situations was measured with the weighted Kappa coefficient.
Results:
The observational method and the self-reported questionnaire deployed on the same activity estimated different risk situations.
Conclusion:
This analysis does not reveal that one tool is more powerful than the other one, but shows the probability of different risk estimation. The complementary effect of each method might be considered for further investigation concerning musculoskeletal risk factors.
Introduction
Work-related musculoskeletal disorders (MSD) are multi-factorial, as they are caused by several factors [1]. In 2017, MSD represented 87% of recognized occupational diseases. They are also the leading cause of lost working days due to absenteeism, with more than ten million working days lost in 2015 [2]. Evaluating job characteristics constitutes a significant challenge in identifying the levels of exposure to MSD risk. Practitioners must be able to evaluate MSD risk with valid and reliable assessments [3]. Several methods have been developed to assess the exposure to risk factors, including direct measurements, observational methods, questionnaires, and interviews [4, 5]. However, engineers and ergonomists need reliable and valid data, taking into account the variation and diversity of the job and the individual.
The results of risk estimation could encounter various possible biases related to data collection me-thods. Previous research has shown that direct mea-surement methods provide more reliable data than observation or self-reported questionnaires [6]. How-ever, direct measurement methods are time-consum-ing and require various support and skills [4, 7]. Two methods are commonly used to obtain ergo-nomic data on workers’ activities: observational methods and self-reported questionnaires [4]. Several previous studies have used observational methods to assess MSD risks [8, 9]. Paper-based observational methods, such as Rapid Upper Limb Assessment (RULA), Occupational Repetitive Action (OCRA), Rapid Entire Body Assessment (REBA) and Quick Exposure Check (QEC) tools [10–13] are used to address physical risk factors. Moreover, several large industrial companies have developed their in-house observational methods to identify risk factors that are specific to their sectors. Automotive industries, such as Volvo Car Corporation (VVC), Peugeot-Citroen (PSA), SCANIA and General Motors, have developed their own in-house method for their ergonomics programs [14, 15]. Furthermore, large companies such as Fiat, Bosch, and Volkswagen used European Assessment Worksheet (EAWS) tool [16]. Eliasson et al. reported that knowledge about risk assessment tools is low, and no systematic approach is defined for using these tools [17].
Self-reported questionnaire mostly used in epidemiological studies allows estimating the exposure to MSD risk by operators [18, 19]. Several studies have reported poor-to-moderate validity of the self-reported questionnaire compared to the direct measurement/observational method [20, 21]. However, Descatha et al. reported that self-reported questionnaires were more reliable and sensitive than observational methods [22]. Stock et al. and Barrero et al. noted that current studies of self-reported questionnaires could not demonstrate the validity of self-reported exposure methods due to study design limitations [19, 23]. These studies have differed regarding the methodologies, the study population (age, gender, and education), and the questions so that their unanimous conclusions cannot be generalized [23].
Regarding the diversity of the conclusions about risk assessment tools, this study was designed to investigate the complementary aspects and the risk estimation difference between an in-house observational method and self-reported questionnaires in the automotive industry.
Methods
Data collection
Observational method
An in-house ergonomic observational method [24] with a video recording was used to analyze the selected workstations. This observational method evaluates 20 risk criteria, which are grouped into four categories, including repetitiveness, working posture, manual handling, and energy consumption (Table 1). The weights of objects, the magnitude of forces (using a dynamometer), and handle diameters (using caliper) were measured and recorded. Manual handling and lifting of loads with two hands were studied in more detail using the National Institute of Occupational Safety and Health (NIOSH) equation [25]. The results were classified according to a color-coding scale. The green level indicated an acceptable situation, with minimal MSD risk. The yellow level indicated a moderate risk situation, which needs to be improved in the future, and the red and double red level corresponded to high-risk situations, which must be modified as soon as possible. The observational method had two high-risk categories (red and double red) that we merged both of them as a red category to facilitate the study analysis. After studying each criterion of the observational method for each workstation, the number of green, yellow and red criteria determined the final color of that workstation (Table 2). This color-coded method is based on Swedish guidelines, and it has been used in other observational methods, particularly in the automotive industry [14, 26]. Regarding the daily rotations of the various operators to all workstations in an Improvement Groups (IG), we developed a color-coded method representing the risk level for each criterion of an IG (including several workstations). This method is based on the logic of color attribution to one workstation (Table 2). The five thresholds were thus defined to determine the final color (green, yellow and red) for a criterion in an IG (Table 3). All threshold definitions were rounded down, and the most severe color decided the final color of each criterion for the IG.
Explanation of criteria evaluated by the observational method and the self-reported questionnaire. The similar sub-criteria for each method were regrouped as the exposure criteria
Explanation of criteria evaluated by the observational method and the self-reported questionnaire. The similar sub-criteria for each method were regrouped as the exposure criteria
*Several questions for one criterion regrouped based on the frequency of occurrence of the exposure level (rare, often, and always) to achieve a final evaluation.
Prioritization of risk factors by the observational method and the NIOSH equation method for a workstation
*The worst color dictates the final evaluation of the work position. **Final color based on the number of criteria evaluated Yellow and Red.
Final color definition for each criterion for the improvement group (IG: including several workstations). This method is developed based on the color-coded method explained in Table 2
*N = the number of measurements (workstation) in an IG. **The worst color dictates the final evaluation of the IG for a criterion.
A self-reported questionnaire was used to evaluate the operators’ perceptions of the physical exposure of their jobs. Several ergonomic epidemiological studies in France have used this tool to evaluate physical exposure [18, 27]. This tool is composed of questions designed to identify the potential physical risk factors for MSD. It was developed using the European consensus criteria document for the evaluation of MSD [28]. The questions concern repetition, neck, shoulder, wrist/hand and back postures, material handling, and force/effort for the entire body and wrists (Table 1). The response scale for each question has four levels: Never/Rarely/Often/Always. As shown in Table 1, several questions were asked to assess one ergonomic criterion. To ensure a single answer for each criterion, we combined the responses of several questions. If, for example, the answer to any of the three questions was ‘Always,’ then the final answer was ‘Always.’ If the answer to one of the three questions was ‘Often,’ then the final answer was ‘Often,’ otherwise, it was ‘Never/Rarely.’
Data collection
This study was performed from September 2012 to August 2013. Analysis using the observational method was conducted at the first stage from September 2012 to March 2013 by an ergonomist worked in the industry. Twelve operators included in the observational phase were again assessed in different scenarios based on the video recordings in August 2013 (Table 4). Twenty-nine scenarios were evaluated (various truck models) for three IGs. Questionnaires were distributed in July 2013 on a Friday to allow operators to complete them carefully over the weekend, and they were collected on Monday, ensuring a high response rate. Fifteen operators responded to all questions, and they were included in the final analysis (Table 4). The institutional review board approved the study and informed consent was obtained from all subjects.
An ergonomist with the help of an assistant ob-served and video recorded all the scenarios. The ergonomist analyzed the workstation by the observational method by viewing work in person. The video recordings were also used to review and revise the assessments.
The number of operators included in the study and the number of scenarios evaluated by the observational method for the workstations in the various Improvement Groups (IG). An ergonomist observed and assessed all the scenarios
The number of operators included in the study and the number of scenarios evaluated by the observational method for the workstations in the various Improvement Groups (IG). An ergonomist observed and assessed all the scenarios
*The worst results of each operator considered as a final evaluation of the workstation. **Two operators were excluded from the observation phase because of unavailability. ***A video recording was performed for each scenario.
We selected 11 criteria from the observational method and the self-reported questionnaire for comparison (Table 1). When several questions existed for one criterion in the questionnaire, they were regrouped by a statistical method based on the frequency of occurrences of the exposure level (rare, often, and always) to achieve a final evaluation. The “manual handling of loads with two hands” and “one hand” criteria were studied at two different levels, as the questions on the self-reported questionnaire concern manual handling of various loads and do not specify whether these loads are handled with one or two hands. Therefore, two subgroups were defined for these two questionnaire items: manual handling with two hands allowing analysis of loads weighing from 10 to 25 kg and loads weighing more than 25 kg, and manual handling with one hand allowing analysis of loads weighing from 1 to 4 kg and > 4 kg.
The criteria for effort/force of arms and the effort/force of the entire body corresponded to the same questions in the self-reported questionnaire. The various compared criteria from the questionnaire and observational method are presented and defined in Table 1.
Statistical analysis
We used descriptive statistics to summarize the percentage of risk factors in different IGs. The responses to the questionnaire (exposure to risk factors of different workstations in one IG) were compared with the results of the observational method for the IG. Categories of ‘Never/Rarely’ from the questionnaire and ‘green’ from the observational method were considered to be low risk. ‘Often’ from the questionnaire and “yellow” from the observation were a moderate risk, and ‘Always’ from the questionnaire and ‘red’ from the observation were high risk. The agreement between the criteria of the two methods was assessed using the weighted Kappa coefficient [29]. The Kappa coefficient interpretation is presented in Table 5 [30]. The unit of comparison between both methods was the operator.
Interpretation of the Kappa coefficient (30)
Interpretation of the Kappa coefficient (30)
Observational method
Table 6 presents the results of the ergonomic risk assessment for three IGs (IG1, IG2, and IG3), according to the observational method (the results of various workstations are shown in the Appendix).
Analysis of observational method and self-reported questionnaire of physical risk factors for Improvement Groups (IG) 1, 2, and 3 for 11 criteria of physical risk factors (see Appendix for details)
Analysis of observational method and self-reported questionnaire of physical risk factors for Improvement Groups (IG) 1, 2, and 3 for 11 criteria of physical risk factors (see Appendix for details)
aEight scenarios were evaluated at the Improvement Group 1, 12 scenarios at the Improvement Group 2, and 9 scenarios at the Improvement Group 3 (see the Appendix of this paper). b“Green” and “Never/Rarely” show low risk; “Yellow” and “Often” show moderate risk; “Red” and “Always” show high risk. cThe items of the observational method for two and one-handed manual handling evaluation. dThe questions of the self-reported questionnaire for manual handling evaluation.
Whole-body work and back, neck, shoulder, and wrist postures were the main risk factors identified in IG1. Awkward wrist posture was reported for all workstations. Exposure to risk factors, such as one-handed manual handling and surface area for pressure, was low in IG1 (Table 6).
The results for IG2 showed high-risk exposure for the wrist and shoulder. The categories repetitiveness and manual handling with two hands were low, while back and neck posture, manual handling with one hand, and whole-body force/effort were moderate. The final risk evaluations for back, neck, shoulder, and wrist postures and whole-body force/effort for this IG were high (red). Awkward body posture was observed at most workstations in IG3 (see the Appendix). Wrist posture and manual handling with two hands were red at many workstations, while repetition and surface area for pressure were green.
Analysis of the self-reported questionnaires for the three IGs showed that 13 operators (87%) identified awkward back postures as being present often at their work positions. Repetitiveness and awkward whole-body work postures were identified as being “Often” present for 5 (33%) and 9 (60%) operators, respectively. Furthermore, 10 operators (67%) reported “Always” for exposure to repetitiveness (Table 6).
All operators in IG1 and IG2 reported that they were often exposed to awkward back postures. More than half of the operators in IG1 reported that they were always exposed to manual handling, awkward wrist postures, and excessive effort/force of the body. For IG3, force and effort of the whole body were often or always present. The majority of operators reported “Often” for exposure to different risk factors (see the Appendix).
The difference of risk estimation between the observational method and self-reported questionnaires
Table 7 presents the results of a comparison of the data derived from the observational method and the self-reported questionnaire for the three IGs. Both tools identified several risk factors, while the results for certain factors differed considerably, according to the method of analysis, particularly for the back (weighted Kappa coefficient = –0.29), shoulder, neck, and wrist postures and repetitiveness (weighted Kappa coefficient = 0). For most criteria, the risk estimation of the two tools was similar in the moderate risk range but different for extreme situations (high risk and no risk). The agreement between both methods for whole-body effort/force, as well as for effort of palms of hands, was better than that for the other criteria (weighted Kappa coefficient = 0.07 to 0.09). Handling criteria (component size) and two-handed manual lifting imply a low agreement between the operators’ estimations and the ergonomist’s assessments in the material handling criteria (Kappa factor = –0.05 and –0.03); however, the percentage of agreement was 53% between two methods for the one-handed manual lifting criterion.
Comparison between observational method and questionnaire with the calculation of the Kappa coefficient and proportion of agreement
Comparison between observational method and questionnaire with the calculation of the Kappa coefficient and proportion of agreement
This study compared the difference of risk estimation obtained with two risk assessment tools: the in-house observational method and the self-reported questionnaire. The agreement between these tools was investigated for the identification of physical risk factors in a truck assembly plant. This study showed that the observational method and the self-reported questionnaire do not represent a similar risk estimation, and they provided contradictory risk estimation for the analysis of certain physical risk factors. Several studies have performed this type of comparison and have reached different conclusions. Descatha et al. concluded that the results of self-reported questionnaires differed from those of the observational method and that the self-reported questionnaire was a better predictor of the incidence of future MSDs [22]. The study by Spielholz et al. and Chiasson et al. reported that the operators’ perceptions were different from the results of the reference methods (observation and direct measurement), and the self-reported questionnaire was unreliable [21, 31]. Three review articles have reported a low-to-moderate agreement between self-reported questionnaires and observational methods [19, 32]. The observational method is based on the expertise of an ergonomist. The observation grid and the ergonomist’s knowledge guide the evaluation [32]. In contrast, data from the self-reported questionnaire were essentially based on the operators’ perceptions concerning their job, and it might differ in terms of the perceived muscular fatigue and musculoskeletal pain [31, 32].
These two methods estimated repetitiveness, differently, as the observation tool revealed a low level of exposure, while the self-reported questionnaire identified repetitiveness as a commonly present risk factor. Other studies have also reported poor agreement for repeated movements evaluated by questionnaires and other reference methods [34]. Lowe and Krieg reported low accuracy of the counted number of repetitive motions by observation analysis compared to direct measurements, although the severity of repetitive motions was accurately estimated [33].
The results concerning whole-body working, neck, and back postures varied considerably, as the self-reported questionnaires revealed a lower risk than the observational method. Burdof and Laan reported that operators might underestimate the trunk postures adopted at work [35]. Takala et al. and Stock et al. reported that micro-postures (the neck and wrists and trunk rotations) are difficult for observers and operators to determine [5, 19]. Lowe reported that the ergonomists might underestimate posture duration severity because of discrete intervals of angular position [36].
Observational analysis of shoulder postures revealed high exposure in all groups, while only 13% of the operators identified these postures as being constraining on the self-reported questionnaire. The results for wrist postures presented a similar tendency, with higher sensitivity for the observational method and less definitive results for the self-reported questionnaire. Operators’ reports in our study often underestimated the postures adopted and tended to focus on the pain experienced at a particular point in time. Some types of pain are experienced in the context of overexposure, and it is only at this time that the operator becomes aware of the posture adopted [31]. Previous studies have reported inconsistent results regarding the presence of MSDs and estimation of the risk exposure by operators. Balogh et al. and Chiasson et al. reported the impact of MSDs on the overestimation of the exposure [31, 37], while Burdorf and Laan, and Toomingas et al. found no relationship between MSDs symptoms and overexposure reporting [35, 38].
The self-reported questionnaire and the observational method did not provide consistent results for the handling characteristics. Few studies have compared the results of various analysis tools for this type of criterion. Nevertheless, Stock et al. reported a moderate correlation between the results of such tools for the handling of large objects [19].
Both tools identified the whole-body force/effort risk factor. A slight agreement was observed between the two methods. The force/effort required by a task was measured by a dynamometer and reported as the results of the observational method, while the questionnaire provided the general perceptions of the operators. Based on practice and experience, operators might identify their exposure to the effort. However, an operator may become so accustomed to the working conditions that he/she no longer accurately perceives the effort involved in performing an activity. Working habits, individual experience, and perceptions are essential elements in identifying high-risk exposure, as the level of sensitivity of an operator can result in different responses to the same situation [31]. Other studies have reported a good agreement between these tools for the whole-body force/effort criterion [19]. However, a low agreement was reported in four studies that compared assessments of push/pull forces using questionnaires and observational methods [32].
The proportions of the agreement for manual handling of loads with two hands and one hand were moderate. Despite certain limitations concerning the analysis of these criteria, our results align with those reported by Stock et al., who demonstrated poor agreement, particularly for questions about the number of hours/working day spent lifting or carrying loads [19].
Several explanations can be proposed for the different results obtained with these two methods. First, the low Kappa coefficient that was, generally, observed between the compared criteria might not necessarily be related to the disagreement between the methods. A highly agreed estimation may receive a low Kappa coefficient because other factors, such as limited variability in the distribution of exposure in the categories influenced this coefficient [39]. We used in this study a weighted Kappa coefficient that is more adapted for the data with limited variability in the distribution of exposure in the categories.
The limitations of our study such as the reliability of the observational methods and questionnaire (question formulation, response scale, etc.), small sample size, and the respondents’ pain/fatigue, were the sources of errors in this experimentation. In the current study, pictograms were used to represent degrees of flexion/extension of each body segment in the questionnaire (Table 1), while the categorical limit was used for observational methods. One limitation may be that we did not compare identical variables. Nevertheless, providing categorical limits in the questionnaire might be a source of error, as the operators might incorrectly estimate the degree of flexion/extension on a numerical scale [36, 41]. The pictogram considered the workers’ mental representations of the workload and provided meaningful measures. Another limitation was the small sample size, but the real setting constraints make it impossible to include more participants. The bias induced by the time interval was negligible as we had a short time interval between two measurement methods. We also revised and modified the results of the observational method using videos recorded less than two months after the operators completed the questionnaires. Few variations occurred in the work situations over this short time interval.
Conclusion
This study compares the difference in risk estimation by two frequently used methods in the field. The probability of risk estimation differed between the observational method and the self-reported questionnaire. This difference could enable us to recognize the positions and roles of these tools in the analysis of MSD risk factors and might highlight the complementary aspect of each method. This study did not analyze the validity of these two methods or demonstrate the superiority of any method. However, the findings raise several questions concerning the level of risk estimation using the two frequently used ergonomic methods in the field. We propose to extend this comparison to other tools used in risk assessment, such as interviews and the direct measurement method, which would provide more information on the validity and the use of each method during risk assessment in the workplace.
Conflict of interest
No conflict of interest was reported for this study.
Funding
This study was financially supported by the Agence nationale de sécurité sanitaire de l’alimentation, de l’environnement et du travail, ANSES, (EST-12-007; 2012/2/007) and Angers SCANIA Production.
