Cross-cultural adaptation and reliability assessment of the Indonesian version of the Computer Workstation Ergonomics: Self-Assessment Checklist

Abstract

BACKGROUND:

Prolonged use of computer devices can have adverse health effects, but these can be mitigated by setting up computer workstations according to ergonomic principles. The Computer Workstation Ergonomics: Self-Assessment Checklist can guide workers in implementing these principles. However, the checklist is only available in English, which may reduce the accuracy of the instrument when used by non-English speakers, including Indonesians.

OBJECTIVE:

The objective of this study was to cross-culturally adapt the Computer Workstation Ergonomics: Self-Assessment Checklist for use in Indonesia. Additionally, a reliability assessment was conducted on the adapted checklist.

METHODS:

This study followed a six-stage cross-cultural adaptation process, including translation, synthesis of translation, back translation, expert committee review, pretesting, and documentation submission. The final version of the adapted checklist underwent testing for intra-rater reliability, inter-rater reliability, and internal consistency.

RESULTS:

The cross-cultural adaptation process resulted in an Indonesian version of the Computer Workstation Ergonomics: Self-Assessment Checklist. Furthermore, the Krippendorff’s alpha values for the intra-rater and inter-rater reliability of the adapted checklist ranged from 0.59 to 0.78 (mean = 0.70) and 0.20 to 0.82 (mean = 0.56), respectively.

CONCLUSION:

The study produced an Indonesian version of the Computer Workstation Ergonomics: Self-Assessment Checklist that had an adequate reliability. The adapted checklist can serve as a practical tool for evaluating and improving computer workstations in Indonesia.

Keywords

Adaptation computers workstation ergonomics assessment checklist Indonesia

1 Introduction

Digital devices, particularly computers, are essential working tools nowadays. In 2016, more than 70% of American households owned a computer and had an internet subscription [1]. The pandemic that induced tremendous digital transformation in the workplace was also increase computer use. Any business activities, including teaching and learning, shopping, or even doctor consultations, could be undergone without moving the legs through the information technology platforms [2 –5]. Majumdar et al. [5] found that screen time among office workers in India was 1.5 times higher than before the pandemic. The average increase in screen time was about 4 hours which could result in an average screen time of about 8 hours/day [6].

Although computer use may ease the work due to the reduced travel time, improperly managed computer work is associated with several adverse health effects. A high percentage of office workers who work with a computer daily experience musculoskeletal symptom in the neck, lower back, upper back, wrists, and shoulders [7, 8]. The pandemic also worsened musculoskeletal problems, where the prevalence in several body parts, e.g., neck and back, was four times higher [9]. In addition to musculoskeletal problems, computer work is also associated with eye problems. Office workers who work with a computer for a prolonged time are associated with an increased risk of dry eye disease [10]. The pandemic that increased screen time was also, unfortunately, increasing the prevalence of dry eye disease and other eye problems by more than 20% [11].

In addition to the prolonged duration that causes static sitting postures, adverse health effects caused by computer work could arise due to other risk factors, including repetitive movements, awkward postures, and contact stress. While typing using keyboards or operating a mouse, the wrist and fingers must be exposed to repetitive movements that could lead to work-related upper extremity disorders [12]. Awkward postures during computer work are commonly found when remarkably neck flexion is performed while working with a low monitor position, particularly the laptop without additional stands [13]. The edge sharp of office desks are a common source of contact stress found during computer work [14].

Office workers need to assess the risk factors that lead to adverse health effects observed in their workplace. Hence, they can establish ergonomic tasks and workstations when performing computer work. One approach that is relatively accessible due to the minimum investment is the observational method [15]. While an ergonomic assessment with direct measurement instruments captures the risks through the sensor recording, observational methods require the instrument users to make some observations guided by validated paper-based methods [16]. The observational methods, particularly in the form of a checklist, could be utilized by users with minimum ergonomic knowledge. Some checklists could be utilized with minimum training and experience [17, 18]. Also, since it only requires the users to answer the presence of such risk factors in the workplace using Yes/No responses, checklist use can be considered a rapid assessment tool compared to other methods [19].

A specific checklist, i.e., the Computer Workstation Ergonomics: Self-Assessment Checklist [20], is available to be utilized by the workers for conducting self-assessment to identify the room for improvement in their computer workstations. However, this checklist was written in English, which could create language barriers for office workers who do not use English for daily conversations. This includes the office workers working in Indonesia, where English was not the official and daily communication language. The language barriers could affect their perception on the words and sentences of the checklist, affecting the accuracy of the assessment results [21]. Hence it is essential to get the Indonesian version of the checklist, which should ease the utilization of the instruments [22, 23].

This study aimed to perform cross-cultural adaptation to get the Indonesian version of the checklist that office workers could utilize for developing ergonomic computer workstations. Also, reliability assessments were conducted by assessing the intra- and inter-rater reliability of the Indonesian version of the checklist. Establishing a reliable adapted checklist hopefully lessens the adverse health effects caused by computer work of Indonesian office workers.

2 Cross-cultural adaptation

2.1 The computer workstation ergonomics: self-assessment checklist

The Computer Workstation Ergonomics: Self-Assessment Checklist was a paper-based assessment tool that can be utilized to help office workers develop ergonomic computer workstations. By developing such a workstation, office workers should have a safe and healthy workplace that could minimize the adverse health effects associated with computer work. This checklist was disseminated by National Institute of Health, Office of Research Service, Division of Occupational Health, and Safety [20]. The checklist consists of seven sections, including inquiries regarding office chairs, input devices (keyboard and mouse), worksurface, breaks, accessories, laptops, and hot desking. In each section, the users need to respond to the inquiries by choosing either ‘Yes,’ ‘No,’ or ‘N/A.’ If the ‘Yes’ was chosen, the part of the workstation was considered to conform to the correct ergonomic principles. On the other hand, if the users considered that a component of the workstations did not match the description of the checklist inquiries, the user can review the suggested actions in the rightest part of the instrument.

2.2 Cross-cultural adaptation results

In getting the Indonesian version of the Computer Workstation Ergonomics: Self-Assessment Checklist, cross-cultural adaptation through six phases was performed following the standardized guidelines introduced by Beaton et al. [21]. The adaptation began with translation. Following that phase was the synthesis of translation, back translation, expert committee review, pretesting, and submission of documentation.

2.2.1 Translation

Two native Indonesian translators with different educational backgrounds carried out the translation of the checklist. The first translator was a certified English-to-Indonesian translator. The translator had a linguistics background without any prior knowledge of ergonomics. Another translator was an industrial engineering assistant professor with high English proficiency. The second translator had research interests as well as some international peer-reviewed publications in ergonomics. The translation was performed independently by the translators to minimize bias [24]. The translators were encouraged to write down any notes on the translation process, in addition to producing their translated versions (T1 and T2) of the checklist. Most of the sentences could be translated by both translators, but issues arose during the translation stage, including several difficult-to-translate words with no equivalent words in Bahasa Indonesia, ambiguous words or clauses that could be translated with more than one meaning, and some words that seemed more familiar in their English forms. Difficult-to-translate words included “workstation,” “lumbar roll,” “docking station,” and “hot desking.” Ambiguous words with more than one translation included “proper,” “fully,” “obtain,” and “holder.” Meanwhile, words that seemed better to be written in their English forms included “checklist,” “mouse,” and “keyboard.”

2.2.2 Synthesis of translation

The purpose of the synthesis was to establish a translation consensus (T12) between the two translators who were involved in the previous phase. To achieve this, an online focus group discussion (FGD) was moderated by one of the researchers (IPM). The two translators were asked to resolve any discrepancies in the translation results between T1 and T2. For some difficult-to-translate words, such as “workstation,” combinations of Indonesian words were used, even if they were not found in the dictionary. Conversely, words like “docking station” and “lumbar roll” were kept in their original English form due to the lack of equivalents in Bahasa Indonesia. The translators reached a consensus on other words that could be interpreted with multiple words in Bahasa Indonesia, such as “proper” and “fully.” Furthermore, an ambiguous clause, “the keyboard should be flat,” was clarified by adding some adjectives to make it more understandable. Words such as mouse, monitor, and keyboard, which Indonesians may be more familiar with in their English forms, were not translated.

2.2.3 Back translation

Two professional translators with American college/university degrees were commissioned to translate the Indonesian version of the checklist back into English. No information was given about the original checklist before the back translations. Both translators worked independently, submitting their own back translation reports (BT1 and BT2) upon completion.

2.2.4 Expert committee review

Three Indonesian university professors, who taught courses related to ergonomics, were selected as experts to review all documents and reports from the previous stages (T1, T2, T12, BT1, and BT2) and produce the prefinal version of the adapted checklist. These experts were proficient in English, having obtained their doctoral degrees in engineering from English-speaking countries. They all had research interests in ergonomics and had produced some peer-reviewed publications in the field. A consensus was reached by conducting a focus group discussion (FGD) to develop the prefinal version. The consensus also revealed the need to add illustrations for non-familiar words in the computer workstation checklist, such as “angle board,” “document holder,” “docking station,” “footrest,” “lumbar roll,” and “monitor stand.” Additionally, some illustrations were requested to clarify suggested working postures, including avoiding pressure from the chair seat on the back of the knees and maintaining a straight wrist position.

2.2.5 Pre-test

Ten administrators working in schools and universities who use computers for at least four hours per day in their workplace voluntarily participated in the pretesting of the adapted checklist. They were asked to evaluate their workstations using the prefinal version of the checklist while being videotaped, and to express any difficulties they encountered through a thinking-aloud protocol. After the assessment, a debriefing session was conducted through an interview to assess the comprehensibility of the questions. The pretesting revealed several difficulties related to the prefinal version of the checklist, including the formatting of some inquiries that consisted of two questions, as well as the use of non-question format sentences. Additionally, some participants faced difficulties due to their unfamiliarity with certain workstation components, such as docking stations, document holders, and angle boards. Based on the pretesting results, a second expert committee review was conducted to refine the adapted checklist, which resulted in the final version. The revision process included reformatting the inquiries to a question format and improving the spacing to enhance the readability of the inquiries.

2.2.6 Submission of documentation

An email was sent to the National Institute of Health, Office of Research Service, Division of Occupational Health and Safety to submit the final version of the adapted checklist and other reports of the adaptation process.

3 Reliability assessment of the adapted checklist

3.1 Participants

The reliability of the adapted checklist was assessed by involving 20 office workers (15 females, five males) who worked as administrators in universities or offices in Yogyakarta, Indonesia. All of them work with computers for more than 4 hours per day. The age of the participants ranged between 23–46 years, with a mean age of 29.25 (S.D. 7.56) years. Most participants graduated with a bachelor’s degree (95%).

3.2 Methods

Each participant was assigned a computer workstation that was developed based on typical workstations found in Indonesian offices [25, 26]. Prior to assessing the workstation using the Indonesian version of the checklist, participants were asked to perform a 15-minute simple data analysis task using spreadsheet software. Repeated assessments were conducted with a seven-day interval. At the end of the assessment, the results were evaluated by one of the authors (IPM) to determine if any deviated responses existed. The golden standards for comparison were determined through consensus among the authors [27]. Intra-rater reliability was assessed by comparing the assessment results of the repeated measurements, while inter-rater reliability was assessed by comparing the assessment results to the golden standards. Both reliability measures were calculated using Krippendorff’s alpha and calculations were performed using JASP 0.16.3.0 (University of Amsterdam, The Netherlands).

3.3 Reliability assessment results

The Krippendorff’s alpha statistics show that the intra-rater reliability of the adapted checklist ranged from 0.59 to 0.78 (Table 1). The highest alpha statistics of the intra-rater reliability were observed on inquiries regarding worksurface, while the lowest could be found on inquiries regarding laptops. Furthermore, Krippendorff’s alpha statistics indicating the inter-rater reliability ranged from 0.20 to 0.82. The highest alpha statistics of the inter-rater reliability were observed on inquiries regarding worksurface and breaks. The lowest alpha statistics of the inter-rater reliability were observed on inquiries regarding hot desking.

Table 1
Krippendorff’s alpha statistics indicating the reliability assessment results

Items Intra-rater Inter-rater

Reliability Reliability

Office chair 0.69 0.53

Keyboard and mouse 0.76 0.65

Worksurface 0.78 0.82

Breaks 0.70 0.82

Accessories 0.70 0.60

Laptop 0.59 0.27

Hot desking 0.67 0.20

Mean (SD) 0.70 (0.06) 0.56 (0.24)

Items	Intra-rater	Inter-rater
Office chair	0.69	0.53
Keyboard and mouse	0.76	0.65
Worksurface	0.78	0.82
Breaks	0.70	0.82
Accessories	0.70	0.60
Laptop	0.59	0.27
Hot desking	0.67	0.20
Mean (SD)	0.70 (0.06)	0.56 (0.24)

4 Discussion and conclusion

The Indonesian version of the Computer Workstation Ergonomics: Self-Assessment Checklist was developed through a series of cross-cultural adaptation phases, following the guidelines introduced by Beaton et al. [21]. Some noteworthy findings include the importance of retaining certain words in English format, as well as the inclusion of illustrations and reformatting to make the assessment process easier for potential users of the checklist.

The adaptation process of the checklist revealed that certain words should be presented in their English form, despite the fact that they have existing Indonesian translations in the dictionary. These words include “keyboard,” “mouse,” and “checklist.” This phenomenon can be attributed to the impact of push technology, which often introduces products into non-English speaking markets using their original English terms [28, 29]. Consequently, technology users in Indonesia become more accustomed to utilizing the English terms rather than their Indonesian translations [30]. Another research has also suggested that the widespread use of English terms in the technology sector has resulted in the creation of hybrid language, where English and Indonesian are combined in everyday conversations [31].

The expert committee review of the adaptation process revealed that there is a significant need to enhance the checklist by adding more descriptive language and illustrations. This would be instrumental in ensuring that users have a clear understanding of the instrument, thereby minimizing discrepancies between tool designers, professional raters, and novice users. As a result, this would increase the validity and reliability of the checklist for cross-cultural use. In particular, the use of illustrations would be beneficial in enhancing comprehension and reducing potential misinterpretations, especially among elderly or less-educated users [32 –34]. By incorporating more illustrations, the checklist would become more accessible and culturally appropriate, improving its overall effectiveness [32 –34].

This study also revealed that Indonesian office workers who participated in the study appeared to be confused when some inquiries were not formatted as questions. The question format was easier to recognize, as users are accustomed to providing answers to questions. This confusion may have arisen because they did not perceive that the statement was not solely informational. A similar finding was also observed in Dominguez et al. [35], where the expert committee review opted to rephrase certain statements as questions to create more concise sentences that are easier to comprehend.

In the future, the adapted checklist could be transformed into a smartphone application, which should increase its portability and affordability [36]. Additionally, this could lead to a reduction in paper and printer usage, which would be particularly beneficial if the assessments were conducted at the company level. By doing so, the assessment process could be made more eco-friendly, ultimately benefiting the environment [37]. The smartphone application could also incorporate augmented reality (AR) technology, which could provide workers with more visually intuitive instructions on how to perform interventions in their computer workstations. High-fidelity AR technology would allow users to experience a sense of presence in a virtual environment, which could improve their immersion and comprehension of the instructions without the need for physical products [38].

The reliability assessment of the adapted checklist revealed that the worksurfaces section of the checklist, which includes inquiries related to monitor positions, glare, lighting, and office stationery, demonstrated the highest reliability. This may be attributed to the more tangible nature of the objects on the work surface. This finding is consistent with Spielholz et al. [39], who noted that exposure variables that are easier to estimate by examining objects, such as postures, tend to have higher reliability compared to more abstract variables like force and duration. On the other hand, the laptop and hot desking sections had the lowest alpha, indicating low inter-rater reliability. This may be due to the unfamiliarity of the participants with docking stations, which have only recently become common in Indonesian offices. Additionally, users may tend to respond “No” to hot desking inquiries instead of providing “N/A” responses due to fatigue from the assessment, as these questions were found in the last section of the checklist. However, these inaccurate responses are unlikely to have a significant impact on the assessment of the working station, as hot desking is not a typical setting in Indonesian offices.

Ethical approval

The study protocol was approved by the Ethics Committees of Universitas Gadjah Mada (No. KE/UGM/003/EC/2022).

Informed consent

All participants provided written informed consent prior to participation in the study.

Footnotes

Acknowledgments

The authors express their sincere gratitude to the National Institute of Health (NIH) for granting permission to adapt the Computer Workstation Ergonomics: Self-Assessment Checklist.

Conflict of interest

The authors declare no conflicts of interest related to this research study.

Funding

This work was supported by a grant from the Directorate of Research of Universitas Gadjah Mada through the “Rekognisi Tugas Akhir Batch II” program (No. 5722/UN1.P.III/Dit-Lit/PT.01.05/2022).

The English and Indonesian versions of the Computer Workstation Ergonomics: Self-Assessment Checklist are available from .

References

Ryan

, Lewis

Computer and internet use in the United States: 2016. US Department of Commerce, Economics and Statistics Administration, US . . . ; 2017.

Basilaia

, Kvavadze

. Transition to online education in schools during a SARS-CoV-2 coronavirus (COVID-19) pandemic in Georgia, Pedagogical Research 2020;5(5)

Wiguna

, Anindyajati

, Kaligis

, Ismail

, Minayati

, Hanafi

, et al. Brief research report on adolescent mental well-being and school closures during the COVID-19 pandemic in Indonesia, Frontiers in Psychiatry 2020;11(November):1–9.

Ardiyanto

, Mulyadin

, Santi

, Dharma

IGBB

. Online classrooms during COVID-19 pandemic: A survey of industrial engineering instructors in Indonesia, Higher Learning Research Communications 2021;11(1):27–46.

Majumdar

, Biswas

, Sahu

. COVID-19 pandemic and lockdown: Cause of sleep disruption, depression, somatic pain, and increased screen exposure of office workers and students of India, Chronobiology International 2020;00(00):1191–200.

Bahkir

, Grandee

. Impact of the COVID-19 lockdown on digital device-related ocular health, Indian Journal of Ophthalmology 2020;68(11):2378.

Janwantanakul

, Pensri

, Jiamjarasrangsri

, Sinsongsook

. Prevalence of self-reported musculoskeletal symptoms among office workers, Occupational Medicine 2008;58(6):436–8.

Griffiths

, MacKey

, Adamson

, Pepper

. Prevalence and risk factors for musculoskeletal symptoms with computer based work across occupations, Work IOS Press 2012;42(4):533–41.

Kayabınar

, Kayabınara

, Onal

HYZ

, NK

The musculoskeletal problems and psychosocial status of teachers giving online education during the COVID-19 pandemic and preventive telerehabilitation for musculoskeletal problems

2021;68:33–43.

10.

Uchino

, Yokoi

, Uchino

, Dogru

, Kawashima

, Komuro

, et al. Prevalence of dry eye disease and its risk factors in visual display terminal users: The Osaka study, American Journal of Ophthalmology 2013;156(4):759–66.e1.

11.

Salinas-Toro

, Cartes

, Segovia

, Alonso

, Soberon

, Sepulveda

, et al. High frequency of digital eye strain and dry eye disease in teleworkers during the coronavirus disease pandemic, International Journal of Occupational Safety and Ergonomics 2021;28(3):1787–92.

12.

De Lima Machado

, Balbin Villaverde

. Design of an electronic instrumentation for measuring repetitive hand movements during computer use to help prevent work related upper extremity disorder, International Journal of Industrial Ergonomics 2011;41(1):1–9.

13.

Yadegaripour

, Hadadnezhad

, Abbasi

, Eftekhari

, Samani

. The effect of adjusting screen height and keyboard placement on neck and back discomfort, posture, and muscle activities during laptop work, International Journal of Human-Computer Interaction 2021;37(5):459–69.

14.

Davis

, Kotowski

, Daniel

, Gerding

, Naylor

, Syck

. The home office: Ergonomic lessons from the “New Normal, ” Ergonomics in Design: The Quarterly of Human Factors Applications 2020;28(4):4–10.

15.

David

. Ergonomic methods for assessing exposure to risk factors for work-related musculoskeletal disorders, Occupational Medicine 2005;55(3):190–9.

16.

, Buckle

. Current techniques for assessing physical exposure to work-related musculoskeletal risks, with emphasis on posture-based methods, Ergonomics 1999;42(5):674–95.

17.

Takala

, Pehkonen

, Forsman

, Hansson

GÅ

, Mathiassen

, Neumann

,et al. Systematic evaluation of observational methods assessing biomechanical exposures at work, Scandinavian Journal of Work, Environment & Health 2010;36(1):3–24.

18.

Janowitz

, Stern

, Morelli

, Vollowitz

, Hudes

, Rempel

. Validation and field testing of an ergonomic computer use checklist and guidebook, Proceedings of the Human Factors and Ergonomics Society Annual Meeting 2002;46(13):1007–11.

19.

Stanton

, Young

. Giving ergonomics away? The application of ergonomics methods by novices, Applied Ergonomics 2003;34(5):479–90.

20.

National Institute of Health, Office of Research Service, Division of Occupational Health, and Safety. Coputer Workstation Ergonomics: Self-Assessment Checklist [Internet]. 2017 [cited 2017 Jun 1]. Available from: https://ors.od.nih.gov/sr/dohs/Documents/checklistergonomics-computer-workstation-self-assessment.pdf

21.

Beaton

, Bombardier

, Guillemin

, Ferraz

. Guidelines for the process of cross-cultural adaptation of self-report measures, Spine 2000;25(24):3186–91.

22.

Noviyanti

, Santoso

, Hadi Putra

. A Cross-Cultural Adaptation of Chatbot Usability Questionnaire (CUQ): Indonesian Version, 4th International Conference on Information and Communications Technology (ICOIACT) 2021)248–51.

23.

Sharfina

, Santoso

Sharfina Z, Santoso HB. An Indonesian adaptation of the System Usability Scale (SUS). In: 2016 International Conference on Advanced Computer Science and Information Systems (ICACSIS). IEEE; 2016, pp. 145-8.

24.

Erdinc

, Hot

, Ozkaya

. Turkish version of the Cornell Musculoskeletal Discomfort Questionnaire: Cross-cultural adaptation and validation, Work 2011;39(3):251–60.

25.

Alfatiyah

. Penilaian postur kerja dengan SNQ (Standart Nordic Quistionnare) dan metode rosa (Rapid Office Strain Assessment) pada instansi susanto education, Jurnal Teknik Industri 2020;10(1):78–84.

26.

Septiawati

, Hasyim

, Najmah . Faktor risiko ergonomi saat mengetik dan hubungannya dengan carpal tunnel syndrome, Jurnal Ilmu Kesehatan Masyarakat 2013;4(3):237–44.

27.

Murni

Adaptasi Bahasa Indonesia dari Computer Workstation Ergonomics: Self-Assessment Checklist [Master’s thesis]. Universitas Gadjah Mada; 2022.

28.

Abbas

, Aman

, Nurunnabi

, Bano

. The impact of social media on learning behavior for sustainable education: Evidence of students from selected universities in Pakistan, Sustainability 2019;11(6):1683.

29.

Bartram

, Berberoglu

, Grégoire

, Hambleton

, Muniz

, van de Vijver

. ITC guidelines for translating and adapting tests (Second Edition), International Journal of Testing 2018;18(2):101–34.

30.

Zaman

, Muhyiddin

, Nugraha

Analisis keberterimaan istilah-istilah komputer: Studi kasus sosiolinguistik untuk para teknisi komputer. Prosiding Seminar Nasional Linguistik dan Sastra (SEMANTIKS) 2020. 2020;19-26.

31.

Setiawan

. English code switching in indonesian language, Universal Journal of Educational Research 2016;4(7):1545–52.

32.

, Villarreal

, Pouliot

. Pictograms: Can they help patients recall medication safety instructions? Visible Language 2016;50(1):127–51.

33.

Meppelink

, Smit

, Buurman

, van Weert

JCM

. Should we be afraid of simple messages? The effects of text difficulty and illustrations in people with low or high health literacy, Health Communication 2015;30(12):1181–9.

34.

AWY

, Chan

AHS

, Ho

VWS

Comprehension by older people of medication information with or without supplementary pharmaceutical pictograms, Applied Ergonomics 2017;58:167–75.

35.

Dominguez

, Balkrishnan

, Ellzey

, Pandya

. Melasma in Latina patients: Cross-cultural adaptation and validation of a quality-of-life questionnaire in Spanish language, Journal of the American Academy of Dermatology 2006;55(1):59–66.

36.

Green

, Cheydleur

, Li

, Deng (Jo)

, Sanchez

. Free smartphone apps to support human factors/ergonomics work, Ergonomics in Design: The Quarterly of Human Factors Applications 2018;26(1):4–16.

37.

Sellen

, Harper

RHR

The myth of the paperless office. MIT Press; 2003.

38.

Brown

, Hicks

, Rinaudo

, Burch

The use of augmented reality and virtual reality in ergonomic applications for education, aviation, and maintenance. Ergonomics in Design: The Quarterly of Human Factors Applications. 2021;106480462110034.

39.

Spielholz

, Bao

, Howard

, Silverstein

, Fan

, Smith

, et al. Reliability and validity assessment of the hand activity level threshold limit value and strain index using expert ratings of mono-task jobs, Journal of Occupational and Environmental Hygiene 2008;5(4):250–7.