Abstract
BACKGROUND:
The importance to be able to work is recognised as one of the United Nation’s Sustainable Development Goals. Occupational therapists working in vocational rehabilitation require cost effective, valid, and reliable tests that offer flexibility during the work evaluation process. The use of self-developed work samples standardized using Modular Arrangement of Predetermined Time Standards (MODAPTS) as an objective measure of work speed could meet this need.
OBJECTIVE:
Evaluate the face, content, and criterion validity of MODAPTS work samples to assess work speed.
METHODS:
We conducted a quantitative cross-sectional, descriptive study. Occupational therapists completed electronic surveys to evaluate face and content validity. We evaluated criterion validity during a multiphase process that involved a realist synthesis, online survey, and comparing MODAPTS to Methods Time Management (MTM) time standards using three work samples that measured work speed.
RESULTS:
MODAPTS had good face validity for measuring work speed. We also confirmed the content and criterion validity of codes used to analyse basic movement and handling of smaller and larger articles as well as other body actions.
CONCLUSION:
MODAPTS demonstrated adequate face, content and criterion validity of tasks involving basic movements, handling of articles and other body actions.
Introduction
Work is one of the most significant occupations in which adults engage [1]. Work contributes meaningfully to financial stability, self-efficacy, and a sense of purpose [1, 2]. The importance of work is further recognised by the United Nations as one of the Sustainable Development Goals [2]. Not being able to work causes financial hardship and can also change a person’s self-image and influence their role in society [1]. In occupational therapy, vocational rehabilitation specifically aims to improve the employability of people who have limitations in performing work [3]. An important aspect of vocational rehabilitation includes assessing work ability and determining the best fit between worker and the work to which they are suited [4]. Vocational assessments, or functional capacity assessments, assess whether people are able to meet the physical, psychosocial and cognitive demands of a task and if they are able to work with adequate quality and speed, contributing to workforce productivity.
Work assessments can be used to determine rehabilitation goals, a person’s readiness, and ability to return to work, as well as whether they are entitled to compensation [5, 6]. A person’s ability or inability to work impacts various stakeholders including employers, insurers, families, and social security agencies [1]. Work assessments are usually conducted by occupational therapists, who rely on various tools and assessment strategies to accurately predict work performance and safety in performing work.
Work assessments can be performed using either bottom-up or top-down approaches [7]. Work assessments can also follow a bi-directional approach [7]. Bottom-up work assessments generally assess body structure and function to predict a functional outcome and describe impairment [8]. Bottom-up approaches have, however, been cautioned against, since their predictive value remains uncertain [9]. Functionally, many people who return to work report persistent disability or re-injury, which calls into question the ability of bottom-up assessments to accurately predict work readiness [1]. Alternatively, top-down assessments measure or predict performance, and have been promoted as delivering more accurate work assessments [10]. Top-down assessments simulate real-life situations and do not rely on novel and abstract tasks to assess work ability, which aligns with the core philosophy of occupational therapy [10]. Top-down or performance-based assessments usually rely on work samples as they can be used to observe a client’s performance in a specific task as opposed to evaluating impairments on a body function or body structure level. Work samples, such as those developed using Modular Arrangement of Predetermined Time Standards (MODAPTS) are short tasks that depict an aspect of the client’s work and are usually more physical in nature. They have a well-defined start and stop point and may represent the actual work requirements or simulate them. Work samples have a time standard for the task to be completed and therapists also make observations of the client’s behaviour and task engagement as a part of the assessment[11, 12].
MODAPTS work samples are used in combination with other assessments in work assessments and is indicated when the evaluation of speed of performance is necessary. Occupational therapists can also observe the influence of psychosocial and environmental factors on work ability during a work sample [1]. The use of work samples allows clients to use procedural knowledge and facilitates better task performance than when having to perform novel and abstract tasks that are usually included in neuropsychological assessments [10].
Work samples have been shown to adequately predict work performance and are able to predict safety when performing a physically demanding job [13]. Work samples are usually developed using Predetermined Time Standards (PTS), which represent a reasonable time for a worker to complete a given task [14, 15]. The PTS can be used as a benchmark both in a prospective workplace as well as during rehabilitation. In a resource constrained country MODAPTS as a PTS can be used to standardise a work sample to be used as part of a work assessment.
A variety of PTS tools are available including MODAPTS, Methods Time Measurement (MTM), Work Factor and Master Standard Data (MSD). PTS are used successfully in engineering, production and manufacturing, and in the construction industry [14, 16].
Initially, MODAPTS was developed by Chris Heyde, an Australian engineer, to determine production standards and measure ergonomic outcomes [14]. MODAPTS is widely used in production research [17, 18], to determine production time and aid in design and planning of ergonomic processes [19]. MODAPTS can also be used to estimate safety and quality, without the task having to exist [19]. A published MODAPTS manual contains codes and code sets which are applicable to most manual and clerical tasks. The manual also describes layouts and actions clearly, thus attempting to control as many variables as possible [20]. MODAPTS works on the assumption that all body movements can be described as a multiple of the time that it takes to move a single finger. This time is 0.129 seconds and represents one MOD. Using the manual occupational therapists are able to analyse a task using an alphanumeric code and allocate a standard time to complete the task. For instance, moving one’s arm to pick up a pen on your desk requires moving through a specified distance, keeping your elbow relatively still. This will be classified as an M3. To grasp the pen a G1 code will be used as the pen meets the criteria for a simple grasp of a raised object. Reaching for and grasping your pen would therefore be described using M3G1. This is then translated to four MODS, which is multiplied by the standard of 0.129 seconds. The standard time for this task is therefore 0.52 seconds. Using MODAPTS to analyse tasks in terms of each movement, occupational therapists are able to analyse a variety of tasks to determine its reasonable time standard. It is therefore possible to choose a task that is relevant to the client, the job and the context. Nearly any task can be analysed, with the exception of tasks that require higher cognitive skills, reasoning, abstraction, planning or where complex decisions have to be made. Since MODAPTS is criterion referenced it is generalizable and not based on a single population.
Occupational therapists initially used MTM to develop work samples for rehabilitation clients [21]. MTM is considered to be complicated PTS with many codes. The MTM led to the development of the Valpar Component Work Samples (VCWS), which include a set of standardized work samples [22]. The VCWS have been widely used in occupational therapy to evaluate both abled bodied individuals and persons with disability [22]. Although VCWS is a criterion referenced work assessment method and can be applied to people living outside of America [5], the work samples are imported and expensive which limits their utility in resource poor settings. Most of the VCWS are not portable, which further lessens their clinical utility.
Once MODAPTS was introduced, it soon became known for its simplicity and ease of learning [12, 21]. MODAPTS was introduced in South Africa in the early 1980s by Judith Farrell, an Australian occupational therapist and was adopted as a cost and time effective scientific approach to work assessment, especially of manual or clerical tasks [5]. Since MODAPTS was introduced into South African occupational therapy education and training programmes, it has been used to varying extents in clinical practice as an assessment of client work speed [5]. Although some South African Higher Education Institutions provide undergraduate and postgraduate training in MODAPTS, very limited research has been conducted to describe the application or appropriateness of MODAPTS to assess work speed as an indicator of functional ability. In contrast MODAPTS has been widely researched as a PTS in the field of engineering and has been found valid and reliable [15, 19]. It is to date being used by international companies to set production standards.
Some therapists have used MODAPTS to measure change during intervention and research [23], however MODAPTS has not been validated for use in occupational therapy contexts as an assessment of work speed. Although MODAPTS has been recommended for use in vocational rehabilitation, some occupational therapists seem to be hesitant in adopting MODAPTS as a routine tool for conducting work assessments [24].
To establish why therapists are hesitant to adopt MODAPTS as a tool for conducting work assessments it is necessary to determine the face, content and criterion validity. Face validity assesses whether a test subjectively appears to test what it aims to test. Tests with low face validity are perceived as irrelevant, which may skew results and impact on effort [28]. Face validity is determined by the collective opinion of experts [25, 28]. In this study, we determined the face, content- and criterion validity of MODAPTS as tool for assessing work speed.
Methodology
Research design
We used a quantitative cross-sectional, descriptive study design [25]. The research process comprised the three aspects of validity, namely face, content and criterion validity, with criterion validity consisting of three phases (Fig. 1). Ethical approval was granted by the University of Pretoria with ethics number 146/2017. The research was conducted between July 2017 and December 2017.
Research overview.
Figure 1: Overview of the research process to assess face, content and criterion validity of MODAPTS for use in vocational rehabilitation.
Participants and sampling
To assess face validity, we identified occupational therapists working in the private and public health sectors in South Africa. We used total sampling due to an expected low response rate. Eligibility criteria included: occupational therapists who were registered with the Health Professions Council of South Africa, who received undergraduate or postgraduate training in MODAPTS, including the development of MODPATS work samples, and who were working in the field of vocational rehabilitation. We sent emails to all occupational therapists on the Occupational Therapy Association of South Africa’s database. We also used snowball sampling to include other therapists who met the inclusion criteria.
Data collection instrument
Qualtrics, an electronic survey tool was used to develop and distribute an online questionnaire. The questionnaire included a cover letter and informed consent and if consent was not provided, the questionnaire closed. This was followed by questions related to the eligibility criteria and only when the required criteria were met, did the rest of the questions related to face validity open. Table 1 provides the questions included related to face validity. The questionnaire was piloted and critiqued by an expert in vocational rehabilitation, an occupational therapist working in academia and a research expert. Questions were in the form of statements with a four-point Likert scale to measure participants’ level of agreement. Responses were electronically selected, and data were extracted into an Excel sheet.
Perception of face validity of MODAPTS among occupational therapist working in the field of vocational rehabilitation
Perception of face validity of MODAPTS among occupational therapist working in the field of vocational rehabilitation
For face validity, we evaluated therapists’ agreement using the Lynn method with a confidence interval of 78%. Cronbach’s alpha was used to describe internal consistency [26, 27].
Content validity
Participants and sampling
Content validity was assessed by stringently selected participants to ensure familiarity with MODAPTS. We invited six participants who were considered experts in MODAPTS to participate. The eligibility criteria included the entire list described under face validity and additionally: working in the field of vocational rehabilitation or engaged in tertiary education in the field of vocational rehabilitation and have experience in providing training in MODAPTS and work assessments or have published on thesubject.
Data collection instrument
The same process described under face validity was followed with different questions included that related to content validity. The specific questions related to content validity are described in Table 2.
Content validity of MODAPTS by expert vocational rehabilitation occupational therapists
Content validity of MODAPTS by expert vocational rehabilitation occupational therapists
Content validity was established using a confidence interval of 100% to account for small sample size [26]. Cronbach’s alpha was used to describe internal consistency [26, 27].
Criterion validity
Overview
We measured criterion validity in three phases (Fig. 1). The first phase was a realist synthesis conducted to determine the content of work assessments and the constructs involved in assessing work ability. Phase two involved developing a list of VCWS which are considered relevant in work assessments and determining the content validity of this list. Phase three involved analysing these tasks using MODAPTS and comparing the standard times developed by MODAPTS to that determined by MTM in the specific tasks.
Realist synthesis
We performed the review using an iterative search. The review focussed on specific questions, namely, the purpose of work assessments, selection criteria for work assessments and the qualities of a well-designed work assessment. We reviewed 56 articles and included 20 articles in our final review. The review was conducted in July 2017 – August 2018. From these results, we developed a list of VCWS that are relevant to work assessments.
Phase 2: Identifying relevant tasks to analyse
Participants and sampling
Experts in work assessments were purposively identified and invited to evaluate the content of the list of tasks developed based on the realist synthesis review results. The eligibility criteria included: occupational therapists registered with the Health Professions Council of South Africa, working in the field of vocational rehabilitation, having had undergraduate or postgraduate training in the conduction of work assessments that involved a moderated work assessment performed as part of training and at least three years’ experience in the field of vocational rehabilitation and having published or presented at a national or international conference within the field of vocational rehabilitation.
Data collection instrument
The same process described for face and content validity was used to disseminate an electronic survey that evaluated the content validity of the list of VCWS developed.
Data analysis
The experts’ responses were analysed using the Lynn Method to validate the list of articles retrieved during the realist synthesis review using a confidence interval of 100% to account for a small sample size [26]. This method was specifically utilised as the validity for applying it using a 100% confidence interval to finite samples have been proven [26].
Phase 3: Analysing tasks using MODAPTS and comparing MODAPTS standard time to MTM standard time.
MODAPTS analysis: The relevant tasks identified in phase 2 were analysed using MODAPTS. This involved analysing all movements required to perform the tasks and allocating the appropriate alphanumeric codes to each movement. A standard time for each task to be completed in was determined. The analyses were moderated by three independent experts to ensure rigor.
Data analysis
A deterministic model was used to compare the standard times obtained for specific tasks using MODAPTS and MTM. This was done under the guidance of a biostatistician.
Results
Face validity
Fifty-seven respondents completed the face validity questionnaire of which 40 questionnaires met the inclusion criteria. Most respondents indicated more than 15 years of occupational therapy experience, while 5% of respondents had fewer than two years’ experience.
Table 1: Perceptions of occupational therapists working in vocational rehabilitation on the face validity of MODAPTS.
Most (95%) of the respondents were confident that MODAPTS could assess work speed and felt that MODAPTS would be useful in clinical settings. Seventy percent of participating occupational therapists reported that they found it difficult to develop work samples using MODAPTS. The majority of participants (97.5%) reported that administering developed MODAPTS samples was easy. Occupational therapists seemed to be more comfortable with administering MODAPTS samples than with developing new samples. Reported reasons for reluctance to develop work samples included a lack of confidence, developing new work samples was too time consuming and too complicated.
Content validity
Three out of five purposively selected participants returned questionnaires. The confidence interval was adjusted to 100% to account for small sample size (n < 6) [26]. All experts agreed that the code classes involving basic movements, body movements and large movements would be able to measure work speed (Table 2). However only two experts agreed that the clerical codes and codes for mental operations would adequately measure work speed if the equipment used for clerical work is considered and basic mental operations are measured. Internal consistency was confirmed by the Cronbach’s alpha of 0.92 where > 0.7 was considered acceptable.
Criterion validity
Phase 1: Realist synthesis
The realist synthesis aimed to answer specific questions related to the application and construct of work assessments. After data extraction was performed, articles were grouped according to the relevant topics and themes covered. The articles included in the review (n = 20) are summarised in Table 3 with a description of the themes covered in the articles.
Summarises the characteristics of the documents included in the review (n = 20) after the inclusion criteria was applied
Summarises the characteristics of the documents included in the review (n = 20) after the inclusion criteria was applied
Most articles identified the Dictionary of Occupational Titles as the theoretical basis for developing functional capacity evaluation batteries. The Dictionary of Occupational Titles is an American developed electronic database with a wide variety of job descriptions that span across industries and levels of employment. It has not been recently reviewed; however, it remains one of the frequently used tools when analysing jobs in amongst South African occupational therapists in the absence of other classifications. It also describes twenty core physical functions related to work, such as walking, bending or lifting. The frequency at which these functions occur is used to determine the classification of the job i.e., sedentary, or medium for instance. Most of the articles described initial test development and validation, with few articles presenting evidence of replicating results or generalising results with different populations. Most articles described work assessments of physical capacity and there was less research associated with the cognitive and psychosocial aspects of work assessment, although these were identified as important aspects to be evaluated in terms of work ability.
Phase 2: Developing a list of VCWS for inclusion in a vocational rehabilitation MODAPTS
The DOT lists 20 physical demands that are generally included in most work assessments as indicated in the articles retrieved in the realist synthesis review. We generated a list of VCWS based on these 20 physical demands. The content validity of the list was determined by the collective opinion of experts in vocational rehabilitation. VCWS assessing work speed included: VCWS 9 – a work sample that evaluates whole body range of motion and dexterity. The client is asked to transfer wooden shapes from one panel to another and requires reaching, grasping, placing, bending and crouching as well as loosening and fastening of nuts. VCWS 4 – a work sample that evaluates dexterity and upper limb range of motion. The client is asked to screw and unscrew nuts of different sizes onto bolts inside a box at various angles. VCWS 6 – a work sample that evaluates problem solving. A client looks at and compares shapes and colours presented on a master card and test booklet. The client has to identify and indicate discrepancies between the master card and test booklet.
Phase 3: Comparing MODAPTS and MTM time standards
We analysed the selected tasks using MODAPTS in the manner described in the introduction. Three experts with the required qualification and experience in MODAPTS (See inclusion criteria) participated in a moderation session to scrutinize the accuracy of the MODAPTS analysis. Once the MODAPTS analyses were validated, the time standards were compared to the MTM time standards given in the VCWS manuals. According to the MODAPTS manual, a qualified worker can be described as a person who is familiar with the task, knows what is expected and has the adequate experience and attributes to complete the task. Based on this definition, the MODAPTS time standards were compared to MTM time standards that would be used to evaluate a client’s first attempt. (Figs. 2–4).
MODAPTS standard time compared with MTM standard time for VCWS 4.
MODAPTS standard time compared with MTM standard time for VCWS 9.
MODAPTS standard time compared with MTM standard time for VCWS 6.
Figure 2: MODAPTS time standard compared with MTM time standard for VCWS 4.
Figure 3: MODAPTS time standard compared with MTM time standard for VCWS 9.
Figure 4: MODAPTS time standard compared with MTM time standard for VCWS 6.
As demonstrated in Figs. 2–4, MODAPTS time standards compared favourably with MTM time standards for VCWS 4 and VCWS 9, but not for VCWS 6.
In this study, we determined the face, content, and criterion validity of MODAPTS to assess work speed for vocational rehabilitation. Our results suggest that MODAPTS is valid for assessing work speed in vocational rehabilitation, especially for measuring manual dexterity and range of motion.
Face validity
In our study, occupational therapists agreed that MODAPTS had adequate face validity as an instrument to assess work speed but reported that developing MODAPTS work samples are not user friendly and confirms the view of Van Biljon et al. [24]. Occupational therapists agreed that once developed, work samples were easy to administer and useful. In our study, 42.5% of participants indicated that they administered MODAPTS samples weekly. Occupational therapists may be reticent to administer MODAPTS if they lack confidence, which may be exacerbated if the manual instructions are unclear or there is limited undergraduate training in the use of MODAPTS.
South African occupational therapists also often attend certification training in specific FCE batteries [22] and may hesitate to use MODAPTS if there are few continued professional development courses available. The courses may be expensive, especially if they are presented by international companies. In our study, occupational therapists were willing to use MODAPTS and it is likely that they would also be willing to attend MODAPTS courses. Occupational therapists may also use MODAPTS more frequently if the manual is updated for clarity, and if new equipment is designed to align with technological advances.
Content validity
Our results confirmed content validity for basic movements, gets and puts, body movements and large movements. We could not confirm the content validity of MODAPTS for measuring work speed associated with clerical tasks and mental operations. MODAPTS is ideally suited to measuring physical tasks because it was developed for the production industry in a pre-computerised era. Since then the design of equipment, tools and technology have changed significantly. Mental function is often measured in a different manner such as writing or typing being measured as words per minute, which makes it difficult for occupational therapists to select cognitive assessments that measure cognitive and psychosocial skills [29, 30]. Although many occupational therapists advise performance-based assessments for measuring mental function, it is difficult to predict time standards for higher order cognitive tasks [6, 10]. Where physical limitations readily manifest in psychomotor slowing and decreased work speed, the impact of physical injury or disability on clerical tasks is harder to enumerate. Vocational therapists may need to consider the importance of speed in executive functioning, and whether qualitative observations and sound clinical reasoning carry more weight than work speed.
Criterion validity
We compared the time standards for MODAPTS and MTM for the same tasks and found that VCWS 4 and VCWS 9 were similar indicating good criterion validity. These tasks were analysed using codes that describe/ involve basic movements, gets and puts that describes physical and dexterous tasks. Compared to VCWS 4 and 9 which measure physical skills, VCWS 6 measures mental or cognitive function and involves decision making, error recognition and simulates administrative type work. In our study, MTM and MODAPTS returned different time standards for VCWS 6. This supports our content validity results which indicate that MODAPTS in its current form cannot adequately assess work speed in clerical tasks and cognitive tasks.
Limitations
Since this is the first study to assess the face, content, and criterion validity of MODAPTS to assess work speed, we could not compare our results to previous studies that focus on occupational therapy practice. The obtained results cannot be generalized for the South African context due to the small number of registered occupational therapists that are trained and who are actively using MODAPTS in their practice. Our study focussed only on three work samples that assessed work speed, which means that our findings cannot be generalized to the whole MODAPTS suite. We evaluated the validity of MODAPTS codes, which does not necessarily confirm the validity of developed work samples that therapists may have developed. These samples may have errors in the MODAPTS analysis, and we recommend that work samples be developed and moderated by a team of therapists as opposed to one therapist working alone. An in-depth understanding of why occupational therapists agreed or disagreed was not obtained based on the study’s research design which did not include qualitative data.
Conclusions and recommendations
We evaluated the validity of MODAPTS work samples to assess work speed. Face validity was established according to the collective opinion of South African occupational therapists. We confirmed content and criterion validity for physical and manual tasks but not for clerical and mental operations. The use of MODAPTS work samples in various settings, including vocational rehabilitation, requires further validation. Due to the nature of data, a deterministic model of mathematical equations was used to evaluate criterion validity. Expanding on this study, client testing would enable further statistical analyses. Future research should evaluate the validity and reliability of specific MODAPTS work samples and a variety of tasks. We suggest that that a MODAPTS task group in South Africa start to develop and disseminate work samples to be used across sectors in vocational rehabilitation.
Footnotes
Acknowledgments
The authors thank Dr. Cheryl Tosh for editing and Prof B Girdler-Brown for statistical support and analysis.
Ethical approval
Ethical approval was granted by the University of Pretoria (ethics number 146/2017).
Conflict of interest
There are no conflicts of interests to be declared.
Funding
The research was not funded.
Informed consent
Written informed consent was obtained from all participants.