A comparative analysis of multidimensional computerized adaptive testing for the DASH and QuickDASH scores in Dupuytren’s disease

Introduction

Patient-reported outcome measures (PROMs) are important in hand surgery as they capture the patients’ perspective. Comprising 30 items, the Disabilities of the Arm, Shoulder and Hand (DASH) score has been reported as the most used PROM in hand surgery research (Lloyd-Hughes et al., 2019). An abbreviated (short form) version, known as the QuickDASH, was derived by Beaton et al. (2005). This was produced by concept-retention method, an item reduction approach, which resulted in an 11-item outcome measure that performed comparably on a psychometric basis with the full DASH score when measuring patient symptoms and disability.

In patients with Dupuytren’s disease, the DASH and QuickDASH measure two distinct domains, which might be interpreted as motor function and sensory symptoms (i.e. pain and paraesthesia) (Rodrigues et al., 2016; Stirling et al., 2021). The application of item response theory might improve the validity of DASH scores, while not changing the questionnaire items or response options themselves. Unlike classical psychometric test theory, item response theory uses probabilistic modelling (statistical equations) to assess the measurement properties of individual items, allowing scores from the questionnaire to be calibrated on a true continuous (rather than ordinal) scale. Furthermore, item response theory can quantify the reliability of the score for each individual, with a personalized confidence interval, based on a reliability statistic called standard error of measurement (SEm), which measures the spread of possible ‘true’ scores for a given test score, similarly to standard deviation. Finally, item response theory items function independently; even when participants answer different combinations of items from within the PROM, their scores can still be validly compared. This is the basis of computerized adaptive testing (CAT).

By adapting the testing to the individual patients, CAT uses algorithms to make PROMs shorter and more personalized. The algorithms administer an item, predict a person’s score after each item, then select the next best item for a person, based on their predicted score. This continues until a stopping rule is met, for example, a level of reliability has been reached.

Multidimensional CAT (MCAT) is a more advanced, and potentially more efficient form of CAT that uses multidimensional item response theory. In this theory, each questionnaire item can measure more than one health trait at a time, allowing data entry simultaneously on two different continuous scales. For example, in MCAT, items could measure both constructs (pain and function) on two different scales at the same time. This is more accurate and more valid than blending information about someone’s pain and function into one numerical value. This is particularly relevant to Dupuytren’s disease, which tends to impact motor function more than sensory symptoms.

The aim of this study was to develop an MCAT for the DASH in Dupuytren’s disease and use a simulation experiment to compare this to QuickDASH in terms of measurement reliability (SEm) and the number of questions asked.

Methods

Results

Data collection

Following listwise exclusion, 507 of the original 760 respondents were included in the analysis. Of the 253 excluded participants, 141 were excluded for failing to respond to item 21, which refers to sexual function in the original DASH questionnaire, and 112 were excluded for missing responses to other items. Demographics and treatment details are as summarized in Table 1.

OPEN IN VIEWER

Table 1.

Demographics and clinical details, where available, for included participants (n = 507).

Demographic variable	Value
Gender (number)
Male	431
Female	76
Age (years)
Median	67
IQR	61–73
Procedure (number)
Dermofasciectomy	88
Fasciectomy	315
Needle aponeurotomy	104

IQR: interquartile range.

Multidimensional item response theory modelling and MCAT simulation

The MCAT asked a mean of 8.51 items (SD 2.93) and as few as three items in some cases, 265/1000 simulated respondents needed to complete five items or fewer (Table 2). This compares to the QuickDASH short form where 11 items were administered in all cases.

OPEN IN VIEWER

Table 2.

Numbers of items posed by the multidimensioanl computerized adaptive testing algorithm before the stopping rule was reached for simulated individuals.

Number of items posed in MCAT	Number of simulated individuals posed that number of items
3	31
4	120
5	133
6	85
7	38
8	28
9	32
10	20
11	513

MCAT: multidimensional computerized adaptive testing.

The median SEms of QuickDASH were 0.320 for hand function and 0.290 for sensory symptoms, whereas the median SEms of DASH MCAT were 0.299 for hand function and 0.256 for sensory symptoms, indicating better precision than the QuickDASH.

Overall, we found that MCAT works most efficiently in patients with moderate–poor motor function and moderate–severe sensory symptoms, as illustrated in Figure 1. In our sample, the median logit score for hand function was 0.039 (IQR –0.954 to 0.859) and for sensory symptoms was 0.035 (IQR –0.833 to 0.547). These scores are lower than where MCAT is expected to have peak performance (Figure 1), meaning that MCAT is likely to perform even more efficiently in groups with more severe symptoms than were described by our cohort. OPEN IN VIEWER

Figure 1.

Multidimensional computerized adaptive testing (MCAT) efficiency at different test scores. The X and Z axes represent a person’s motor function score and sensory symptom score, respectively, while the Y axis represents the amount of information available to the MCAT for efficient decision making. In this figure, scores are measured on a continuous logit scale, with a higher score indicating a poorer clinical state. The information peak occurs at a medium–high score in both motor function and sensory symptoms, meaning that the MCAT works most efficiently in patients with moderate–poor motor function and moderate–severe sensory symptoms.

Discussion

In this study, we used multidimensional item response theory to develop a MCAT system to deploy the DASH in patients with Dupuytren’s disease. We achieved a way of collecting PROM information that was more structurally valid and more reliable (as reflected by lower SEm) than QuickDASH, even though fewer questions were often posed. By intelligently selecting the most relevant items for an individual, this system personalizes the assessment to the respondent. The precision achieved in our simulation (SEm <0.3) is comparable with that achieved by the Patient-Reported Outcomes Information System (PROMIS) measures, which are widely considered to have excellent psychometric properties (Hung et al., 2013), and equivalent to an over 90% reliability. If feasible for deployment in clinical practice, an MCAT system of modelling, like the one demonstrated in this study, could potentially harness the multidimensional nature of DASH to get more valid PROM measurements for patients with Dupuytren’s disease.

The multidimensionality of DASH has been criticized in the past, as blending sensory and motor scores into a single index of disability may be oversimplistic and introduce measurement error unnecessarily (Rodrigues et al., 2016). By using the MCAT system, the multidimensionality of DASH can be leveraged to administer the questionnaire more efficiently, while capturing two discrete scores. Given its greatly reduced burden for patients because of fewer questions that need answering, MCAT may also permit more frequent PROM sampling in research or clinical practice, thus providing deeper insights into day-to-day variations of symptom severity and functional impairment.

Differential analyses of the results from multidimensional PROMs like DASH and QuickDASH are particularly relevant to studying symptom severity and treatment effectiveness in a condition like Dupuytren’s disease. Dupuytren’s disease typically has a more severe impact on motor function than sensory symptoms (such as pain and paraesthesia) preoperatively. When Dupuytren’s disease is treated surgically, we might expect to see an improvement in hand motor function, but it is also possible that the patients can experience postoperative pain. In this scenario, if we ignore the multidimensional nature of DASH and instead simply blend pain and function scores together as is often the case, it is possible for the improvement in function and deterioration in pain to cancel each other out somewhat. A person’s scores may suggest no overall change has happened, when in fact two important changes have occurred – an improvement in function but a deterioration in pain. MCAT avoids this pitfall, providing more granular and actionable measurements.

In our study, we found that items 22, 23, 27 and 30 of the DASH capture information from both domains (Tables S1, S3, and S3 in the supplementary material). As such, we were able to sense check the data-driven suggestions that came from the analysis with clinical reasoning, for example, we deduct that items 22, 23 and 30 relate to social activities, work activities and confidence, respectively. Therefore, they are less explicit than the task-based or sensory symptom items and may reflect both sensory symptoms and impaired motor function. It is also possible that the word ‘weakness’ in item 27 was interpreted in different ways by different respondents. Qualitative interviews with affected patients could investigate these hypotheses in future.

Our study population had a relatively low degree of functional impairment (median logit score 0.039) and mild sensory symptom severity (median logit score 0.0352). As such, these patients will fall into the low-test information zone (Figure 1). Therefore, the algorithm may be functioning at a low level of efficiency in this cohort. In contrast, patients with conditions leading to ‘worse’ DASH scores, such as carpal tunnel syndrome, who may present with more severe sensory symptoms and higher degree of functional impairment, would have scores that are distributed at a higher level on both scales (i.e. closer to the zenith of the information curve, Figure 1). This means that our MCAT algorithm is likely to work even more efficiently for patients with conditions like carpal tunnel syndrome, although this remains to be proven.

While it might seem appealing to have allowed the MCAT algorithm to run with an SEm-based stopping rule only, so that more than 11 items could be presented, in this study we were specifically interested in the comparison with QuickDASH, and the broader question as to whether an item response theory-based questionnaire design is effective. We therefore elected to constrain the MCAT algorithm to a maximum of 11 items, as discussed in the methods.

Despite the several advantages to using an MCAT, there are challenges to its implementation in clinical practice. For one, the MCAT is limited by the need for a computer/mobile device to provide a score, whereas QuickDASH may be completed with pen and paper. However, in the post-COVID era, remote monitoring through electronic PROMs may be particularly appealing. The parameters we have presented in our supplementary material may easily be applied as a mobile application to enable patient-friendly, remote reporting of symptoms.

There are limitations in this study. It is assumed that the order of items does not affect item response (in our simulation, all item responses were predetermined), however, studies from other fields suggest the impact of this is minimal (Li et al., 2012). Also, our study population is limited to patients with Dupuytren’s disease who were treated in the UK, and the generalizability of our findings to other conditions and demographics remains unconfirmed.

In conclusion, by applying multidimensional item response theory to DASH responses from patients with Dupuytren’s disease, an MCAT can be developed that can be more accurate and less burdensome than QuickDASH. In clinical practice, this could take the form of a smartphone application that administers frequent, short and personalized versions of DASH to patients so that hand surgeons or physiotherapists might monitor them remotely. In research, MCAT might improve DASH or QuickDASH completion rates (by lowering the response burden of the questionnaire) and provide higher quality measurement in clinical trials (by accounting for the multidimensional nature of DASH and QuickDASH). Future work may test the acceptability of these tools, and their generalizability across other conditions and patient groups. Similar work could also apply contemporary psychometric techniques to other existing PROMs, and the scoring of tools originally developed without these methods might be updated to reanalyse existing datasets, for example from previous trials.

Supplemental Material

Supplemental Material