Abstract
PURPOSE:
This study aimed to investigate the internal consistency, inter-rater, and test-retest reliability of the balance subtest of the Bruininks-Oseretsky test of motor proficiency-second edition (BOT-2) and to estimate the minimum detectable change (MDC) and minimum clinically important difference (MCID) of the balance subtest of the BOT-2 in children with cerebral palsy (CP).
METHODS:
In total, 20 children with CP participated in the present study. Internal consistency, test-retest, and inter-rater reliability were computed to establish reliability of the balance subtest of the BOT-2. The MDC95 was estimated from the standard error of measurement (SEM) to determine a real change for an individual child with CP. The anchor- and distribution-based MCID were calculated to determine the smallest change that might be important to clinicians. For concurrent validity, the correlation between the balance subtest of the BOT-2 and the pediatric balance scale (PBS) were calculated using Spearman’s correlation.
RESULTS:
Internal consistency was good (Cronbach’s alpha coefficient = 0.89). The BOT-2 had excellent test-retest (ICC = 0.99, p < 0.001) and inter-rater reliability (ICC = 0.99, p < 0.001). The balance subtest of the BOT-2 had an SEM of 0.70, MDC95 of 9.61, and MCIDs of 2.54 (anchor-based) and 1.38 (distribution-based). Additionally, there was a moderate positive correlation between the balance subtest of the BOT-2 and the PBS (Spearman’s rho = 0.629, p = 0.003).
CONCLUSIONS:
Our experimental results indicate that the balance subtest of the BOT-2 had good internal consistency, along with excellent test-retest and inter-rater reliability. The change in scores of an individual child with CP should attain 9.61 points on the balance subtest of the BOT-2 to indicate a clinically important change. The MDC95 and MCID values could be helpful in understanding therapeutic effects and evaluating balancing ability using the balance subtest of the BOT-2 in children with CP.
Keywords
Introduction
Balance is achieved by integrating the vestibular, somatosensory, and visual systems [1]. It is essential for performing functional skills and recreational activities [2]. Due to poor balance and associated anticipatory postural adjustment deficit, children with cerebral palsy (CP) are likely to experience difficulty with activities of daily living such as standing and walking [3]. Previous studies have demonstrated that children with CP have lower balancing ability than typically developing (TD) children [4, 5]. Children with CP demonstrate lower anterior and posterior single-stepping thresholds compared to TD children, indicating limitations in balance-reaction capabilities [6].
The Bruininks-Oseretsky test of motor proficiency-second edition (BOT-2) was developed to measure gross and fine motor skills. It is composed of 53 test items under 8 subtests including fine motor control, manual dexterity, bilateral coordination, balance, running speed, and strength. The balance subtest of the BOT-2 contains 9 sub-items. A raw score is computed from seconds or steps and used to derive a point score from 0–5 for each item [7]. The BOT-2 has shown moderate to high inter-rater and test-retest reliability in healthy children in the United States [7]. Wuang and Su [8] reported that test-retest reliability (intraclass correlation coefficient [ICC] = 0.90), inter-rater reliability (ICC = 0.92), and internal consistency (Cronbach’s alpha coefficient = 0.92) were excellent in children with intellectual disability. In addition, Wolter et al. [9] found that children with cochlear implant (CI) failure demonstrated lower balance subtest scores on the BOT-2 (7.8 points) than children who did not experience CI failure (12.2 points). Although previous studies have used the BOT-2 to assess the balance ability of children with CP [10, 11], reliability of the balance subtest of the BOT-2 in children with CP has not been studied.
The minimum detectable change (MDC) is the smallest amount of change or difference that is not the result of measurement error, or the smallest amount of change that can be detected by a measure that corresponds to a noticeable change in ability [12]. Another measure of clinical relevance is the minimum clinically important difference (MCID), which is the smallest amount of change in an outcome that might be considered important by the patient or clinician [13]. The MDC and MCID can support effective clinical decision-making [14, 15]. Investigation of the MDC and MCID of the balance subtest of the BOT-2 is necessary because it is a frequently used outcome measure in pediatric rehabilitation. The purpose of this study was to: (1) investigate the internal consistency, inter-rater, and test-retest reliability of the balance subtest of the BOT-2 in children with CP, (2) estimate the MDC and MCID for the balance subtest of the BOT-2 in children with CP, and (3) determine the concurrent validity between the balance subtest of the BOT-2 and the pediatric balance scale (PBS).
Methods
Participants
A sample size of 12 was estimated based on an effect size of 0.67, a 1-β error (power) of 0.80 (80%), and an α level of 0.05 (G-Power version 3.1 software [University of Dusseldorf, Dusseldorf, Germany]). Therefore, we recruited 20 children with CP (11 boys and 9 girls) from pediatric rehabilitation units in Gyeongsangbuk-do. Inclusion criteria for children were: (1) aged between 4 and 12 years, (2) classified at Gross Motor Function Classification System (GMFCS) level I or II, (3) able to stand without an assistive device for more than one minute, (4) able to understand instructions, and (5) participating in a physical rehabilitation program at the time of study. The exclusion criteria for children were: (1) having uncontrolled seizures, and (2) having visual and auditory impairments. The demographic and clinical characteristics of children are summarized in Table 1. This experimental protocol was approved by the Ethics Committee for Human Studies (2017-01-012). Informed consent was obtained from all parents or caregivers before the study.
Demographic and clinical characteristics of the children (N = 20)
Demographic and clinical characteristics of the children (N = 20)
CP: cerebral palsy; SD: standard deviation.
Each child performed the balance subtest of the BOT-2 and PBS at baseline (T1), one week later (T2), and four weeks later (T3). Two raters (pediatric physical therapists with more than 6 years of experience) were trained to administer and score the balance subtests of the BOT-2 and the PBS. The BOT-2 and PBS were measured on the same day, and the test order was determined by a random allocation software (Isfahan University of Medical Sciences, Isfahan, Iran). To examine test-retest reliability, the same rater administered both balance tests twice within a week. For inter-rater reliability, a video of the children performing was recorded and then scored by the second rater. The first assessment data were used to determine inter-rater reliability. To estimate the MDC and MCID, comparison scores by two raters between T1 and T3 (after four weeks) were used. No children dropped out of this study.
The balance subtest of the BOT-2
The nine items of the balance subtest of the BOT-2 are: standing with feet apart on a line, walking forward on a line, standing on one leg on a line, walking forward heel-to-toe on a line, standing on one leg on a balance beam, and standing heel-to-toe on a balance beam. The speed and quality of test items were measured by time to complete item (seconds) and number of steps. The maximum raw score attainable per item was 10 seconds or 6 steps. A 5 or 6 (0–4 or 0–5) point ordinal scale was used to score individual balance items [7].
PBS
The PBS is a criterion-referenced test that measures functional balance skills in the pediatric population. The PBS was assessed using the following items: sit to stand and stand to sit, transfer, sitting and standing unsupported, standing with eyes closed, standing with feet together, standing with one foot in front, one leg standing, turning 360 degrees, turning to look behind, retrieving an object from the floor, stepping on a stool, and reaching forward. Each item was scored on an ordinal scale (0 = cannot perform, 4 = can perform the task completely). Previous studies confirm that test-retest and inter-rater reliability of the PBS are excellent (ICC > 0.90) [16]. The PBS scores were used to determine the anchor-based MCID of the balance subtest of the BOT-2 in this study. If the child’s PBS score had improved by 3.66 points (6.5%) after treatment compared to before treatment, the child was classified into the group with MCID [15].
Statistical analysis
All statistical analyses were performed using PASW Statistics 18 (Norusis/SPSS Inc., Chicago, IL, USA). The significance threshold was set at 0.05. For internal consistency of the BOT-2, Cronbach’s alpha coefficient was computed based on Bujang’s study [17]. Generally, coefficients above 0.7 are considered to be acceptable, above 0.8 good, over 0.9 excellent [18]. To establish test-retest and inter-rater reliability, the ICC was calculated [19]. Generally, coefficients between 0.50 and 0.75 are moderate, between 0.75 and 0.90 good, above 0.90 excellent [20]. Spearman’s correlation analysis was used to determine the concurrent validity between the balance subtest of the BOT-2 and PBS. Spearman’s coefficient between 0.20 and 0.39 are considered as weak, between 0.40 and 0.69 moderate, between 0.70 and 0.89 strong, over 0.90 very strong [21].
MDC
The MDC at 95% confidence interval (MDC95) was estimated using the follow formula:
The z score was based on 95% confidence interval (z = 1.96), SD is the standard deviation, and r is the coefficient of the test-retest reliability [22]. If a child has change scores equal to or over the MDC95 threshold, the changes are indicative of a change in ability rather than measurement errors.
MCID
The MCID of the balance subtest of the BOT-2 was determined using two methods [23]. First, the anchor-based MCID was computed as the mean change score on the balance subtest of the BOT-2, equivalent to children who were defined as having MCID (PBS change scores of 3.66 points) [15]. Second, distribution-based MCID was estimated with the Cohen’s effect size [24]. An effect size of 0.2 indicated a significantly important change and was used to determine the MCID threshold in this study [25].
Results
Reliability
Internal consistency for the balance subtest of the BOT-2 was good (Cronbach’s alpha coefficient = 0.89). The test-retest reliability of the balance subtest of the BOT-2 was excellent (the ICC was 0.99, p < 0.001) (Table 2) and the inter-rater reliability of the balance subtest of the BOT-2 was also 0.99 (p < 0.001) (Table 3).
Test-retest reliability of two balance tests
Test-retest reliability of two balance tests
T1: baseline; T2: one week later; T3: four weeks later; ICC: intraclass correlation coefficient; CI: confidence interval; PBS: pediatric balance scale; BOT-2: the balance subtest of Bruininks-Oseretsky test of motor proficiency-second edition.
Inter-rater reliability and concurrent validity of two balance tests
ICC: intraclass correlation coefficient; CI: confidence interval; PBS: pediatric balance scale; BOT-2: the balance subtest of Bruininks-Oseretsky test of motor proficiency-second edition.
Results of the standard error of measurement (SEM), MDC95, and MCID values of the balance subtest of the BOT-2 are shown in Table 4. As calculated from the 11 children whose PBS change scores were equal to or greater than 3.66 points, the anchor-based MCID was 2.54. The findings indicated that 55% of children had clinically important changes that surpassed the anchor-based MCID of the balance subtest of the BOT-2. In addition, the distribution-based MCID (0.2 SD) for the balance subtest of the BOT-2 was 1.38 (Table 4).
MDC95 and MID of the balance subtest of BOT-2
MDC95 and MID of the balance subtest of BOT-2
SD: standard deviation; r: test-retest reliability; SRM: standardized response mean; SEM: standard error of measurement; MDC95: minimum detectable change; MID: minimum important difference.
With respect to the two balance tests, there was a moderate positive correlation between the balance subtest of the BOT-2 and the PBS (Spearman’s rho = 0.629, p = 0.003) (Table 3).
Discussion
To the best of the authors’ knowledge, this is the first study to investigate the reliability, MDC, and MCID of the balance subtest of the BOT-2 in children with CP. The present findings provide helpful standards for clinicians to interpret whether children’s change scores on the balance subtest of the BOT-2 after treatment could be interpreted as clinically important changes and to make clinical decisions with regard to the child’s progress.
The internal consistency (Cronbach’s alpha) of the balance subtest of the BOT-2 was 0.89. Wuang and Su [8] reported that internal consistency of the BOT-2 was 0.92 and internal consistency of the balance subtest of the BOT-2 was 0.85 in children with intellectual disability. The internal consistency (Cronbach’s alpha) of the balance subtest of the BOT-2 for children with CP was somewhat higher than the internal consistency for children with intellectual disability [8]. However, these findings were lower than in TD children (Cronbach’s alpha coefficient = 0.95) [7]. Test-retest reliability coefficient (ICC) was also 0.99, indicating excellent test-retest reliability. A previous study of test-retest reliability of the BOT-2 in children with intellectual disability showed that the balance subtest ranged from 0.98 to 0.99 [8]. This means that the balance subtest of the BOT-2 has excellent test-retest reliability in children with CP. The inter-rater coefficient (ICC) was 0.99. Since this study is the first trial to determine the inter-rater reliability of the balance subtest of the BOT-2, it is hard to compare with the results of previous studies. In the present study, inter-rater reliability was excellent, and in most cases the two rater’s score were exactly the same. The excellent inter-rater reliability probably occurs because raters provide clear instructions for tasks and when deciding to begin and stop timing. The standardized response mean (SRM) and the effect size (ES) were used to define group-level change. The values of the SRM and the ES for the balance subtest of the BOT-2 were 0.39 and 0.38, respectively. The ES was low because the difference between baseline and four weeks later was too small. This means that long-term intervention is required to improve balance and motor performance in children with CP.
The MDC95 and MCID were used to determine individual-level change. The MDC95, the anchor-based MCID, and the distribution-based MCID of the balance subtest of the BOT-2 were 9.61, 2.54, and 1.38. This suggests that if the difference scores of an individual child with CP between two measurements reaches 9.61 points on the balance subtest of the BOT-2, the clinicians could interpret the change in scores as true beyond measurement errors at the 95% confidence interval level. In clinical practice, the measurement errors for individual-level are higher than the group-level of the average of a group of children with CP [26]. In the present study, the MDC95 of the balance subtest of the BOT-2 of 9.61 used for an individual child with CP is higher than the SEM of 0.70 used for a group of children with CP. The MDC95 may be meaningful when less than or equal to the MCID in individual-level use [27]. However, some studies show that this may not always the case [28]. In the present study, the MDC95 of the balance subtest of the BOT-2 was 9.61, which is higher than the anchor-based MCID of 2.54 and the distribution-based MCID of 1.38. If the MDC95 exceeds the MCID, it is recommended to consider both values in clinical decision making.
The combined anchor- and distribution-based approaches were calculated to determine MCID for the balance subtest of the BOT-2. This method has been recommended by Yost [29]. Combining the anchor- and distribution-based approaches, the changes should be in the range of 2 (above distribution-based MCID) to 3 points (above anchor-based MCID) on the balance subtest of the BOT-2 to meet the demands for MCID. In the present study, the balance subtest of the BOT-2 improved from 20.35±7.01 points (pre-intervention) to 22.80±6.74 points (post-intervention). These findings showed that the mean difference of the balance subtest of the BOT-2 was 2.45 points, which exceeded the distribution-based MCID of 1.38. However, the mean difference of the balance subtest of the BOT-2 did not exceed the anchor-based MCID of 2.54. The mean change scores reached the values of MCID established in the present study, suggesting that the findings could be interpreted as clinically important.
A few limitations in the present study should be considered. First, the sample size was small for generalization. However, power analysis was conducted based on the results of a pilot study. A larger sample of children with CP is necessary to confirm the MDC and MCID of the balance subtest of the BOT-2. Second, the findings of this study may not be generalized to all children with CP as only children with GMFCS levels I-II were included in the present study.
Conclusions
The results indicate that the balance subtest of the BOT-2 had good internal consistency, along with excellent test-retest and inter-rater reliability. The change scores of an individual child with CP should reach 9.61 points on the balance subtest of the BOT-2 to indicate a meaningful change. The balance subtest of the BOT-2 is a reliable tool for measuring the balance of children with GMFCS levels I-II. In addition, the MDC95 and MCID values could be helpful in understanding therapeutic effects and evaluating balancing ability using the balance subtest of the BOT-2 in children with CP. Using the MDC95 and MCID values to compare responsiveness of outcome measures used is suggested for clinical decision-making.
Footnotes
Acknowledgments
We would like to thank the children, their parents, and their physical therapists for their involvement.
Conflict of interest
The authors have no conflict of interest or funding sources to report.