The Head Control Scale: Development,inter-rater reliability,and utility

2.1 Phase one

The focus of the initial phase of this project was to develop a scale for objective assessment of head control. The Head Control Scale (HCS) (Fig. 1) in its current form was based on the Clinical Rating Scale for Head Control (CRSHC) developed by an occupational therapist [26]. Chavan’s scale was developed with input from a pediatric physician and two occupational therapists with more than 10 years of experience. The scale is a 5-point ordinal scale for two positions (prone and supported sitting) and a 4-point ordinal scale for one position (supine) [26]. The CRSHC was used as an initial structure for a new scale and was done with the author’s permission.

OPEN IN VIEWER

The primary researchers for the current study are a physical therapist with over 13 years of experience and an occupational therapist with 20 years of experience, both in pediatric settings. The two authors used the original CRSHC along with a review of head control elements in other standardized assessments, literature and textbooks on the development of postural control, and personal clinical experience to develop a new version, renamed the Head Control Scale (HCS). In the HCS, descriptors for each rating were clarified and simplified, a section was added dedicated to head position during a pull to sit, and the four sections were rated on a common scale (0–4).

Next, a group of clinicians was recruited to review and discuss the HCS in a process similar to the Nominal Group Technique (NGT), a five step method designed to aid in group decision-making [27]. The NGT was used because it is time and cost effective, offers both qualitative and quantitative data, and has been shown to be useful in rehabilitation research [27, 28] as well as in the development of assessment tools [29]. The group consisted of 10 physical therapists, eight occupational therapists, and one speech therapist representing a variety of clinical settings (outpatient, acute, early intervention and school-based practice) and experience levels (ranged from one to 39 years of practice).

After a brief presentation by the primary researchers (NGT Step 1) [27], the group members were given the scale to review (NGT Step 2) [27], after which group members shared their ideas for improving the HCS (NGT Step 3) [27]. Group discussion proceeded to help clarify thoughts and ideas (NGT Step 4) [27]. True NGT has an additional step consisting of voting by the group [27]; this step was not done because the primary researchers made the decisions on how best to use the feedback and discussion to modify the HCS. Changes included adding brief instructions to the HCS form, adding a comment box for each section, and making minor changes to the wording of each rating’s description.

The Northern Arizona University Institutional Review Board reviewed the proposal and deemed the research project exempt. The second phase of the project was designed to assess the agreement of HCS scores among multiple raters, when a single subject was evaluated from a video. Informed written consent was obtained from the parent of the video subject as well as of the recruited raters. The child in the video is a three-year-old male with spastic quadriplegia CP, Gross Motor Classification System (GMFCS) Level V. The GMFCS is a standardized system to categorize levels of CP; levels range from the least severely impaired (I) to the most severely impaired (V) [30]. This child was purposefully selected to represent severe functional limitations as this is the population for which the HCS will most likely be used and for whom head control is more challenging to describe. The subject was recorded spending 30–40 seconds in each position of the HCS (prone, supine, pull to sit, and sitting). One researcher (PT) facilitated the movements (pull to sit, supported sitting) while the other researcher (OT) helped motivate the subject with auditory and visually appealing toys.

A convenience sample of therapists (PTs, OTs, and SLPs) was recruited through flyers sent to local clinicians. Video viewing and rating were done at a time and place of convenience for the raters and often occurred in groups; the video was shown on a large television screen or computer, but all ratings were completed independently. No formal training was provided, though brief instructions were given to participants prior to video viewing and rating. A total of 20 raters viewed the video and completed the HCS.

A high percent of agreement among raters can indicate good interrater reliability [31]. As reported in the Results section below (Table 1), the high rate of agreement among the raters in Phase 2 using the HCS indicated to the researchers that the scale was reliable enough to proceed with Phase 3 of the study.

In addition to rating the subject, each participant was asked to complete a survey, the purpose of which was to collect demographic information and elicit feedback from participants. The survey consisted of multiple choice questions, statements rated on a Likert scale, and a comment box for free text (Appendix A). Feedback from the survey was used to make additional changes to the HCS, which included moving the instructions to the top of the page and editing the video for clarity.

Though the results of Phase 2 indicated a high percentage of agreement, this measure alone is limited in its usefulness because Phase 2 was based on a single subject (video) and does not account for the role of chance, which may provide an overestimation of reliability [31]. Therefore, Phase 3 was conducted in which 26 participants used the HCS to rate five different video-taped subjects; a description of the subjects can be found in Table 2. Video 1 was used in both Phases 2 and 3; all other videos were used only in Phase 3. The video-taped subjects were chosen to represent all the ranges of head control the HCS represents, from none to full control. Though three of the videos included children who were typically developing, these were chosen due to their age; at 10 days, 6 weeks, and 16 months, a wide range of head control ability was represented. Each subject in the video was recorded spending time in each of the four positions. Videos varied slightly in length, ranging from one minute thirty-nine seconds to two minutes, forty-seven seconds. The varied length of videos was due to each subject’s performance; those who were more mobile were less likely to spend much time in certain positions such as prone or supine. The number of videos made was limited due to the resources required to produce them.

A convenience sample of raters was again recruited by emails to local clinic owners. None of the raters from Phase 2 participated in Phase 3. A brief demogr-aphic/post-rating opinion survey was filled out by each participant. Demographic results from that survey are given in Table 3. Data collection occurred on-site at the participants’ clinic. No formal training was provided, though brief instructions were given, after which raters viewed the videos on a computer or television screen and rated them simultaneously, but individually. All participants had the same amount of time to view the video.

Average percent agreement across the 20 raters of the video used in Phase 2 was calculated for each HCS section (Prone, Supine, Pull to Sit, and Supported Sitting) and overall (the total score across the four sections). For rater data from Phase 3, in addition to average percent agreement, Fleiss’s weighted kappa coefficient [32] was estimated to assess evidence for rater agreement beyond what can be attributed to chance alone, for each HCS section score and overall. All analyses were carried out in the R statistical programming environment [33] using code provided by Gwet [34]. Calculations of average agreement and Fleiss’s weighted kappa were made using the ordinal weight option, appropriate to the measurement scale used in each HCS section in this study.

3.1 Phase two

Twenty raters were each asked to rate the single subject video on all four sections of the HCS in Phase 2, but some raters chose not to complete all four sections. For each section, the number of raters was at least 18. The average percent agreement across raters in scoring each section of the HCS is reported in Table 1. Average agreement exceeded 89% for each section, and overall was 98.1%. Because only one subject was rated in Phase 2, it is not possible to calculate kappa statistics for this data.

3.2 Phase three

Twenty-six study participants each rated the five subjects on the four sections of the HCS. For each HCS section and overall, Table 4 gives the median of the twenty-six participants’ scores for each subject and a summary of the agreement in scoring the five subjects across participants. For each HCS section, median scores across subjects range from zero to four. For each HCS section score and the overall score, the average percent agreement across the raters in Phase 3 was at least 90%.

Fleiss’s weighted kappa coefficient falls between - 1 and + 1, and measures the extent to which the observed agreement is greater than what can be expected by chance alone. A value greater than zero occurs if the observed agreement is better than chance alone, and a value less than zero indicates that the observed agreement is less than what would be expected by chance alone [34].

Landis and Koch [35] provide guidelines for interpreting the value of the kappa statistic. With the exception of HCS Supine, with a kappa of 0.68 – falling in the “substantial” agreement range – all of the remaining HCS sections (and overall) have a kappa coefficient that would be categorized as “almost perfect” agreement (kappa greater than 0.81).

The 95% confidence intervals reported in Table 4 for kappa show that for each HCS section, the set of plausible values for kappa for this study falls between “moderate” or “substantial” agreement to “almost perfect” agreement. The exception to this is Supine score where the lower end of the confidence interval is just below 0.40.

Results concerning the opinion questions in the post-rating survey are given in Table 5. Nearly all (24 out of 26, or 92%) Phase 3 participants answered either “Agree” or “Strongly Agree” to the statement “I feel there is a need for this scale in some practice settings.” For each of the two statements, “I feel that I could use this scale in my clinical practice,” and “I felt that the scale was easy to use,” a slightly lower percentage (22 out of 26, or 85%) of participants answered “Agree” or “Strongly Agree.” For each of these last two statements, there was one rater (out of the 26) who disagreed with the statement. Rater 7 was a speech therapist who did not feel the scale was applicable to her clinical practice. Rater 13 was an OT student who reported in the comment section that there was difficulty in distinguishing midline and flexion postures.