Abstract
In interventional clinical trials for persons with spinal cord injury (SCI), the influence of experimental biological, pharmacological, or device-related interventions must be differentiated from that of physical and occupational therapy interventions, as rehabilitation influences motor-related outcomes. The International Spinal Cord Injury (ISCI) Physical Therapy
Introduction
Pre-clinical evidence in animal models of spinal cord injury (SCI) indicates that physical activity, practice, and training influence the motor outcomes of randomized intervention studies.1–4 In humans with SCI, direct assessment of the efficacy of standard rehabilitation via randomized comparison of outcomes among patients with SCI who do and do not receive rehabilitation is precluded by ethical considerations. However, a systematic review of experimental interventions for persons with SCI concluded that clinical trials that include a physical therapy (PT) and/or occupational therapy (OT) component are more likely to have meaningful effects on motor-related outcomes. 5
As rehabilitation influences motor-related outcomes, in interventional clinical trials, the influence of the experimental biological, pharmacological, or device-related intervention must be differentiated from that of physical and occupational therapy interventions. The International Spinal Cord Injury Physical Therapy–Occupational Therapy Basic Data Set (ISCI PT-OT BDS, v.1.2) was developed with the intent to track the content and time of rehabilitation interventions that are delivered concurrently with experimental interventions. 6 The ISCI PT-OT BDS enables standardized documentation of PT and OT interventions delivered in the clinical setting as part of a controlled clinical trial that is intended to improve voluntary motor function. PT and OT interventions are assigned to seven categories, of which five are activity-directed (bed/seated activities; standing activities; walking, stairs [inside, outside]; gross motor upper extremity [UE]; fine motor UE), and two are impairment-directed (strength training and/or electrical stimulation administered to increase strength; endurance training and/or electrical stimulation administered to increase endurance).
When the ISCI PT-OT BDS is used to document therapy interventions that are part of a clinical trial, the individual documenting the content of a therapy intervention may be a PT and/or an OT. Differences in training and clinical emphases between the disciplines of PT and OT may result in different perceptions of the intervention being delivered to a patient. Likewise, differences in perceptions about an intervention may be attributable to therapist years of experience working with individuals with SCI.
Apart from therapist-related factors, there are factors related to the patient whose treatment session is being documented that may account for differences in therapist perception of the intervention. The content of PT and OT sessions is likely to be largely determined by the degree of impairment of the individual with SCI. Different types of intervention are appropriate based on whether the individual being treated has motor-complete tetraplegia, motor-incomplete tetraplegia, motor-complete paraplegia, or motor-incomplete paraplegia. As such, the way the content and time of the therapy session are documented may be influenced by the degree of impairment of the individual whose therapy session is being documented.
Differences in the setting in which a patient with SCI is being treated may result in differences in what is documented in inpatient versus outpatient settings. Differences between U.S.-based versus non-U.S.-based therapists in what content is emphasized during therapy may be a source of differences in documentation of therapy content. Finally, diferences in the types of interventions that are done, in the equipment, and in the amount of time allocated to treatment must also be considered.
The purpose of this study was to determine whether the ISCI PT-OT BDS could be used to reliably capture information about the content of a therapy session. To address this question, we assessed the inter-rater reliability of information collected by pairs of therapists who had been trained to use the ISCI PT-OT BDS based on instructions in the syllabus. We assessed the level of agreement between therapist pairs in terms of therapy category selection and time spent in each category. The documentation was intended to reflect interventions delivered to a patient with SCI during a regularly scheduled treatment session.
Methods
To the extent possible, the methods adhered to the recommendations of the ISCI Data Set Committee. 7 The ISCI PT-OT BDS work group developed the study syllabus (ISCI PT-OT Basic Data Set syllabus v1.2) 6 and recruited 10 SCI centers in 7 countries (4 in the United States and 1 each in Denmark, Germany, Ireland, Norway, Switzerland, and the United Kingdom) for participation in the study. All centers met the ethics requirements of their respective country/institution before the initiation of study activities at their respective center. Each participating center verified that their center offered the full range of interventions described in the BDS. Each center had one site leader serving as the liaison to the work group. The site leaders recruited PTs and OTs who had experience in the rehabilitation of persons with SCI at their respective center to participate in the study. Several online sessions were held to review the study syllabus with the site leaders and therapists and to clarify any areas of uncertainty about the syllabus.
At each center, data were recorded during the course of regularly scheduled therapy sessions with 20 different patients with SCI. All patients were at least 18 years of age, had injury etiology of either traumatic or nontraumatic SCI of any postinjury time point, and were receiving therapy in either the inpatient or the outpatient setting. Each participating center endeavored to provide data that reflected the spectrum of SCI level and severity, including from individuals with motor-complete tetraplegia (American Spinal Injury Association Impairment Scale [AIS] A & B), motor-incomplete tetraplegia (AIS C & D), motor-compete paraplegia (AIS A & B), or motor-incomplete paraplegia. 8 To ensure that the data collected reflected the broadest possible range of characteristics of individuals with SCI, each individual with SCI could be observed only once. To be included in the data analysis, the observed intervention session had to be at least 30 min in duration.
Therapist raters were either PTs or OTs, with any level of experience in SCI rehabilitation. Each participating site had at least three therapists who contributed data, and centers were encouraged to have both PTs and OTs participate in the study. For each therapy session in which data were collected, therapists worked in pairs comprising a treating therapist and an observing therapist; pairs could comprise PT/OT, PT/PT, or OT/OT. It was deemed acceptable to have a single therapist be the observer for all observations, so long as there are at least two other treating therapists.
Data were collected using the ISCI PT-OT BDS v.1.2, the treating therapists recorded the intervention category and amount of time they believed was spent in the activity- or impairment-directed intervention when they transitioned from one intervention to another. If an intervention qualified to be considered in both an activity-directed and an impairment-directed category, then therapists were instructed to score it only in the activity-directed category. Therapists reported time spent on a specific intervention by 15-min intervals (<15 min, 15–29 min, 30–44 min, 45–59 min, or ≥ 60 min). The observing therapist observed the entire session, using a timer to time the interventions, and recorded the time spent in the categories he/she considered appropriate. The treating therapist and observing therapist were blinded to each other’s entries.
Participating centers were provided with a formatted electronic data workbook (Excel) in which to enter their center’s data for analysis. Each site also sent electronic (scanned) versions of the paper data collection forms, and members of the work group performed random fidelity checks for quality control on 10% of the records based on a random assignment.
In addition to information about the therapy session, the participating therapists indicated on the data collection form information about the following: (1) whether they were the treating therapist or observer; (2) their professional discipline (PT, OT); (3) their years of experience in working with individuals with SCI (<3 years, ≥3 years); (4) degree of impairment of the individual with SCI who is receiving the intervention (motor-incomplete tetraplegia, motor-complete tetraplegia, motor-incomplete paraplegia, motor-incomplete paraplegia); (5) therapy setting (inpatient, outpatient); and (6) and unique center code (to identify center as U.S.-based vs. non-U.S.-based in analysis).
Data analyses
Agreement between observing and treating therapists was assessed on a categorical level and a time level. If both therapists provided any amount of time for a category, then they were deemed to have agreed that the category occurred during the session. Agreement regarding time was characterized as providing either the same time interval or one time interval difference for a specific intervention. For example, instances in which the treating therapist selected 15–29 min for an intervention while the observing therapist selected 30–44 min, the therapists are considered to be in agreement. Disagreement was determined by any instances in which therapists indicated a difference of two or more time intervals for any given intervention. If a therapist did not report a time interval for a specific intervention during the therapy session being assessed, and the other therapist did report a time interval for a specific intervention during that therapy session, then this was deemed to be a disagreement. Instances in which neither therapist recorded a specific intervention were not included in the agreement analyses to avoid overinflation of agreement percentage.
Percent agreement and intraclass correlation coefficients (ICCs) were calculated to assess agreement among therapists across interventions for intervention time intervals. Percent agreement of 75% or greater was considered high, 50–75% was considered medium, and less than 50% was considered low. ICC values were interpreted as follows: ICC ≥ 0.90 indicating excellent agreement, 0.75 ≤ ICC < 0.90 indicating good agreement, 0.50 ≤ ICC < 0.75 indicating moderate agreement, and ICC < 0.5 indicating poor agreement. 9 One-way random-effects models were used to account for the study consisting of multiple sets of raters across sites.
Differences between treating and observing therapists in terms of time intervals for interventions were investigated using independent sample t-tests. For the purpose of this analysis, time intervals were treated as continuous data with 1 = <15 min, 2 = 15–29 min, and so on. Response proportion, the proportion of instances in which the treating or observing therapist reported an intervention, was also assessed. Instances where a therapist reported any amount of time were coded as reporting that the intervention occurred, regardless of duration.
Percent agreement and Pearson chi-square analyses were conducted to examine differences between therapist pairs because of discipline (PT, OT), years of experience (<3 years, ≥3 years), degree of impairment (motor-complete tetraplegia, motor-incomplete tetraplegia, motor-complete paraplegia, motor-incomplete paraplegia), therapy setting (inpatient, outpatient), and global location of the center (U.S.-based, non-U.S.-based).
Results
At each of the 10 centers, 20 therapy sessions were observed by pairs of therapists resulting in a total of 400 observations. Fidelity checks revealed errors in the data of two centers, prompting a full check of data from all centers. Any errors identified in the transfer of data from the data collection forms to the electronic forms were corrected.
Therapist and patient characteristics
Of the 400 observations, 68.5% were conducted by PTs, whereas 31.5% were conducted by OTs. The PT-OT providers averaged 7.7 years of experience (standard deviation [SD] = 8.46). Among the patients being observed, 41% had motor-incomplete tetraplegia, 13% had motor-complete tetraplegia, 26.5% had motor-incomplete paraplegia, and 19.5% had motor complete paraplegia.
Agreement on intervention category and time
As displayed in Table 1, the category agreement between the treating and observing therapist pairs was high (75% or greater) for five of the seven intervention categories. The exceptions were gross UE and endurance training, in which agreement between the treating and the observing pairs was medium (≥50% and <75%). Time agreement between the treating and observing therapist pairs was high for three of the seven intervention categories, including bed/seated control activities, walking/stairs, and strength training. The exceptions were standing control, gross UE, fine motor UE, and endurance training, in which percent agreement between the treating and the observing pairs was medium. ICC estimates demonstrated moderate-to-good reliability for all intervention categories with the exceptions of gross UE training and endurance training.
Intervention (A–E Activity Based; F–G Impairment Based) Agreement
Number of data entries for which at least one rater provided data for duration of intervention.
Run as a one-way random-effects model.
CI, confidence interval; ICC, intraclass correlation coefficient.
Treating and observing therapist interactions
As shown in Table 2, for six of the seven intervention categories, there were no significant differences between the treating and the observing therapists in the percentage of instances that a specific category was selected. However, the treating and observing therapists did differ in the percentage of instances where endurance training category was selected (25% vs. 15%, for the treating and observing therapists, respectively). Relative to the treating therapist, the observing therapists selected the category of endurance training significantly less often. Reported time spent on a category did not differ between the treating and the observing therapists for any of the seven categories.
Treating and Observing Therapist Interactions
SD, standard deviation.
Influence of therapist, patient, and center characteristics
In addition to assessing agreement between therapists on the intervention, differences between the treating and the observing therapists with respect to discipline, years of experience, location of the center, setting, and patient level/severity of injury were also examined. As shown in Table 3, no differences were identified for these variables regardless of whether the pair consisted of two PTs, two OTs, or a PT/OT.
Influence of Therapist, Patient, and Center Characteristics
OT, occupational therapist; PT, physical therapist.
Discussion
The intent of this study was to validate the ISCI PT-OT BDS by assessing agreement between pairs of therapists using the ISCI PT-OT BDS to classify category and time of interventions applied during usual clinical practice. In actual use, it is anticipated that the treating PT or OT who is administering therapy would be the one to document the content and time (dose) of therapy. Having an observing therapist also complete the form was a study design strategy to allow comparison of the documentation by two trained therapists based on a single episode of therapy to assess reliability of the form. Therapists were trained to use the paper-based PT-OT BDS form according to the information provided in the syllabus. For future applications, an Android-based app is available.
Agreement on intervention category and time
The findings of this study support the reliability of the ISCI PT-OT BDS based on percentage agreement on category between pairs of therapists documenting information about the same session. There was high agreement between the treating and the observing therapists for five of the seven intervention categories and medium agreement for the remaining 2 categories (Table 1). For the activity-directed interventions of bed/seated control, standing control, walking/stairs, fine motor UE interventions, and for the impairment-directed strength training interventions, there was agreement of over 75%. However, the agreement between therapist pairs for the activity-directed intervention of gross motor UE training, and impairment-directed category of endurance training did not meet this threshold, with medium agreement of 66% and 50%, respectively.
Documentation of time spent in an intervention had less agreement between the treating and the observing therapist pairs than for category. The likely reason for this is that the observing therapist was instructed to time the intervention with a stopwatch, whereas the treating therapist was not given this instruction. Agreement on time was high for three of the seven intervention categories and medium for four of the seven categories. However, it is valuable to note that for two interventions that achieved only medium agreement the agreement was near 75% (standing, 74.3% and fine motor, 73.2%). Within each category, the close agreement between the percentages for category and time indicates that there were few instances in which there was no agreement on the category and time. Accordingly, the reliability between therapist pairs as indicated by ICC values was moderate to good for all interventions with the exception of gross UE training and endurance training, for which ICC values indicated poor reliability.
In the syllabus and in the training sessions, instructions indicated that if an intervention could logically fall into both an activity-related category and an impairment-directed category, then the therapist should document that intervention in the activity-related category only. As the treating therapist is deciding on the intervention based on an intent that is known to him or her, but not known to the observing therapist, it is possible that the observing therapist defaulted to selecting an activity-directed goal when unsure.
Treating and observing therapist interactions
The treating therapist and the observing therapist selected the same intervention category for the majority of interventions (Table 2). The exception was the endurance training category, for which the observing therapist selected the category significantly less often than the treating therapist. Conversely, the observing therapist chose the bed/seated control activities, gross UE, and strength categories more often than the treating therapist. Although these differences were not significant, it may suggest that these categories are difficult to differentiate for an observer who is not aware of the treating therapist’s intent. It is possible that observing therapists were more likely to document an intervention as strength training when the treating therapist viewed the intervention as endurance training. Alternatively, it is possible that rather than selecting the endurance training category, the observing therapists defaulted to selection of an activity-related category (bed mobility, gross UE training) based on the instructions to select the activity-directed category if an intervention could reasonably be classified as being either activity-directed or impairment-directed.
Influence of therapist, patient, and center characteristics
We investigated factors that could result in differences between the treating and the observing therapist in documentation using the ISCI PT-OT BDS. These factors included therapist characteristics (professional discipline and years of experience working with individuals with SCI), degree of impairment (tetraplegia/paraplegia, complete/incomplete), therapy setting (inpatient, outpatient), and whether the center was a U.S.-based versus a non-U.S. center. The finding of no difference in the intervention documentation because of any of the factors analyzed indicates that the BDS is reliable for use among both PTs and OTs with different amounts of experience, with patients with SCI across all injury levels and severities, in both inpatient and outpatient therapy settings, and across international SCI centers.
For the purposes of this study, in which we were interested in evaluating the reliability of items in the BDS, the study procedure involved both a treating therapist and an observing therapist. However, in real-world implementation, although there may be instances where there is value in an observer completing the ISCI PT-OT BDS to document the content of a therapy session, in most instances, the expectation is that the treating therapist will be the one to complete the documentation.
Lessons learned
A key advantage of the ISCI PT-OT BDS is its simplicity compared with other available taxonomies.10–12 In this study, the actual time for each of the components of the therapy was documented by the observing therapist. Differences in time between the observing and the treating therapists indicate that in real-world use, it will be important for the treating therapist to time each component of the therapy session. During the course of the study, questions arose from therapists, which indicated additional details in the syllabus or the inclusion of a FAQ section would improve the guidance for clinicians. For example, the information on the documentation related to power wheelchair use, and additional guidance related to differentiating walking versus standing would be valuable.
Limitations
It is expected that in actual use, the ISCI PT-OT BDS will be completed by the treating therapist. For this reason, assessing the reliability of the BDS via an observing therapist was the only reasonable approach to answer the question about whether the BDS could be used to reliably capture information about the content of a therapy session. Differences in agreement are therefore attributable to the fact that the treating therapist knew the intent of each component of the interventions within a session while the observing therapist could only surmise based on observation. Conversely, the observing therapist timed the various components within a session with a stopwatch, while the treating therapist did not. When there was agreement between therapists, it was not possible to confirm that they were scoring the same period of the therapy session. We did not create video recordings of the sessions, and so it was not possible to review how specific categories were scored.
Conclusions
There was generally moderate-to-strong agreement between the treating and observing therapists recording content of therapy sessions using the ISCI-PT-OT-BDS v.1.2. The BDS is reliable for use across settings, countries, and with patients of various impairment levels.
Rigor and Reproducibility
The study was designed based on methods recommended by the International Spinal Cord Injury Data Set Committee. 7 All participating therapists engaged in an instructional session to standardize the data collection process and to ask any questions they may have had. The treating and observing therapists, whose data were being compared, were blinded to each other’s entries. The data were entered into an electronic database at each of the 10 participating centers. Members of the work group performed random fidelity checks for quality control on 10% of the records based on a random assignment, in which electronic data entries were compared with information entered on the data collection forms.
Footnotes
Acknowledgments
The authors are grateful to all the PTs and OTs who participated as treating or observing therapists for this study and to all the patients who consented to have their therapy session observed. They especially thank Ryan Lovell, PT, DPT, NCS, for his contributions to data collection and article development.
Authors’ Contributions
Conceptualization—E.C.F.-F., K.D.A., R.R., L.J., M.S.-R., V.K.N., A.B., S.M., and F.B.-S. Ideas—E.C.F.-F., K.D.A., R.R., L.J., M.S.-R., V.K.N., A.B., S.M., and F.B.-S. Methodology—E.C.F.-F., K.D.A., R.R., L.J., M.S.-R., V.K.N., A.B., S.M., and F.B.-S. Programming—M.C. and R.R. Validation—E.C.F.-F., K.D.A., R.R., L.J., M.S.-R., V.K.N., A.B., S.M., and F.B.-S. Formal analysis—M.C., R.R., and L.J. Investigation—M.W., H.H.P., V.J., E.S., and M.P. Resources—M.C. Data curation—M.C. Writing—original draft—E.C.F.-F. Writing—reviewing and editing—E.C.F.-F., K.D.A., R.R., L.J., M.S.-R., V.K.N., A.B., W.W., and F.B.-S. Visualization—M.C. Supervision—E.C.F.-F., K.D.A., and F.B.-S. Project administration—E.C.F.-F. and K.D.A.
Author Disclosure Statement
The authors have no competing interest to disclose.
Funding Information
No funding was provided for this research.