Validation of the Short Form Shoulder Instability Return to Sport After Injury (SIRSI-5) and Its Association With Return to Sports

Abstract

Background:

Psychosocial readiness is thought to be important in establishing whether athletes are ready to return to sports (RTS). The 5-item Shoulder Instability–Return to Sport After Injury (SIRSI-5) is a short-form version of the 12-item questionnaire (SIRSI-12), which was validated in a postoperative Argentinian population and associated with psychological readiness to RTS. It is unknown if the SIRSI-5 is valid in surgical and nonsurgical populations in other geographic locations.

Purpose:

To establish whether the SIRSI-5 is valid in another population and if it is associated with RTS.

Study Design:

Cross-sectional study (diagnosis); Level of evidence, 3.

Methods:

A total of 79 participants with anterior shoulder instability managed operatively and nonoperatively were included. The SIRSI-5 and SIRSI-12 total scores were calculated. Intraclass correlation coefficients (ICCs) and Pearson correlations assessed convergent validity between SIRSI-5 and SIRSI-12. Logistic regression was used to assess the discrimination ability (leave-one-out cross-validation [LOOCV]) of a preexisting SIRSI-5 cut point score (≥59/100) to determine participants who had returned to their preinjury level of sport. In addition, the discrimination ability (LOOCV) of multivariable machine learning models including age, sex, time since injury (months), SIRSI-5 (continuous score), surgery, and contact/noncontact sport to predict return to preinjury level sport was also assessed. The Youden index was calculated to assess the models’ performance.

Results:

Of the 79 participants, 62 (78.5%) were male and 27 (34%) had been treated surgically. Only 32% (n = 25) of participants had returned to preinjury level sport. The SIRSI-5 and SIRSI-12 were highly correlated (r > 0.9) with high levels of agreement (ICC, 0.90; 95% CI, 0.76-0.95). The prediction accuracy of the SIRSI-5 cut point score model was 66% (Youden index, 0; sensitivity, 4%; specificity, 94%). The best prediction accuracy obtained through a machine learning model (multilayer perceptron) was 72% (Youden index, 0.4; sensitivity, 64%; specificity, 76%).

Conclusion:

The SIRSI-5 has excellent convergent validity with the SIRSI-12 to measure psychological readiness to RTS and can be used clinically with less patient burden than the SIRSI-12. However, in a population of surgically and nonsurgically treated contact athletes, the SIRSI-5 with a cut point of ≥59/100 had low levels of prediction accuracy for RTS at preinjury level. The SIRSI-5 should not be utilized in isolation to determine readiness to RTS, and clinicians should consider other factors such as age, type of sport, and time since injury to underpin their clinical reasoning when deciding for RTS.

Keywords

shoulder instability return to sports validity psychological aspects of sport

Psychological readiness to return to sports (RTS) is one of several factors that is important in an athlete's RTS decision-making after an anterior shoulder instability event.^14,19 Rates of recurrent shoulder instability range from 36%¹³ to 100%.¹² Individuals with anterior shoulder instability return to play more quickly compared with those who have had an anterior cruciate ligament (ACL) reconstruction (ACLR) and play with higher levels of fear of reinjury.^14,19 Early RTS with high levels of fear may influence the rate of recurrent shoulder instability and is worthy of further exploration.

A patient-reported outcome (PRO) to measure psychological readiness to RTS after an ACLR was developed in 2008.²¹ The ACL–Return to Sport after Injury (ACL-RSI) scale has since been translated and validated in many languages,^3,10,18 has been shown to be associated with successful RTS after ACLR surgery, and was later revised to include a 6-item short version.²⁰ This scale has been adapted to measure psychological readiness to RTS after a shoulder injury.⁹ The 12-item Shoulder Instability–Return to Sport After Injury (SIRSI-12) scale is composed of 4 constructs: performance confidence, fear and reinjury risk, emotions, and rehabilitation/surgery. Higher scores indicate a more positive psychological response.¹⁴

While the use of PROs is important for clinicians, they can be burdensome for patients. This is particularly noticeable when there are several constructs that researchers are examining.¹⁵ Thus, the development of short form versions of PROs is thought to decrease participant burden.²⁰ However, these short form versions must be validated and examined for similar constructs as their longer-version counterparts.

Pasqualini et al¹⁵ developed a short form 5-item SIRSI (SIRSI-5). Patients who had shoulder stabilization surgery (open Latarjet, arthroscopic Bankart repair, or arthroscopic Bankart repair with remplissage) completed the scale ≥12 months after surgery. Items were reduced from the categories of confidence in sports performance (2 items), confidence in shoulder function, emotions and risk appraisal, and rehabilitation/surgery. The 6-month SIRSI-5 was shown to be predictive of RTS (cut point of ≥54) and return to preinjury level (cut point of ≥59) at 12 months. Divergent validity in a second population at the same center was established, and a significant difference was shown in SIRSI-5 scores between those who had returned to their preinjury level of sports and those who did not return.

The development of the SIRSI-5 is an important step in establishing a PRO that measures psychological readiness of athletes that is not burdensome. External validation and generalizability of the SIRSI-5 are required to establish if the SIRSI-5 is similarly valid in other countries and in athletes with shoulder instability who have not undergone surgery.¹⁵ Thus, the purpose of this study was to examine the construct validity of the SIRSI-5 compared with the SIRSI-12 and determine whether the SIRSI-5 was associated with RTS at preinjury level in a population in a different country that includes both operative and nonoperative patients. Several models were developed and tested to examine the relationship with RTS in this population. It was hypothesized that the SIRSI-5 would have convergent validity with the SIRSI-12 and be associated with RTS.

Methods

A cross-sectional study design was used. Ethical consent was granted by Auckland University of Technology.

Participants

Individuals who reported for physical therapy at Auckland Shoulder Clinic after a traumatic anterior shoulder dislocation were invited to take part in the study. Inclusion criteria were participants who had experienced a traumatic anterior shoulder dislocation, aged between 18 and 40 years, participated in sports on a weekly basis before their dislocation, and planned to return or had already returned to sports. Participants were excluded if they had a shoulder injury that was not a dislocation, had a shoulder dislocation ≥2 years ago, or did not comprehend written English. Informed consent was provided, and participants then completed a series of questionnaires and the SIRSI-12 either online or via a paper version. Data were collected between June 2015 and March 2020.

SIRSI Questionnaire

The SIRSI has been adapted from the ACL-RSI scale and has previously been reported.¹⁴

SIRSI-5

The SIRSI-5 questionnaire was developed in Argentina in a population who had undergone Latarjet or arthroscopic Bankart surgery.¹⁵ The SIRSI-12 was administered to competitive or recreational athletes, with a minimum follow-up period of 12 months. Factor analysis of the data was undertaken, and the scale reduced to 5 items with 3 of 8 items representing the construct of confidence in performance and 2 of 4 items representing the construct of emotions and risk appraisal. This was then validated in a second population.¹⁵ It is expected that the SIRSI-5 will take over 50% less time to complete than the SIRSI-12.

Statistical Analysis

Data analyses were performed in R Version 4.0.4 (RStudio) and Python 3.9 (Spyder). Validity was examined using intraclass correlation coefficients (ICCs) and 95% CIs (ICC, 2,1). Measurement error of the scores was determined by calculating the standard error of measurement (SEM).⁶ Convergent validity between the SIRSI-12 and SIRSI-5 was assessed using Pearson r correlations and ICC for absolute agreement. To provide a graphical representation of the agreement between the SIRSI-12 and SIRSI-5 scores, Bland-Altman plots were created.²

An a priori power calculation was performed to determine the sample size required to identify a large association (Cohen d effect size, 0.5) between SIRSI-12 and SIRSI-5 using Pearson r. Using G*Power software (Version 3.1.9.7),⁸ the alpha level was set to .05 and power to 0.80. The hypothesis was set to 2-tailed. With these parameters, the minimum sample size calculated was 29 participants.

After assessing the validity of SIRSI-5, a cut point score (≥59/100) was utilized to assess the ability of SIRSI-5 to determine which players had returned to preinjury sport versus those who had not through a simple logistic regression model. No future prediction but a simple classification attempt was made on cross-sectional data. The cut point score was chosen based on previously published work.¹⁵ In addition, a series of multivariable models were also developed and tested to determine their ability to correctly classify players who had returned to sports. The predictors utilized in the multivariable models were based on previous research and expert opinion. This included age, contact versus noncontact sport, history of surgery (yes/no), time from injury, sex, and SIRSI-5.

For the univariable logistic regression (SIRSI-5 cut point score), a leave-one-out approach was utilized to develop and test the model. Furthermore, only the original data were utilized during this analysis. For the multivariable models, a more sophisticated approach was taken. In particular, the lower-class category (RTS, n = 25) was oversampled through the Synthetic Minority Over-sampling Technique to provide a more balanced data set for model training (oversampled RTS, n = 54; no RTS, n = 54).⁴ After this process, all the predictors were scaled by subtracting individual values from the mean and dividing it by the standard deviation. Then the leave-one-out strategy (for the original data only; 79 iterations) was utilized to separate the test participant before training the multivariable model. A personalized approach to training the model was taken by selecting a training subset (n = 30) from the whole training data set. In particular, the personalized approach selects the cluster of training data with predictor values closest to the test participant via assessment of Euclidean distances in the multivariable space.⁷ Once the cluster of training data was identified, the multivariable model was trained and tested on the data point that was initially removed. The overall accuracy of the models was calculated by counting the number of test data that were correctly classified after training the model.

The steps above were repeated when implementing several machine learning models, which included multilayer perceptron, multiple linear regression, logistic regression, classification and regression tree, k-nearest neighbor, support vector machine, and random forest. The Youden index, which provides a single estimate of the performance of the model, was calculated for each model (Youden index = specificity + sensitivity – 1). A Youden index of 0 or 1 indicates none or perfect model discrimination, respectively.²² To provide a visual representation of the importance of the predictors in the multivariable models, a signal-to-noise ratio (SNR) is reported for the selected variables (SNR = absolute value (mean RTS – mean no RTS)/(SD RTS + SD no RTS).

Results

Patient Sample

A total of 80 participants were included in the present study. One participant was removed from the data set due to missing data, which left a total of 79 participants. The SIRSI-12 was completed by participants who presented for initial physical therapy treatment at a single physical therapy clinic irrespective of whether they had already returned to sports. Demographic and clinical characteristics of the participants are reported in Table 1.

Table 1

Participant Demographic and Clinical Characteristics (N = 79)^a

Variable	Value
Age, y	22.6 ± 5.8
Male	62 (79%)
Time since injury, mo	9.2 ± 8.5
Contact sport	59 (75%)
Surgical/nonsurgical First-time TASD/recurrent instability	27 (34%)/52 (66%) 35 (44%)/44 (56%)
SIRSI-12	47 ± 16
SIRSI-12 RTS	46 ± 15
SIRSI-12 no RTS	47 ± 16
SIRSI-5	51 ± 19
SIRSI-5 RTS	51 ± 17
SIRSI-5 no RTS	51 ± 20

Data are reported as mean ± SD or, for categorical variables, n (%). SIRSI-5, 5-item Shoulder Instability–Return to Sport After Injury; SIRSI-12, 12-item Shoulder Instability–Return to Sport After Injury; RTS, return to sports; TASD, traumatic anterior shoulder dislocation.

Validation of SIRSI-5

SIRSI-5 and SIRSI-12 had high levels of absolute agreement as shown by the ICC values in Table 2. The 2 measurements were also highly correlated (r = 0.93). A visual representation of the differences between SIRSI-12 and SIRSI-5 is provided in the Bland-Altman plots in Figure 1.

Table 2

SIRSI-12 and SIRSI-5 Descriptive Statistics and Validity^a

	SIRSI-12 (Mean ± SD)	SIRSI-5 (Mean ± SD)	ICC (95% CI)	SEM
Combined population	47 ± 16	51 ± 19	0.9 (0.76-0.95)	5.9
Surgical patients	44 ± 16	47 ± 20	0.9 (0.76-0.95)	5.9
Nonsurgical patients	49 ± 16	53 ± 19	0.9 (0.7-0.95)	5.8

ICC, intraclass correlation coefficient; SEM, standard error of the measurement; SIRSI-5, 5-item Shoulder Instability–Return to Sport After Injury; SIRSI-12, 12-item Shoulder Instability–Return to Sport After Injury.

Figure 1.

Bland-Altman plot to visualize differences in mean measurements between SIRSI-12 and SIRSI-5. Black horizontal line represents the mean difference between scores, and the red dashed lines the lower and upper confidence intervals of the mean difference. SIRSI-5, 5-item Shoulder Instability–Return to Sport After Injury; SIRSI-12, 12-item Shoulder Instability–Return to Sport After Injury.

Models to Categorize RTS

A visual representation of the importance of the predictors included to determine RTS is reported in Figure 2. When the logistic regression with a leave-one-out approach was applied to the original data set, a classification accuracy of 66% was achieved (Youden index, 0; sensitivity, 4%; specificity, 94%).

Figure 2.

Signal-to-noise ratio of the predictors included in the modeling. From left to right are the most and least important variables in discriminating between those who returned and those who did not return to sports, respectively. SIRSI-5, 5-item Shoulder Instability–Return to Sport After Injury.

When the multivariable approach for modeling was undertaken, it provided better classification accuracy of RTS. However, the overall results were still underwhelming, and the best accuracy was obtained through the multilayer perceptron (accuracy, 72%; Youden index, 0.4; sensitivity, 64%; specificity, 76%). All the results for the multivariable models are reported in Table 3.

Table 3

Machine Learning Models for Predicting Return to Sports to Preinjury Level^a

Model	No Return to Sports (n = 54)			Return to Sports (n = 25)			Overall Accuracy (N = 79)	Youden Index
Model	CC	IC	%	CC	IC	%	Overall Accuracy (N = 79)	Youden Index
MLP	41	13	76	16	9	64	72	0.4
KNN	33	21	61	14	11	56	59	0.17
MLR	39	15	72	11	14	44	63	0.16
RF	28	26	52	13	12	52	52	0.04
SVM	29	25	54	12	13	48	52	0.02
LOG	35	19	65	9	16	36	56	0.01
CART	36	18	67	8	17	32	56	-0.01

CART, classification and regression tree; CC, correctly classified; IC, incorrectly classified; KNN, k–nearest neighbor; LOG, logistic regression; MLP, multilayer perceptron; MLR, multiple linear regression; RF, random forest; SVM, support vector machine. Bold values indicate the best performing model.

Discussion

The SIRSI-5 is highly correlated with the SIRSI-12 and is a valid measure of psychological readiness to RTS after an episode of anterior shoulder instability. With a cut point of ≥59/100, replicated from a previous study,¹⁵ the SIRSI-5 had limited classification ability for RTS to preinjury level in a population of surgical and nonsurgical athletes. Thus, we are currently unable to predict RTS using the SIRSI-5 in our population. Overall, the SIRSI-5 has high levels of convergent validity with the SIRSI-12.

The SIRSI-5 was first validated in a surgical population of athletes 6 and 12 months after stabilization surgery in Buenos Aires.¹⁵ Similar to our findings, these authors reported that the SIRSI-5 is a valid short form measure of the initial SIRSI that was developed by Gerometta et al⁹ in a population of French rugby players. Two constructs of the SIRSI-12 were identified. The first was labeled confidence in performance and subcategorized as confidence in shoulder function and confidence in sports performance. The other construct was categorized as emotions and risk appraisal and included factors related to fear of sustaining an injury, as well as thoughts about surgery and rehabilitation. The constructs reported in Pasqualini et al¹⁵ are similar to those previously published.¹⁴

Pasqualini et al¹⁵ proposed a cut point of ≥54 for RTS and a cut point of ≥59 for readiness to RTS to preinjury level for the SIRSI-5. This was found to be significantly associated with RTS (Youden index, 0.5; sensitivity, 77%; specificity, 78%) and return to preinjury level (Youden index, 0.8; sensitivity, 88%; specificity, 92%). However, this cut point was not significantly associated with RTS in our New Zealand population. The population in this study differs from that of Pasqualini et al,¹⁵ in that it included both surgical and nonsurgical athletes recovering from a shoulder injury. The athletes in this study were primarily involved in contact or collision sport (75%), with only a few players involved in overhead sports, while 57% of patients in the Argentinian study were involved in contact sport. The mean ± SD scores in the New Zealand population of the SIRSI-12 (47 ± 16) and SIRSI-5 (51 ± 19) were lower than those reported in the Argentinian population (SIRSI-12, 62.4 ± 24.7; SIRSI-5, 65.7 ± 23.6). It may be that the increased percentage of contact and collision athletes in this study influenced the results.

Cut points for the SIRSI-12 were also proposed by Pasqualini et al.¹⁵ to determine an athlete's psychological readiness to RTS, a cut point of ≥52 was used to RTS (Youden index, 0.6; sensitivity, 77%; specificity, 84%), while for RTS at the preinjury level, a cut point of ≥55 (Youden index, 0.8; sensitivity, 88%; specificity, 92%) was proposed. This cut point level of ≥55 for the SIRSI-12 was used previously in a similar Argentinian population to determine an athlete's psychological readiness both to RTS and to RTS at the preinjury level (Youden index, 0.9; sensitivity, 100%; specificity, 90%).¹⁷

Additionally, in a French population of athletes who had undergone surgical stabilization following an instability event 6 months prior, the cut point level of 60.5 was reported (sensitivity, 39%; specificity, 87%) to discriminate athletes who were able to RTS.¹¹ Our questionnaire was administered at a mean of 9 months after injury, and it may be expected that a longer time after injury would result in higher SIRSI-5 scores. Thus, it appears that there is no single cut point value that is consistent across either the SIRSI-5 or the SIRSI-12. Clinicians should exercise caution when using an established cut value of either SIRSI when returning to sports and instead should incorporate the results of this questionnaire into a shared decision-making process with the athlete.

Given that we found no significant association between the cut point levels and RTS, we further examined whether a combination of variables may be more likely to be associated with RTS. We found that age, type of sport (contact or not), surgical or nonoperative management, time since injury, and sex were also important in detecting an association with RTS. Our analysis also showed that the importance of SIRSI-5 was limited when categorizing who had returned to sports. Therefore, we propose that these factors be combined with functional impairment measures (such as strength and range of motion) as well as physical performance tests when deciding whether to RTS. It appears that the SIRSI makes up just 1 component of the decision-making process, as outlined, in models that guide clinicians through the RTS decision-making process.^1,5,23

Limitations

This study presents with several limitations. This was a cross-sectional design rather than prospective. We did not predict future return to play but rather recorded SIRSI values on presentation to the clinic at various stages along the rehabilitation journey. Despite its being a limitation, this approach is like what other studies have attempted to achieve in the past.^11,16 Further prospective studies are required to determine the predictive validity of the SIRSI-5 when utilized as a sole predictor. Another limitation is due to the sample characteristics, which included a larger percentage of male athletes, as well as surgical and nonsurgical patients. Previous research that has shown predictive validity of the SIRSI-5 has been used solely in postsurgical patients.^11,15 Further exploration of the use of the SIRSI-12 and SIRSI-5 in nonsurgical patients with shoulder instability is required. Since this was a relatively small sample size for machine learning, it is likely to overestimate our ability to predict RTS. Even with this overestimation and inclusion of multiple variables, which have not been tested on an external subgroup of participants, our results continued to show our inability to accurately classify those who had successfully returned to sports.

Conclusion

Footnotes

Final revision submitted March 7, 2024; accepted April 4, 2024.

The authors declared that they have no conflicts of interest in the authorship and publication of this contribution. AOSSM checks author disclosures against the Open Payments Database (OPD). AOSSM has not conducted an independent investigation on the OPD and disclaims any liability or responsibility relating thereto.

Ethical approval for this study was obtained from the Auckland University of Technology Ethics Committee (No. 15/132).

References

Ardern

Glasgow

Schneiders

, et al. 2016 consensus statement on return to sport from the First World Congress in Sports Physical Therapy, Bern. Br J Sports Med. 2016;50(14):853-864.

Bland

Altman

DG.

Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;327(8476):307-310.

Bohu

Klouche

Lefevre

Webster

Herman

Translation, cross-cultural adaptation and validation of the French version of the Anterior Cruciate Ligament-Return to Sport after Injury (ACL-RSI) scale. Knee Surg Sports Traumatol Arthrosc. 2015;23(4):1192-1196.

Chawla

Bowyer

Hall

Kegelmeyer

WP.

SMOTE: Synthetic Minority Over-Sampling Technique. J Artif Intell Res. 2002; 16: 321-357.

Creighton

Shrier

Shultz

Meeuwisse

Matheson

GO.

Return-to-play in sport: a decision-based model. Clin J Sport Med. 2010;20(5):379-385.

Denegar

Ball

DW.

Assessing reliability and precision of measurement: an introduction to intraclass correlation and standard error of measurement. J Sport Rehabil. 1993;2(1):35-42.

Doborjeh

Kasabov

Doborjeh

Enayatollahi

Gandomi

AH.

Personalised modelling with spiking neural networks integrating temporal and static information. Neural Networks. 2019;119:162-177.

Faul

Erdfelder

Lang

Buchner

G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods. 2007;39(2):175-191.

Gerometta

Klouche

Herman

Lefevre

Bohu

The Shoulder Instability-Return to Sport after Injury (SIRSI): a valid and reproducible scale to quantify psychological readiness to return to sport after traumatic shoulder instability. Knee Surg Sports Traumatol Arthrosc. 2018;26(1):203-211.

10.

Kvist

Österberg

Gauffin

Tagesson

Webster

Ardern

Translation and measurement properties of the Swedish version of ACL-Return to Sports after Injury questionnaire. Scand J Med Sci Sports. 2013;23(5):568-575.

11.

Louati

Bouche

Bauer

Hardy

Translation and validation of the shoulder instability-return to sport after injury (SIRSI) score in French. J Exp Orthop. 2022;9:41.

12.

Marans

HJJ

Angel

KRR

Schemitsch

EHH

Wedge

JHH

. The fate of traumatic anterior dislocation of the shoulder in children. J Bone Joint Surg Am. 1992;74(8):1242-1244.

13.

Olds

Ellis

Parmar

Kersten

Who will redislocate his/her shoulder? Predicting recurrent instability following a first traumatic anterior shoulder dislocation. BMJ Open Sport Exerc Med. 2019;5(1):e000447.

14.

Olds

Webster

KE.

Factor structure of the Shoulder Instability Return to Sport After Injury scale: performance confidence, reinjury fear and risk, emotions, rehabilitation and surgery. Am J Sports Med. 2021;49(10):2737-2742.

15.

Pasqualini

Rossi

Brandariz

Tanoira

Fuentes

Ranalletta

The short, 5-item Shoulder Instability–Return to Sport After Injury (SIRSI) score performs as well as the longer version in predicting psychological readiness to return to sport. Arthroscopy. 2023;39(5):1131-1138.e1.

16.

Pasqualini

Rossi

Tanoira

Ranalletta

Return to sports, functional outcomes, and recurrences after arthroscopic Bankart repair in soccer players. Shoulder Elbow. 2020;14:16-20.

17.

Rossi

Pasqualini

Brandariz

, et al. Relationship of the SIRSI score to return to sports after surgical stabilization of glenohumeral instability. Am J Sports Med. 2022;50(12):3318-3325.

18.

Silva

Mendes

LMR

Lima

POP

Almeida

GPL

. Translation, cross-adaptation and measurement properties of the Brazilian version of the ACL-RSI Scale and ACL-QoL Questionnaire in patients with anterior cruciate ligament reconstruction. Braz J Phys Ther. 2018;22(2):127-134.

19.

Tjong

Devitt

Murnaghan

Ogilvie-Harris

Theodoropoulos

JS.

A qualitative investigation of return to sport after arthroscopic Bankart repair: beyond stability. Am J Sports Med. 2015;43(8):2005-2011.

20.

Webster

Feller

JA.

Development and validation of a short version of the Anterior Cruciate Ligament Return to Sport After Injury (ACL-RSI) scale. Orthop J Sports Med. 2018;6(4):2325967118763763.

21.

Webster

Feller

Lambros

. Development and preliminary validation of a scale to measure the psychological impact of returning to sport following anterior cruciate ligament reconstruction surgery. Phys Ther Sport. 2008;9(1):9-15.

22.

Youden

WJ.

Index for rating diagnostic tests. Cancer. 1950;3(1):32-35.

23.

Yung

Ardern

Serpiello

Robertson

A framework for clinicians to improve the decision-making process in return to sport. Sports Med Open. 2022;8(1):52.