Usefulness of Current Patient-Reported Outcome Scales for ACL Injury: A Mixed-Methods Evaluation of Stakeholder-Perceived Utility of Specific Constructs and Items Across the Rehabilitation Timeline

Abstract

Background:

Numerous patient-reported outcome measures (PROMs) have been used in patients with anterior cruciate ligament reconstruction (ACLR), often with overlapping constructs of interest and limited content validity. Inefficient scale application increases burden and diminishes overall usefulness for both the patient and practitioner.

Purpose:

To isolate specific PROM items across a diverse set of constructs that patients and practitioners perceive as having the greatest value at various stages of recovery and return to sport (RTS) in patients after ACLR.

Study Design:

Cross-sectional study.

Methods:

A combined 77 stakeholders participated in this 2-phase mixed-methods investigation. In phase 1, a total of 27 patients and 21 practitioners selected individual PROM items from various constructs that had the greatest utility or importance. In phase 2, the highest rated items were further tested in a head-to-head comparison with 29 stakeholders who attended the 2022 ACL Injury Research Retreat. In addition to the utility assessment, practitioners answered other questions related to importance and timing of PROM assessments.

Results:

In phase 1, both patients and practitioners shared the same top item in 6 of the 8 (75%) constructs assessed. In phase 2, the construct of psychological burden was rated as “extremely important” by 59% of respondents, followed by physical function (54%), symptoms (35%), and donor site issues (10%). The PROM items of confidence, perceived likelihood of reinjury, and difficulty stopping quickly were rated by a respective 93%, 89%, and 86% of the sample as either “very useful” or “extremely useful.” All constructs except donor site issues were rated by most stakeholders to be absolutely necessary to evaluate treatment progress and RTS readiness at the 6-month postoperative time and at RTS.

Conclusion:

Overall, psychological burden, with specific items related to confidence and reinjury likelihood, were rated as most important and useful by both patients and practitioners. The second most important and useful PROM item was related to higher intensity function (eg, decelerating or jumping/landing activities during sports).

Keywords

construct relevance content validity item-relevance patient-reported outcomes psychosocial recovery rehabilitation return to sport usability

Anterior cruciate ligament (ACL) injury reconstruction (ACLR) is a common surgical procedure, with approximately 200,000 cases performed annually in the United States with rising incidence in young athletic populations.^23,29 The process of rehabilitation and return to sport (RTS) after ACLR is complex, involving both physical recovery as well as the influence of psychosocial factors such as perceived pain and function, readiness, quality of life, confidence, social support, and kinesiophobia.^1,7,32 After ACLR, patients undergo extensive rehabilitation to restore physical functioning and must also overcome psychosocial-related barriers to maximize their likelihood of achieving satisfactory long-term outcomes important for their individual quality of life.^15,37 Restoration of both physical and psychosocial function are associated with higher rates of RTS and lower rates of reinjury after RTS.^1,12,30 Thus, comprehensive care of an athlete after ACLR requires a detailed understanding of how these factors present and interact throughout recovery.

In a patient-centered model, it is vital not only to understand the athlete's perception of their function and recovery but also to consider input from multidisciplinary perspectives, including the surgeon, physical therapist, athletic trainer, and research teams utilizing these factors in guiding therapeutics. Aside from qualitative methods (eg, interviewing patients), patient-reported outcome measures (PROMs) are the gold standard for evaluating psychosocial-related factors. Via self-report, PROMs provide a means to quantify a construct of interest and are particularly useful for gleaning insight into psychosocial factors that cannot be measured or “seen” with traditional biomedical/imaging assessments (eg, patient pain or confidence). There are numerous PROMs in the literature that are used in current clinical practice, all of which assess differing, and sometimes overlapping, constructs of interest, limiting their optimal utilization in patients with ACLR.³⁵ PROMs used to assess physical and psychosocial domains related to ACL injury include but are not limited to the Knee injury and Osteoarthritis Outcome Score (KOOS),³³ International Knee Documentation Committee (IKDC) Subjective Knee Evaluation Form,²⁴ Knee Numeric-Entity Evaluation Score-ACL (KNEES-ACL),¹⁰ Donor-Site-Related Scale,⁴ and ACL Return to Sport After Injury Scale (ACL-RSI),³⁸ with the KOOS and IKDC used most frequently.^19,42 Although these and other PROMs are utilized widely, there is limited evidence to demonstrate which, if any, are ideal to capture desired constructs (eg, perceived physical function, symptoms, psychological burden, donor-site issues, etc) and/or are uniquely sensitive to predicting long-term patient outcomes (eg, risk of secondary ACL injury).

Broader limitations of PROMs typically utilized in patients after ACLR include overlapping question content (eg, redundancy), items with limited relevance for patients with ACLR, intended measurement constructs lacking a clear definition (eg, general “pain” vs precise “knee pain intensity”),^9,11,22,42 variable utility of questions within a single construct,³⁸ and the inconvenience of completing multiple PROMs.²⁵ Thus, PROMs are often utilized inappropriately, with researchers and clinical practitioners seeking to evaluate a construct or identify an outcome measure outside of the intended purposes of the scale employed.^11,19,26 For example, the IKDC subjective form was initially developed to assess and compare generalized perceptions of knee function across multiple different knee pathologies.²⁴ However, it has been adopted by clinicians and researchers for specific evaluation of ACLR outcomes and successful RTS.^1-3,27,40

It may therefore be beneficial to investigate future development of more ACLR-relevant PROMs with specific domain and construct items to inform clinical decision-making. To initiate this pathway toward the development of more effective and relevant ACLR-related PROMs, consideration of best-practice psychometric scale development methods is warranted.⁵ The first steps include generating new, or selecting from previously developed, domain- and construct-specific items that exhibit good-to-exceptional content utility. This is often performed by having experts and patients in each content area select individual items from current PROMs that they deem as most “important”; if no items exist, new items are then generated.

Given the limited research examining the relevance of PROMs to ACLR-related stakeholders, the purpose of this study was to identify PROM items that patients with ACLR and multidisciplinary practitioners (surgeons, researchers, etc) perceive as having the greatest value. Specifically, we sought to determine item-level content utility, construct relevance, and appropriate timing of assessment in ACL PROMs.

Methods

Overall Approach

The current project employed a 2-phase, cross-sectional, mixed-methods study design to explore ACL PROM construct and item-level utility. The study protocol was considered exempt from institutional review board approval, and all study participants provided written informed consent.

Phase 1 Approach

Participants

In phase 1, we recruited a diverse sample of stakeholders involved in the measurement and interpretation of PROMs. Stakeholders included practitioners such as surgeons, athletic trainers, physical therapists, and researchers (n = 21), as well as patients with ACLR (n = 27) (Table 1). Stakeholders were recruited through word of mouth from our networks and clinical practices. Although the sample size required for usability testing is often debated, it is recommended that data from 3 to 20 participants is adequate.^14,28

Table 1

Sample Demographic Characteristics of Participants in Phase 1 of the Study (n = 48)^a

Characteristic	Value
Practitioners (n = 21)
Age, y	46.32 ± 10.48
Female sex, %	29
Occupation
Physical therapist	8
Surgeon	7
Researcher	5
Athletic trainer	1
Experience in current occupation, y	19.14 ± 10.71
Patients (n = 27)
Age, y	27.44 ± 9.17
Female sex, %	69
Time since ACL injury, mo	8 (1-240)
Stage of recovery
After RTS	13
Before RTS	13
Preoperatively	1

Data are reported as mean ± SD, median (range), or n. ACL, anterior cruciate ligament; RTS, return to sport.

PROMs Used

Given the validity evidence of the various ACL PROMs, the substantial overlap, and the desire to extend previous methods performed for the IKDC and KOOS,^21,36 we evaluated all items from the KNEES-ACL, ACL-RSI, and Donor-Site-Related Scale that covered the following 8 constructs: (1) physical function in activities of daily living (ADL), (2) physical function in sport, (3) stability/fatigue, (4) symptoms, (5) pain, (6) donor site issues, (7) psychological readiness, and (8) psychological burden. The 8 constructs were selected based on various reviews of the diverse factors that may affect recovery after ACLR.^{1,2,8,13,16,19,32,37} We did not use the KOOS or IKDC due to suboptimal content validity,^11,19,22,42 indicating that their scale items do not adequately capture all of the dimensions of the intended construct being evaluated.^9,35,42 Further, the IKDC was developed without patient input,^13,23,44 thus failing to consider the complex biopsychosocial factors that influence how items are interpreted by patients with ACLR. In contrast to the IKDC and KOOS, the KNEES-ACL has good content validity, but it exists only in Danish and has not been properly translated into other languages for more widespread adoption.^10,19,22 A translated version of the KNEES-ACL was provided by the originators, who had planned a construct validation project with researchers/physicians from Boston University (M. Krogsgaard, personal communication, December 15, 2021). The Donor-Site-Related Scale also exhibits adequate content validity, but it is limited to the sole evaluation of donor-site-related functional problems after ACLR.^4,19 In a 2019 study, evaluation of “psychological readiness” via the ACL-RSI showed promise as a meaningful and important construct in younger patients with ACLR given its high sensitivity, albeit with low specificity, in predicting secondary injury.³⁰

A list of all included PROM items is shown in Appendix Table A1.

PROM Items

The goal of phase 1 was to identify specific PROM items that the stakeholders found had the greatest utility for the recovery process. To accomplish this goal, we had stakeholders select the top 2 PROM items within each construct. Practitioners were asked to consider the utility assessment at the preoperative, RTS, and 2-year postoperative timepoints. Patients completed the utility assessment only for their timepoint in the postoperative recovery process.

Specifically, for each of the 8 constructs, the stakeholders answered the question “Information from which two items below would be of greatest utility when determining their (patient) potential for success?” and the patients with an ACL injury answered the question “Which two questions would be most important to measure to help with your recovery process?” Participants selected only 2 items from each construct, as this approach has been shown to align more closely with how people make decisions/value judgments.²⁰ In addition, this assessment design was chosen given the unidimensionality of the constructs and to mimic how one could reduce items using a user-centered/informed approach as opposed to standard statistical-based psychometric approaches such as item response theory.^21,36

Statistical Analysis

We calculated the proportion of participants who rated each item within the top 2 to determine which items within each construct had the highest utility. Given the descriptive and exploratory nature of this initial analysis, further statistical analysis or comparison tests were not performed.

Phase 2 Approach

Participants

Phase 2 stakeholders were experienced researchers and clinical practitioners who participated in the 2022 ACL Injury Research Retreat held March 17 and 19, 2022, in High Point, North Carolina, USA. See Table 2 for sample demographics.

Table 2

Demographic Characteristics of Participants in Phase 2 of the Study (n = 29)^a

Characteristic	Value
Occupation
Researcher	9
Physical therapist	6
Athletic trainer	6
Physical therapist/researcher	3
Athletic trainer/researcher	3
Research consultant	1
Athlete/researcher	1
Female sex, %	48
Age, y	33.82 ± 8.77
Experience in current occupation, y	8.87 ± 7.93
Uses PROMs for ACL injury (never/sometimes/often), %
Research outcome measurement	14/17/69
Progress assessment	21/21/59
RTS decisions	17/24/59

Data are reported as mean ± SD or n. ACL, anterior cruciate ligament; PROMs, patient-reported outcome measures; RTS, return to sport.

Evaluation

The goal of phase 2 was to further identify specific PROM items that the phase 2 stakeholders (inclusive of all participant occupations listed in Table 2) found of greatest utility for the recovery process. Only the top 2 items with greatest utility from the phase 1 stakeholder assessments were selected for further evaluation in a head-to-head quantitative approach.

All phase 2 participants again evaluated the utility of the various PROM items identified in phase 1, but this time they rated the usefulness of each item on a 5-point Likert scale that varied from “not at all useful” to “extremely useful” to compare each item across all constructs. In addition to the utility assessment, participants also rated the importance of the various general constructs and when they should be measured: preoperatively, at 3 months postoperatively, at 6 months postoperatively, at RTS testing, or at 2 years postoperatively. Two open-response questions were also used to solicit new items for patient recovery progress and RTS readiness: “If you could only ask one question to track the patient's recovery progress, what would it be?” and “If you could only ask one question to determine if they are ready to return to sport, what would it be?”

Statistical Analysis

Given the descriptive and exploratory goals, Likert rating and assessment timing data were aggregated into proportions and reported. Open responses were coded independently by 2 researchers (E.J.P. and A.-I.M.) then synthesized and aggregated.

Results

Phase 1

Overall, there was substantial item-level variability regarding utility within each construct (Figure 1). For example, in the psychological burden construct, the item “It has been frustrating not to be able to participate in leisure activities because of my knee problems” was included in the top 2 most useful items by 43% of practitioners and 42% of patients. In contrast, the item “It has been stressful to perform my job well because of my knee problems” was included in the top 2 by only 6% of practitioners and 2% of patients. Overall, the items selected as having the greatest utility within each construct were similar between practitioners and patients and also consistent across the various timepoints by practitioners. Specifically, items that were included in the top 2 were nearly identical between the practitioners and patients for all constructs except the construct of physical function in sport. Here, the items that were selected as having the most utility for the practitioners were “I have had difficulty stopping quickly when I run” and “I have had to lower my expectations regarding how much I can play sports.” Meanwhile, the patients most frequently selected “I have been more careful than usual when I play sports” and “I have had problems playing sports as much as I like to.” Overall, there seemed to be a small number of items within each construct that were perceived as having high utility.

Figure 1.

Stacked bar chart with percentage of respondents that selected each PROM item as one of their top 2 items of greatest utility for each construct. Patient data were acquired for only 1 timepoint, specifically the timepoint that best aligned with the respective patient's current postoperative recovery stage. Preoperative data were not collected from practitioners for the constructs of donor site issues or psychological readiness, as these 2 constructs would have limited relevance. ADL, activities of daily living; postop, postoperative; preop, preoperative; PROMs, patient-reported outcome measures; RTS, return to sport.

Phase 2

The construct of psychological burden was rated as extremely important by 59% of respondents, followed by physical function (both ADL and sport; 54%), symptoms (35%), and finally donor site issues (10%) (Figure 2). The times at which assessment of psychological burden was deemed absolutely necessary were 6 months postoperatively (89%) and at RTS (83%) (Figure 3). Item-level analysis revealed that most items across the various constructs were found to be “very useful” and “extremely useful,” with psychological burden items, on average, being rated highest (Figure 4). Open response data converged with the Likert rating data, indicating that confidence, trust, and comparison with preinjury state were among the most important questions to ask if participants were limited to a single question (Table 3).

Figure 2.

Proportion of responses of importance ratings for various PROM constructs. PROM, patient-reported outcome measure.

Figure 3.

Proportion of responses for when it would be necessary to assess the 4 PRO constructs. ADL, activities of daily living; postop, postoperative; preop, preoperative; PRO, patient-reported outcome; RTS, return to sport.

Figure 4.

Usefulness proportions across various items and constructs. Percentages shown indicate combined proportion of “very useful” and “extremely useful” ratings for each item (light blue + dark blue shadings). PROM, patient-reported outcome measure.

Table 3

Synthesized Responses to the 2 Open-Ended Questions Regarding Tracking Recovery and Assessing RTS^a

Construct/Theme	Example Question	Combined Count
Confidence	Do you feel confident to return to your sports?	8
Preinjury comparison	Why do you feel you are ready to return to sport at the same level as before your injury?	7
Trust	Do you trust your knee?	4
Previous visit comparison	Do you feel better than last time?	3
Normality	On a scale of 0 to 100, how would you rate your knee's function, with 100 being normal?	3
Thinking about knee	Are you able to perform sports activities without thinking about your knee?	3
Comfort during activity	Do you feel comfortable doing any sort of physical activity at any intensity?	2
Recovery process/activities	Describe what you think is going well and what is not going well in your recovery after your knee surgery. What are your current goals, and do you have any current barrier to achieving those goals?	2
Activity modification	Have you modified your activities since your ACL reconstruction? What are you currently doing to protect your knees from future injuries?	2
Symptoms	Are you experiencing any knee symptoms?	1
Fear of reinjury	Do you fear reinjury or completing any sport-specific movements?	1
Limited function	Do you feel limited in any way with your knee function?	1

ACL, anterior cruciate ligament; RTS, return to sport.

Discussion

In phase 1 of our analyses, we found that constructs and items relating to psychological factors and perceived function consistently received the highest ratings in both patient and multidisciplinary practitioner groups. In phase 2, we identified the most useful items across various constructs that are commonly utilized ACL PROMs (KNEES-ACL, ACL-RSI, and Donor-Site-Related Scale) utilizing similar methodology to existing ACLR literature.^21,36 We also discovered that the constructs of psychological burden and physical function were rated as the most important to stakeholders, and items relating to confidence, trust, and comparison with preinjury state were considered among the most important questions to ask. In addition to psychometric testing of PROMs (eg, validity, reliability), user insights into the utility of specific items should be captured to ensure the scales are efficient, effective, and ultimately provide relevant and useful information that will guide clinical decision-making. The present study adds to existing literature by considering both patient and clinician/researcher insights into the relative importance of individual PROM items across multiple constructs regarding recovery and RTS after ACLR.

A previous study assessing item-level content validity found only 2 items (out of 18) from the IKDC subjective form and 1 item from the KOOS (out of 42) achieved even “moderately important” ratings (mean importance rating, >3).³⁶ Further, a separate study of 126 postoperative patients with ACLR found only 3 of 18 IKDC subjective form items and 5 of 42 KOOS items rated at least “moderately important” in the <12 month postoperative period. However, no items in either the IKDC subjective form or KOOS achieved a mean importance rating of “moderately important” in the period >12 months postoperatively or when the sample was considered as a whole.²¹ These data indicate that most items in the IKDC subjective form and KOOS are considered of little or no utility for evaluating constructs or outcomes important for patients with ACLR. Despite this limitation, the study did find that patients deemed the quality-of-life subscale most important in the KOOS.²¹ Similar to the findings in the current study, these previous data showed that psychological burden and fear of reinjury, as assessed in various instruments, were rated by patients as most important (despite not being assessed specifically in either the IKDC subjective form or the KOOS).³⁶ These psychologically related constructs deemed important by patients with ACLR must be taken into account by multidisciplinary clinical and research experts to appreciate the patient experience and align key factors influencing rehabilitation and RTS after ACLR.

Many ACL PROM scales suggest unidimensionality yet require participants to fill out lengthy scales. For example, even in the short version of the ACL-RSI, there are still 6 items to assess a single underlying construct (psychological readiness), which is redundant and adds participant burden. With the increasing popularity of adaptive testing through item response theory analysis, it is plausible to reduce the number of items per construct.^17,18,20 Our data suggest that even within a “unidimensional” construct (eg, function), all items are not perceived as equally useful. As some items have been found to be more relevant than others, minimizing redundancy while focusing on items with the greatest relevance can optimize the utilization of PROMs.³⁸ Indeed, in the present study, there was similarity between practitioners and patients with ACL injury when rating which items are most relevant for ACLR recovery outcomes. Namely, constructs relating to psychological factors and perceived function and items within these constructs consistently received the highest ratings in both groups. This finding may be particularly important in selecting the constructs and items within each construct that should be asked postoperatively to streamline and optimize PROM collection after ACLR.

A key finding in this study was that psychological burden constructs and items within this construct were rated as very important and the most useful for monitoring patient recovery after ACLR across stakeholders. This is consistent with recent guidelines and expert consensus recognizing recovery from ACLR as a complex process and advocating for the inclusion of assessments of psychological readiness in RTS.³¹ There is a growing body of evidence demonstrating the utility of psychologically driven scales such as the ACL-RSI in evaluating readiness for RTS and risk of secondary injury.^30,34 However, moderate patient-level responsiveness indicates the ACL-RSI may not be optimized for isolating meaningful, construct-specific changes that are independent from other confounding factors (eg, time).^19,39 In the present study, the construct of psychological burden was identified by 59% of respondents as the most important construct to assess to guide treatment decisions after ACL injury. Our open-response and Likert rating results both showed that confidence and comparison with preinjury state were rated highest. Specifically, the question “Are you confident that you can perform at your previous level of sport participation?” was found to be the single most useful item by both patients and practitioners.

In a previous study on patients who had undergone ACLR, the same confidence question was also found to be of highest relevance to the patient.³⁸ Patients who experience fear and uncertainty during ACLR recovery report lower scores on ACL PROMs, and it was concluded that it is important for clinicians to address barriers (eg, avoidance, fear) and facilitators (eg, confidence, social support) to provide better patient care.⁷ There is growing evidence that psychological, social, and contextual factors influence all stages of recovery after sport-related knee injury,³⁷ and identification of these factors as well as mitigation strategies to address them may improve postoperative recovery and RTS after ACLR. The present study expanded upon the current literature,^7,8,13,16 identifying the importance of assessing psychological factors in addition to perceptions of physical function after ACLR, warranting further optimization for how these constructs are evaluated empirically.

Currently, it is not well established at which timepoints and how frequently PROMs should be assessed after ACLR. The present study uniquely identified the usefulness of specific constructs based on assessment time. Of the 5 constructs, 4 (physical function in ADL, physical function in sport, symptoms, and psychological burden) were rated by most stakeholders to be absolutely necessary to evaluate treatment progress and RTS readiness at the 6-month postoperative timepoint and at RTS. A recent modified-Delphi expert consensus group recommended minimum 2-year follow-up for comprehensive assessment after ACLR to include reports of adverse events, clinical measurements of knee function and structure, PROMs, patient activity level, and incidence of secondary injury.³⁵ A minimum 2-year follow-up period was recommended, as the 1 to 2 year period was seen as the interval of highest risk for ACL rerupture and revision surgery.³⁵ Regarding PROMs, the consensus group recommended inclusion of at least 1 knee-specific outcome tool, 1 activity rating scale, and 1 measure of health-related quality of life.³⁵ In this consensus, it was also determined that the IKDC subjective form is the recommended knee-related outcome measure for ACL injury and treatment.³⁵

To our knowledge, there is no existing standard for the most appropriate intervals of assessment; however, the modified-Delphi consensus recommended assessment in the early, middle, and end stages of rehabilitation.³⁵ Common intervals in the literature include evaluations at preoperative and 3- to 4-, 6-, 12-, and 24-month timepoints. It was also recommended that any medium- to long-term outcomes (≥5 years) include assessment of measures of posttraumatic osteoarthritis, given the burden that this places on both the patient and the medical system.³⁵ Based on our current findings, we propose that optimal timing of PROM assessment may be domain/construct-specific and evolving over time.

Optimizing PROMs in ACLR requires assessment of the most valid, reliable, and important data for patients undergoing ACLR.³⁵ It is important to determine construct and item utility because the inclusion of irrelevant items can lead to skewed outcome scores that can be misinterpreted by clinicians and researchers.^11,35,42 The current study lays a foundation to determine which PROM items are seen as most valuable by both patients and multidisciplinary practitioners (inclusive of researchers, athletic trainers, surgeons, and more). In assessing outcomes through PROMs, the concept of “responder burden” is of significant concern. The study findings suggest that responder burden may be minimized by assessing targeted PROMs and condition-specific outcomes at time intervals of greatest relevance. For instance, donor site-related issues were found to have the most importance at the 3-month postoperative interval and can be assessed at that timepoint, when they are most likely to affect patient care. Further studies may continue to optimize unidimensionality within constructs and minimize redundancy within future tools while maintaining acceptable content validity. Further, as our knowledge base grows there is likely a role for constructing computer-based PROM tools using item response theory.⁶ Utilizing item response theory, more dynamic patient- and condition-specific measures may be developed integrating patient data to provide assessments most relevant to the person while still capturing valid constructs of interest. As such, further research is needed to fully appreciate the psychometric properties across various items and assess PROMs across sex differences, sport type and level of competition, graft type, injury mechanism, and other variables of interest to identify meaningful outcomes and individualize targeted interventions for patient care.

Limitations

The current study has several limitations that should be considered relative to the interpretation of the results. We did not obtain feedback from any patients who sustained an ACL injury under the age of 18 years. Future work should include this age demographic when soliciting feedback on outcome measurement instruments given the high ACLR revision rates in this group.^5,24 In addition, we did not collect other demographic information such as race or socioeconomic status that may influence results. To determine the optimal timing of PROM collection, future studies should use larger sample sizes across a more diverse yet consistent set of timepoints. We also evaluated various constructs that have been used extensively in research and clinical domains, but future work may consider evaluating other potentially relevant psycho-social-contextual constructs that may influence the recovery process (eg, support, motivation, avoidance behaviors, autonomy, anxiety, etc).^{8,13,16,32,37,41}

Conclusion

Overall, ACL-related stakeholders deemed various items related to psychological burden and higher intensity physical function to be most useful to assess during the ACL injury recovery process. More specifically, items assessing confidence in performance and fear of reinjury were rated highest. In addition, stakeholders felt that measuring the various PROM constructs was most necessary at 6 months and at RTS testing. Focused efforts toward items deemed most useful could lead to the development of PROMs that are less burdensome for patients, while also clinically meaningful to practitioners, to enhance the rehabilitation process after ACL injury.

Authors

Erich J. Petushek, PhD (Department of Cognitive and Learning Sciences and the Health Research Institute, Michigan Technological University, Houghton, Michigan, USA); Jed A. Diekfuss, PhD (Emory Sports Performance And Research Center [SPARC], Flowery Branch, Georgia, USA; Emory Sports Medicine Center, Atlanta, Georgia, USA; Department of Orthopaedics, Emory University School of Medicine, Atlanta, Georgia, USA); Joseph D. Lamplot, MD (Campbell Clinic Orthopaedics, Memphis, Tennessee, USA); Anne-Inger Mørtvedt, MS (the Department of Cognitive and Learning Sciences and the Health Research Institute, Michigan Technological University, Houghton, Michigan, USA); Lauren C. Hoey, BS (Department of Biology, Bucknell University, Lewisburg, Pennsylvania, USA); Kevin Heo, BS (Emory Sports Medicine Center, Atlanta, Georgia, USA; Department of Orthopaedics, Emory University School of Medicine, Atlanta, Georgia, USA); Camryn B. Petit, BS (Emory SPARC, Flowery Branch, Georgia, USA; Emory Sports Medicine Center, Atlanta, Georgia, USA; Department of Orthopaedics, Emory University School of Medicine, Atlanta, Georgia, USA); Kim D. Barber Foss, PhD, ATC (Emory SPARC, Flowery Branch, Georgia, USA; Emory Sports Medicine Center, Atlanta, Georgia, USA; Department of Orthopaedics, Emory University School of Medicine, Atlanta, Georgia, USA); Shayla M. Warren, BS (Emory SPARC, Flowery Branch, Georgia, USA; Emory Sports Medicine Center, Atlanta, Georgia, USA; Department of Orthopaedics, Emory University School of Medicine, Atlanta, Georgia, USA); Alexis B. Slutsky-Ganesh, PhD (Emory SPARC, Flowery Branch, Georgia, USA; Emory Sports Medicine Center, Atlanta, Georgia, USA; Department of Orthopaedics, Emory University School of Medicine, Atlanta, Georgia, USA; Department of Kinesiology, University of North Carolina at Greensboro, Greensboro, North Carolina, USA); Charles D. Kenyon, DO, CSCS (Emory Sports Medicine Center, Atlanta, Georgia, USA; Department of Orthopaedics, Emory University School of Medicine, Atlanta, Georgia, USA); April L. McPherson, PhD (Emory SPARC, Flowery Branch, Georgia, USA; Emory Sports Medicine Center, Atlanta, Georgia, USA; Department of Orthopaedics, Emory University School of Medicine, Atlanta, Georgia, USA; Department of Orthopaedics, The Ohio State University Wexner Medical Center, Columbus, Ohio, USA); Michael E. Biller, PT, DPT (Department of Orthopaedics, Emory University School of Medicine, Atlanta, Georgia, USA; Emory Physical Therapy, Atlanta, Georgia, USA); Michael Newsome, PT (Department of Orthopaedics, Emory University School of Medicine, Atlanta, Georgia, USA; Emory Physical Therapy, Atlanta, Georgia, USA); Brian Jennings, PT, DPT (Department of Orthopaedics, Emory University School of Medicine, Atlanta, Georgia, USA; Emory Physical Therapy, Atlanta, Georgia, USA); John W. Xerogeanes, MD (Emory Sports Medicine Center, Atlanta, Georgia, USA; Department of Orthopaedics, Emory University School of Medicine, Atlanta, Georgia, USA); and Gregory D. Myer, PhD, FACSM, CSCS*D (Emory SPARC, Flowery Branch, Georgia, USA; Department of Orthopaedics, Emory University School of Medicine, Atlanta, Georgia, USA; Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology & Emory University, Atlanta, GA, USA; The Micheli Center for Sports Injury Prevention, Waltham, Massachusetts, USA; Youth Physical Development Centre, Cardiff Metropolitan University, Wales, UK).

Footnotes

Appendix

Appendix Table A1

Included PROM Items by Construct^a

Physical function in ADL

I have had problems running

I have had problems squatting

I have had problems going down stairs

I have had problems walking on uneven ground, for example in the woods

I have had problems walking on level ground

I have had problems crawling on my hands and knees

I have had problems kneeling to pick something up

I have had problems riding a bike

Physical function in sport

I have had difficulty stopping quickly when I run

I have been more careful than usual when I play sports

I have had to lower my expectations regarding how much I can play sports

I have had difficulty landing when I've jumped

I have had problems playing sports as much as I like to

I have had difficulty changing direction when I run

I have problems giving it my all when I have played sports

I feel isolated from the people I used to play sports with before I was injured

I have had trouble jumping

I don't feel I can compete as much as I'd like

Stability/fatigue

I have had a tendency to protect my injured knee

I have felt I could not control my knee when I move

During knee exercises, I get tired more quickly in my injured leg compared to my good leg

During knee exercises, I have a hard time keeping my balance

I have overburdened my good knee

I have had a limp feeling in my knee when I move around

During knee exercises, my leg shakes

Symptoms

I have felt like I couldn't trust my knee when I move

I have had to think about my knee when I move around

My knee has felt loose when I move around

I have felt my knee is unstable because my muscles aren't strong enough

Pain

I have had pain in my knee after knee exercises

It hurts when I twist on my knee

It hurts when I jump

It hurts when I walk up stairs

I have had knee pain after I have taken a long walk

I have had pain in my knee when I sit with my knees bent

It hurts when I walk on uneven ground, for example in the woods

Donor site issues

Do you have weakness in the knee

Do you have pain in front of the knee

Do you have wasting/atrophy of the quadriceps

Do you have difficulty going up or down stairs

Do you have pain at donor site

Do you have kneeling pain at donor site

Does prolonged sitting with the knee bent cause pain

Do you have numbness

Are you satisfied with the way your incision looks

Psychological readiness

Are you fearful of reinjuring your knee by playing your sport

Are you confident that you can perform at your previous level of sport participation

Are you confident that you could play your sport without concern for your knee

Do you think you are likely to reinjure your knee by participating in your sport

Are you nervous about playing your sport

Do you find it frustrating to have to consider your knee with respect to your sport

Psychological burden

It has been frustrating not to be able to participate in leisure activities because of my knee problems

It has been stressful not knowing when my knee would be well again

It has been stressful just trying to get through the day because of my knee problems

It has been stressful to meet the demands of my family life because of my knee problems

It has been stressful to perform my job well because of my knee problems

ADL, activities of daily living; PROM, patient-reported outcome measure.

Acknowledgements

The authors thank the participants who volunteered for this study as well as the organizers of the 2022 ACL Injury Research Retreat for allowing access to collect data at their event.

Final revision submitted May 9, 2023; accepted June 29, 2023.

One or more of the authors has declared the following potential conflict of interest or source of funding: J.D.L. has received education payments from United Orthopedics, Smith & Nephew, and Elite Orthopedics; consulting fees from DePuy Synthes; and hospitality payments from Arthrex and Smith & Nephew. J.W.X. has received education payments from United Orthopedics, consulting fees from Arthrex and Trice Medical, nonconsulting fees from Arthrex, and royalties from Arthrex and has stock options in My-Eye. G.D.M. has received research support from Q30 Innovations and ElMinda and royalties from Human Kinetics and Wolters Kluwer and holds a patent in rehabilitation-enhancing biofeedback technologies (US11350854B2). AOSSM checks author disclosures against the Open Payments Database (OPD). AOSSM has not conducted an independent investigation on the OPD and disclaims any liability or responsibility relating thereto.

Ethical approval for this study was waived by Michigan Technological University (reference No. M2152).

References

Ardern

Taylor

Feller

Webster

KE.

A systematic review of the psychological factors associated with returning to sport following injury. Br J Sports Med. 2013;47(17):1120-1126.

Ardern

Taylor

Feller

Webster

KE.

Fifty-five per cent return to competitive sport following anterior cruciate ligament reconstruction surgery: an updated systematic review and meta-analysis including aspects of physical functioning and contextual factors. Br J Sports Med. 2014;48:1543-1552.

Ardern

Taylor

Feller

Whitehead

Webster

KE.

Sports participation 2 years after anterior cruciate ligament reconstruction in athletes who had not returned to sport at 1 year: a prospective follow-up of physical function and psychological factors in 122 athletes. Am J Sports Med. 2015;43(4):848-856.

Aufwerber

Hagströmer

Heijne

Donor-site-related functional problems following anterior cruciate ligament reconstruction: development of a self-administered questionnaire. Knee Surg Sports Traumatol Arthrosc. 2012;20(8):1611-1621.

Boateng

Neilands

Frongillo

Melgar-Quiñonez

Young

SL.

Best practices for developing and validating scales for health, social, and behavioral research: a primer. Front Public Health. 2018;6:149.

Brodke

Hung

Bozic

KJ.

Item response theory and computerized adaptive testing for orthopaedic outcomes measures. J Am Acad Orthop Surg. 2016;24(11):750-754.

Burland

Howard

Lepley

DiStefano

Lepley

Frechette

What are our patients really telling us? Psychological constructs associated with patient-reported outcomes after anterior cruciate ligament reconstruction. J Athl Train. 2020;55(7):707-716.

Burland

Toonstra

Howard

JS.

Psychosocial barriers after anterior cruciate ligament reconstruction: a clinical review of factors influencing postoperative success. Sports Health. 2019;11(6):528-534.

Comins

Brodersen

Krogsgaard

Beyer

Rasch analysis of the Knee injury and Osteoarthritis Outcome Score (KOOS): a statistical re-evaluation. Scand J Med Sci Sports. 2008;18(3):336-345.

10.

Comins

Krogsgaard

Brodersen

Development of the Knee Numeric-Entity Evaluation Score (KNEES-ACL): a condition-specific questionnaire. Scand J Med Sci Sports. 2013;23(5):e293-e301.

11.

Comins

Siersma

Lind

Jakobsen

Krogsgaard

MR.

KNEES-ACL has superior responsiveness compared to the most commonly used patient-reported outcome measures for anterior cruciate ligament injury. Knee Surg Sports Traumatol Arthrosc. 2018;26(8):2438-2446.

12.

Erickson

Jacobs

Johnson

Ireland

Noehren

Psychosocial factors 3-months after anterior cruciate ligament reconstruction predict 6-month subjective and objective knee outcomes. J Orthop Res. 2022;40(1):231-238.

13.

Everhart

Best

Flanigan

DC.

Psychological predictors of anterior cruciate ligament reconstruction outcomes: a systematic review. Knee Surg Sports Traumatol Arthrosc. 2015;23(3):752-762.

14.

Faulkner

Beyond the five-user assumption: benefits of increased sample sizes in usability testing. Behav Res Methods Instrum Comput. 2003;35(3):379-383.

15.

Filbay

Skou

Bullock

, et al. Long-term quality of life, work limitation, physical activity, economic cost and disease burden following ACL and meniscal injury: a systematic review and meta-analysis for the OPTIKNEE consensus. Br J Sports Med. 2022;56(24):1465-1474.

16.

Forsdyke

Smith

Jones

Gledhill

Psychosocial factors associated with outcomes of sports injury rehabilitation in competitive athletes: a mixed studies systematic review. Br J Sports Med. 2016;50(9):537-544.

17.

Fries

Bruce

Cella

The promise of PROMIS: using item response theory to improve assessment of patient-reported outcomes. Clin Exp Rheumatol. 2005;23(5)(suppl 39):S53-S57.

18.

Fries

Witter

Rose

Cella

Khanna

Morgan-DeWitt

Item response theory, computerized adaptive testing, and PROMIS: assessment of physical function. J Rheumatol. 2014;41(1):153-158.

19.

Gagnier

Shen

Huang

Psychometric properties of patient-reported outcome measures for use in patients with anterior cruciate ligament injuries: a systematic review. JBJS Rev. 2018;6(4):e5.

20.

Gulledge

Smith

Ziedas

Muh

Moutzouros

Makhni

EC.

Floor and ceiling effects, time to completion, and question burden of PROMIS CAT domains among shoulder and knee patients undergoing nonoperative and operative treatment. JBJS Open Access. 2019;4(4):e0015.1-e0015.7.

21.

Hambly

Griva

IKDC or KOOS: which one captures symptoms and disabilities most important to patients who have undergone initial anterior cruciate ligament reconstruction?

Am J Sports Med. 2010;38(7):1395-1404.

22.

Hansen

Jensen

Odgaard

, et al. Four of five frequently used orthopedic PROMs possess inadequate content validity: a COSMIN evaluation of the mHHS, HAGOS, IKDC-SKF, KOOS and KNEES-ACL. Knee Surg Sports Traumatol Arthrosc. 2022;30(11):3602-3615.

23.

Herzog

Marshall

Lund

Pate

Mack

Spang

JT.

Incidence of anterior cruciate ligament reconstruction among adolescent females in the United States, 2002 through 2014. JAMA Pediatr. 2017;171(8):808-810.

24.

Irrgang

Anderson

Boland

, et al. Development and validation of the international knee documentation committee subjective knee form. Am J Sports Med. 2001;29(5):600-613.

25.

Jansson

Oji

Bradley

Zhang

Feeley

BT.

High rate of overlapping question content among commonly used patient-reported outcome measurements for anterior cruciate ligament injury. Arthrosc Sports Med Rehabil. 2021;3(6):e1775-e1801.

26.

Latham

Mehta

Nguyen

, et al. Performance-based or self-report measures of physical function: which should be used in clinical trials of hip fracture patients? Arch Phys Med Rehabil. 2008;89(11):2146-2155.

27.

Lentz

Zeppieri

Jr Tillman

, et al. Return to preinjury sports participation following anterior cruciate ligament reconstruction: contributions of demographic, knee impairment, and self-report measures. J Orthop Sports Phys Ther. 2012;42(11):893-901.

28.

Macefield

How to specify the participant group size for usability studies: a practitioner's guide. J Usability Stud. 2009;5(1):34-45.

29.

Mather

III Koenig

Kocher

, et al. Societal and economic impact of anterior cruciate ligament tears. J Bone Joint Surg Am. 2013;95(19):1751-1759.

30.

McPherson

Feller

Hewett

Webster

KE.

Psychological readiness to return to sport is associated with second anterior cruciate ligament injuries. Am J Sports Med. 2019;47(4):857-862.

31.

Meredith

Rauer

Chmielewski

, et al. Return to sport after anterior cruciate ligament injury: Panther Symposium ACL Injury Return to Sport Consensus Group. Knee Surg Sports Traumatol Arthrosc. 2020;28(8):2403-2414.

32.

Nwachukwu

Adjei

Rauck

, et al. How much do psychological factors affect lack of return to play after anterior cruciate ligament reconstruction? A systematic review. Orthop J Sports Med. 2019;7(5):2325967119845313.

33.

Roos

Lohmander

LS.

The Knee injury and Osteoarthritis Outcome Score (KOOS): from joint injury to osteoarthritis. Health Qual Life Outcomes. 2003;1:64.

34.

Sadeqi

Klouche

Bohu

Herman

Lefevre

Gerometta

Progression of the psychological ACL-RSI Score and return to sport after anterior cruciate ligament reconstruction: a prospective 2-year follow-up study from the French Prospective Anterior Cruciate Ligament Reconstruction Cohort Study (FAST). Orthop J Sports Med. 2018;6(12):2325967118812819.

35.

Svantesson

Hamrin Senorski

Webster

, et al. Clinical outcomes after anterior cruciate ligament injury: panther symposium ACL injury clinical outcomes consensus group. Knee Surg Sports Traumatol Arthrosc. 2020;28(8):2415-2434.

36.

Tanner

Dainty

Marx

Kirkley

Knee-specific quality-of-life instruments: which ones measure symptoms and disabilities most important to patients?

Am J Sports Med. 2007;35(9):1450-1458.

37.

Truong

Mosewich

Holt

Miciak

Whittaker

JL.

Psychological, social and contextual factors across recovery stages following a sport-related knee injury: a scoping review. Br J Sports Med. 2020;54(19):1149-1156.

38.

Webster

Feller

JA.

Development and validation of a short version of the anterior cruciate ligament return to sport after injury (ACL-RSI) Scale. Orthop J Sports Med. 2018;6(4):2325967118763763.

39.

Webster

Feller

JA.

Evaluation of the responsiveness of the anterior cruciate ligament return to sport after injury (ACL-RSI) Scale. Orthop J Sports Med. 2021;9(8):23259671211031240.

40.

Welling

Benjaminse

Lemmink

Gokeler

Passing return to sports tests after ACL reconstruction is associated with greater likelihood for return to sport but fail to identify second injury risk. Knee. 2020;27(3):949-957.

41.

Xiao

van Niekerk

Trivedi

, et al. Patients who return to sport after primary anterior cruciate ligament reconstruction have significantly higher psychological readiness: a systematic review and meta-analysis of 3744 patients. Am J Sports Med. 2023;51(10):2774-2783.

42.

Zsidai

Narup

Senorski

, et al. The Knee Injury and Osteoarthritis Outcome Score: shortcomings in evaluating knee function in persons undergoing ACL reconstruction. Knee Surg Sports Traumatol Arthrosc. 2022;30(11):3594-3598.