How Are We Matching in ACL Reconstruction Research? A Systematic Review of Methods,Reporting,and Covariate Selection

Methods

Study Selection

All identified citations were imported into Covidence (Veritas Health Innovation) for title and abstract screening. Two independent reviewers screened each article for inclusion (J.R.P., A.M.H.). Discrepancies were resolved by a third author (B.S.H.). Articles deemed eligible proceeded to full-text review, where final inclusion decisions were made. Reasons for exclusion at the full-text stage were recorded and documented in the PRISMA flow diagram in Figure 1.

OPEN IN VIEWER

Figure 1.

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram of study inclusion.

Results

Covariate Inclusion

Across the 97 included studies, a total of 60 unique covariates were used in matching algorithms, yielding 76 distinct combinations of covariates. The median number of covariates used per study was 4 (range, 1-16). Studies using PSM employed significantly more covariates than those using direct matching (6.17 ± 2.79 vs 3.75 ± 1.63; P < .0001). Similarly, studies using large database sources matched on more covariates than those from single-center institutions (6.33 ± 2.89 vs 4.08 ± 1.96; P < .0001). The most commonly used covariates were age (n = 94; 96.9%), sex (n = 82; 84.5%), body mass index (n = 40; 41.2%), graft type (n = 23; 23.7%), concomitant meniscal or cartilaginous injury (n = 22; 22.7%), and preoperative activity level (n = 22; 22.7%) (Figure 2). Notably, 26 of the 60 identified covariates (43.3%) were used in only one study and were not replicated across other studies. The mean MINORS score was 20.37 for the included studies.

OPEN IN VIEWER

Figure 2.

Frequency of covariates used for patient matching in ≥2 studies. ACLR, anterior cruciate ligament reconstruction; ASA, American Society of Anesthesiologists; BMI, body mass index; PROM, patient-reported outcome measure; VAS, visual analog scale.

Discussion

The present study found that there was notable variability in ACLR matching methodology including the covariates selected, match ratios, and choice of direct matching versus PSM. Moreover, a small minority of papers provided justification for choice of covariates. The previous data gathered from observational studies pertaining to ACLR have significant value; however, the quality of these studies’ findings are dependent upon the type of analytical method elected for each project. Patient matching is a beneficial technique that can aid in mitigating bias and improve study validity. ⁵⁶ Employing an improper matching technique can have a direct influence on study reliability and interpretability. ¹¹⁶ Our analysis of the 97 included studies revealed significant variability in the matching techniques being used across ACLR literature. Notably, there was heterogeneity in covariates included for the matching processes, with a total of 60 unique covariates being used across 76 combinations. Furthermore, we found that only 6 (6.2%) studies provided justification for choice of covariate inclusion in the match process. Our findings reflect the absence of standardized methodological guidance in this field of research. This degree of variability in how match processes were performed and reported could have negative ramifications for the reliability and generalizability of literature produced in this field.^7,75

Our findings parallel previous critiques of observational methodology found in existing orthopaedics literature, which also report highly variable match processes and insufficient justifications of statistical methodology. ⁷ These trends were demonstrated in the scope of matched shoulder arthroplasty outcomes research, where marked discrepancies in matching techniques and data reporting were found. ¹³¹ These conclusions were congruent with the findings of our analysis, which revealed sizable heterogeneity in the number and type of matched covariates. In the field of epidemiology, the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines were developed to enhance the reporting of epidemiological observational studies. ³⁸ These tailored guidelines serve to not only provide structure for researchers in conducting studies, but they also provide uniformity so that editors, reviewers, and readers can critically appraise and compare articles.^38,128 Developing a more standardized analytical approach to matched observational studies within the orthopaedic literature may improve the generalizability and internal validity of reported findings, consistent with broader methodological recommendations for observational research reporting outlined in the STROBE statement.^14,20,34,96 Additionally, alongside the reported discrepancies in covariate inclusion, there were few studies that provided justification for choice of covariates and overall analytical method. Although some heterogeneity in covariate selection is expected given variation in study objectives and patient populations, prior methodological guidance emphasizes the importance of clearly reporting the rationale for variable selection in observational analyses to ensure transparency and reduce potential bias, as recommended in the STROBE statement and related reporting extensions.^14,38,128 The lack of transparency in covariate selection, particularly in terms of how variables were chosen and prioritized, may introduce residual confounding or overfitting, especially in studies with small sample sizes or numerous candidate covariates. Previous literature has similarly raised concerns about the impact that improperly employed matching methodology may have on study results within the scope of orthopaedics literature. Arguelles et al ⁷ highlighted associations between certain flaws in PSM methodology with the production of statistically significant results. Namely, the study notes that a reduction in sample size after matching can notably diminish the power of the PSM model to analyze covariate balance between cohorts, thereby increasing the risk of type 2 error. If P values are the means used to compare baseline cohort characteristics, then there may be covariate imbalances that are falsely reported as not significant due to a small sample size. Unfortunately, the study also noted that studies using P values instead of standardized mean differences (which is independent of sample size) to analyze covariate balance were more likely to find statistically significant results. ¹⁴⁶ This association is troubling because it again highlights the effect that improper matching methodology can have on conclusions and that thoughtful reporting on methods is crucial for critical analysis of study generalizability and reliability. Future efforts should focus on developing standardized reporting frameworks, potentially as an extension of the STROBE statement, that clearly define minimal reporting elements for matched analyses, including matching algorithms, covariate selection rationale, balance diagnostics, and handling of unmatched observations. ¹⁴ In addition, the development of methodological consensus statements led by major orthopaedic research societies, such as the AOSSM and the American Academy of Orthopaedic Surgeons, may help establish field-wide recommendations for the design and reporting of matched observational studies. As demonstrated by the STROBE initiative, adoption of such standards by investigators and orthopaedic journals could improve methodological transparency, facilitate reproducibility, and promote greater consistency in the design and reporting of matched outcomes studies. ¹⁴

Another item that researchers should consider is whether a direct match or PSM technique is preferred for their respective study. In our review, we found that 57.7% of included studies employed a direct matching technique while 42.3% utilized PSM. The PSM technique can have significant value reconciling a wider number of covariates in the match process.^15,21 This is because PSM is based on a summative score assigned to each study participant, which is derived from a logistic regression model composed of all included covariates. This summative score then becomes the basis by which the match is performed. On the other hand, a direct matching technique attempts to match study participants based upon all covariates chosen by the research team. While matching more directly on covariates may potentially increase the homogeneity of the cohorts, the inclusion of too many matching variables can significantly reduce the sample size and improperly exclude study participants. These known issues related to direct matching are further exacerbated by including more covariates because it increases the rigidity of the matching criteria. ²¹ Our study findings reflect this pattern: we found that studies employing PSM included more covariates than studies utilizing a direct match, 6.17 ± 2.79 vs 3.75 ± 1.63, respectively. To mitigate this effect, researchers should consider utilizing PSM when there is a higher quantity of covariates included in the study.

Given the potential utility of standardizing matching techniques utilized in ACLR research, we recommend that researchers consider the inclusion of age, sex, concomitant meniscal or cartilaginous injury, graft type, and preoperative activity level as a baseline for covariate selection in ACLR studies. Our review found that age (96.9%), sex (84.5%), BMI (41.2%), graft type (23.7%), concomitant meniscal or cartilaginous injury (22.7%), and preoperative activity level (22.7%) were the most utilized matching criteria across the included studies. While BMI was among the most commonly employed covariates, its use in matching models should be interpreted with caution. Unlike appendicular skeletal mass or lean body mass, BMI does not distinguish between fat and muscle tissue, which may reduce its precision in capturing body composition differences that are clinically relevant to musculoskeletal outcomes.^{11,26,30,79,108,137} Nevertheless, BMI is still frequently used in clinical research and registry data sets because it is easily measured, widely recorded, and available for large cohorts, even if it is a less precise surrogate of true body composition.^46,47,143 We also recommend that researchers consider the addition of preoperative laxity and preoperative pivot grade as a part of the match process. Previous studies have pointed to the significant impact that these assessments can have upon ACLR outcomes.^50,55 Despite this documented association between these variables and ACLR outcomes, preoperative laxity was included in 9.3% of studies and preoperative pivot grade was included in just 2.1%. We acknowledge, however, that collecting these variables may not be feasible in all study types, particularly in retrospective database studies where chart-level data are not consistently available. In such cases, researchers could consider more readily accessible covariates in addition to those previously mentioned. Items like comorbidity indices (eg, Charlson Comorbidity Index or Elixhauser Comorbidity Index), smoking status, and time from injury to surgery are frequently documented in both prospective and retrospective data sets.^{16,60-62,65,100,114} Comorbidity indices are particularly valuable as they aggregate multiple patient health conditions, providing a single measure that captures overall health status and related demographic risk factors, which can influence ACLR outcomes and help control for confounding.^{8,49,87,100,130} Moreover, there may be utility in assembling an expert consensus on the most important covariates in this field to help standardize methodological guidance. While researchers may tailor their respective match procedure and covariate inclusion to their specific project, we recommend including a justification for their selections to increase overall transparency and assist in critical evaluation. If researchers choose to match based on a higher number of covariates, PSM should be considered to better incorporate matching variables and prevent the diminishing of sample size.

Conclusion

Matching practices in ACLR studies remain highly variable, with limited justification provided for covariate selection. PSM and database-based studies tend to incorporate a greater number of covariates, yet reporting of matching methodology is often inconsistent. To enhance the quality, reproducibility, and comparability of ACLR research, future studies should adopt standardized reporting practices for matching, including explicit descriptions of covariate selection, matching algorithms, balance diagnostics, and match ratios. These steps can serve as a foundation for a more unified research framework, enabling future studies to collectively generate higher quality, generalizable evidence for ACLR outcomes.