Clinician V ersus Nomogram Predicted Estimates of Kidney Stone Recurrence Risk

Introduction

Kidney stones are prevalent, ¹ costly, ² and can have multiple symptomatic recurrences over a lifetime. ³ Estimates of kidney stone recurrence rates range from 30% to 50% at 10 years. ^4,5 This represents a high recurrence rate overall, but individual patients with kidney stones may have vastly different recurrence rates. ⁶

Inaccurate risk estimation among patients at lower risk for recurrence can result in more frequent testing that is burdensome, and may lead to overuse of preventative dietary and pharmacologic interventions with associated side effects. Conversely, inaccurate risk estimation among patients at a higher risk of recurrence may lead to undertreatment with preventative interventions, contributing to future stone recurrence events. Therefore, individualized estimation of stone recurrence risk, if accurate, allows the clinician to tailor the preventative interventions to the risk.

Nomograms have been published to improve the accuracy of recurrence risk estimation. ^4,6 The most widely used of these is the recurrence of kidney stones (ROKS) nomogram, which estimates recurrence based on 11 items in the medical history. ⁴ However, many clinicians rely on experience and clinical gestalt to estimate an individual patient's risk of recurrence. The commonly cited estimate of 50% recurrence at 10 years applies to a select population of stone formers and may mislead clinicians when addressing individual patients. ⁴ The effect of integration of nomograms into clinical workflow is not known to date, and no comparison has been performed between nomogram and clinicians' estimates of kidney stone recurrence rates. In addition, the impact of an individual patient's stone recurrence risk on how clinicians perceive appropriateness of intervention and surveillance has not been assessed. Therefore, we sought to characterize stone recurrence estimates using nomograms compared with clinical judgment.

Materials and Methods

Urologists with clinical expertise in kidney stone disease were surveyed in April and May 2020 after obtaining institutional review board approval (IRB#200435). An electronic questionnaire was sent to the international e-mail list of Endourological Society members with expertise in kidney stone disease. Demographic information on respondents was collected, including geographic location, year of residency graduation, and any fellowship training related to stone disease. After this, three hypothetical clinical vignettes were provided (Supplementary Appendix SA1), each detailing a patient's presentation with a kidney stone, and including relevant medical history. Enough information was included in each vignette to fully calculate the recurrence risk at 5 years using the ROKS nomogram, although respondents were not prompted to do so. Respondents were then asked to estimate the risk of recurrence of a symptomatic kidney stone episode within 5 years from 0% to 100% in 10% increments.

Respondents were then surveyed on their use of nomograms in clinical practice, type of nomogram used, if applicable, and reason(s) for not using nomograms. Finally, respondents were asked how their clinical management would differ for patients with a high compared with a low risk of kidney stone recurrence. Statistical analysis was performed using a one sample Wilcoxon rank sum test comparing respondent estimates with the ROKS estimate (RStudio). ⁴

Results

The survey was filled by 318 respondents, with 261 complete and 57 incomplete surveys. The survey response rate was 13% based on the mailing list size estimate provided by the Endourology Society of 2500. The demographic information of respondents is outlined in Table 1. The majority of respondents were from North America (n = 127, 40%), followed by Europe (n = 67, 21%), Asia (n = 38, 12%), and South America (n = 37, 11%). North American respondents were distributed across all American Urology Association (AUA) sections, with a maximum of 26 (20%) in the South Central section and a minimum of 8 (6%) in the New York and Mid-Atlantic sections. The majority of respondents were in an academic/university practice (n = 193, 61%), with 92 (29%) in solo/community/private practice and 30 (10%) working in a government facility. Most respondents graduated during or after 2001 (n = 194, 62%) and had fellowship training in stone disease after residency (n = 195, 63%). More than half (n = 171, 55%) of respondents dedicate at least 50% of their practice to stone disease.

Respondent and ROKS nomogram estimates of symptomatic kidney stone recurrence within 5 years for each of the cases are shown in Figure 1. The most common clinician estimated risk of recurrence for all three cases was 50% at 5 years. In all cases, the respondents' estimates were significantly different than the ROKS predicted recurrence rate (Case 1, 50% [interquartile range; IQR 30%–50%] vs 93% p < 0.0001; Case 2, 50% [IQR 40%–70%] vs 60% p < 0.0001; Case 3, 60% [IQR 40%–70%] vs 22% p < 0.0001).

OPEN IN VIEWER

FIG. 1.

Estimated chance of recurrence of symptomatic kidney stone episode within 5 years by number of respondents. Each panel represents 1 of the 3 numbered cases. Means were Case 1, 48%; Case 2, 53%; Case 3, 58%. ROKS nomogram estimates (red arrows) were Case 1, 93%; Case 2, 60%; Case 3, 22%. All differences between respondent and ROKS estimates were significant, p < 0.0001. ROKS = recurrence of kidney stones. Color images are available online.

The difference between the maximum and minimum estimate of risk for each respondent was calculated to determine the range of risk estimates for each individual clinician (Fig. 2). Nearly all of the respondents who answered multiple cases (n = 271, 99%) had a narrower range in risk estimation than the ROKS nomogram's range of 71% for the clinical vignettes. The median range of estimates for clinicians was 20% (IQR 20%–40%). Twenty-two respondents (8%) estimated the same risk for all patients.

OPEN IN VIEWER

FIG. 2.

Absolute difference between maximum and minimum estimated recurrence rate between cases for each participant, grouped by size of estimated range. Absolute difference between ROKS nomogram estimates of recurrence rate between cases labeled in red. ROKS = recurrence of kidney stones. Color images are available online.

Respondents' self-reported use of nomograms in clinical practice is given in Table 2. Overall, 14 respondents (5%) reported using nomograms in practice, of whom 12 used the ROKS nomogram. Of the remaining two respondents, one used a personal nomogram and one did not provide the nomogram name. When queried on reasons for not using a nomogram, the most common reason was lack of known useful nomograms (n = 150, 59%), and the second most common reason was the opinion that nomograms would not be useful (n = 69, 27%).

Respondents reported that their intensity of work-up and treatment would differ among patients if they had a high vs low recurrence risk (Table 2). For example, a patient with an 80% risk of recurrence would have a more intense work-up and treatment than a patient with a 20% risk of recurrence. Specifically, metabolic testing would differ for 221 (83%) of respondents, frequency of surveillance imaging for 226 (85%), modality of surveillance imaging for 147 (55%), and frequency of office visits for 205 (77%).

Discussion

Overall, these data show several important findings. First, most urologists (95%) with expertise in kidney stone disease primarily use clinical judgment to estimate the risk of stone recurrence rather than relying on nomograms. Only 5% of respondents use nomograms clinically to predict recurrence risk for patients. This is notable, since clinical judgment compared with nomogram estimates was markedly different. Second, clinicians may not be able to discriminate between those patients with high and low recurrence risk. In the clinical scenarios, the nomogram predicted estimates ranged from 22% to 93%, whereas the median urologist-derived 5-year risk estimates for each case were narrowly from 50% to 60%. These estimates were nearly identical for fellowship and nonfellowship trained urologists, with a median 50% recurrence risk estimated by fellowship and nonfellowship urologists in all cases except Case 3. In this case, fellowship trained clinicians were less accurate (median estimate 60%, IQR 40%–70%) than non-fellowship trained clinicians (median 50%, IQR 50%–72.5%) with respect to concordance with the ROKS estimate (22%). The similarity between fellowship and non-fellowship groups may be because of the high percentage of stone practice in both groups, and membership in the Endourological Society. Third, at the least, these data show that clinicians do not agree on recurrence risk at the individual patient level. The clinical estimates of kidney stone recurrence varied substantially among urologists for each case. Clinician estimates ranged from 10% to 100% for Cases 1 and 2, and from 10% to 90% for Case 3. Overall, these findings have clinical relevance, since the clinicians reported that they would alter their investigations and surveillance based on perceived recurrence risk. For example, challenges in estimating recurrence risk may, in part, explain variation in rates of follow-up surveillance ^7,8 and preventative treatment. ⁹

There are barriers to broader dissemination and use of predictive nomograms among urologists. One issue is the perceived or actual time burden of nomogram use. In this study, 19% (n = 47) of survey respondents were of the opinion that nomograms are too much work for routine use. Nomograms that require opening a separate window on the computer and manually entering patient data can be time consuming and prone to error; these also violate the principle of Five Rights of Clinical Decision Support (CDS), which recommends providing the right information to the right people, in the right intervention formats, through the right channels, at the right points in workflow. ^10,11 Instead, groups have shown success with integrating nomograms directly into electronic health records (EHRs) in fields such as pediatrics and psychiatry. ^12,13 However, widespread adoption of EHR integrated CDS tools, such as nomograms, is difficult, as the burden of integration is often placed on each individual institution. Strategies to improve uptake include sharing design process, configuration, and code between institutions. ¹⁴

A further barrier identified in this study is uncertainty as to the utility of nomogram estimates of kidney stone recurrence. A majority of respondents reported having no knowledge of useful nomograms (n = 150, 59%) and many reported a perceived lack of nomogram usefulness (n = 69, 27%). The ROKS nomogram (2014 edition) comprised 11 questions used to estimate risk of kidney stone recurrence (Supplementary Appendix SA2). ⁴ Whereas nomograms often suffer from a lack of external validation, ¹⁵ the ROKS nomogram has been externally examined. Using a different patient population, the ROKS nomogram was assessed and showed moderate predictive value for stone recurrence rates with an area under the curve (AUC) of 0.61 at 5 years. ¹⁶ However, the authors used the outcome of surgical intervention for stone recurrence, rather than overall symptomatic stone recurrence as was originally intended for ROKS. ¹⁶ The ROKS nomogram was also prospectively validated on a patient population in the same area it was created, and the AUC was 0.69 for symptomatic recurrence and 0.79 for any recurrence. ¹⁷ Future research is needed to refine and validate usable nomograms across different populations for clinical use.

Another finding from this study is that clinicians report that stone recurrence risk estimates impact clinical care. This is not surprising, but nevertheless necessary to demonstrate if investments are made toward developing CDS tools. The majority (85%) of survey respondents indicated that they would alter the frequency of surveillance imaging for a given patient if they had a low compared with high recurrence risk. Whether or not metabolic testing such as a 24-hour urine collection would be ordered differed for 83% of respondents based on a patient's risk. It is likely that more accurate estimation of risk would help clinicians to determine an individual patient's need for such an investigation. The frequency and modality of follow-up imaging and the frequency of office visits would also be altered by the majority of respondents based on a patient's recurrence risk (Table 2). This would have an impact on the total cost of investigating and treating stone disease.

The methodology of this study has several limitations. The respondents to this survey, who were mostly fellowship trained endourologists, may not represent the views of clinicians from other backgrounds in managing patients with kidney stones. The response rate of 13% is similar to other endourology surveys, ¹⁸ and may indicate the time limitations of endourologists. The membership of the Endourological Society in general is self-selected, and further selected among those who responded to the survey invitation. The definition of kidney stone recurrence was one of many possible definitions used in the literature, which could affect clinician estimates or prevent generalizability of these results to other types of asymptomatic stone recurrence. ¹⁹ In addition, it is not known which estimates, those derived from clinical judgment or those from the ROKS nomogram, more accurately reflect the true recurrence risk. Although the ROKS nomogram was derived from a large longitudinal cohort study, the AUC ranges from 0.61 to 0.79 in two separate external validations. Since the original ROKS nomogram was released in 2014, there has been a subsequent update in 2018. ^4,6 In this study, we chose the initial version because it has fewer questions, and we expected clinicians to have had more time to adopt its use.

Conclusions

Since recurrence risk varies across patients, it is necessary to accurately individualize and personalize care for a given patient. This study shows that clinicians and nomograms are not concordant for predicting stone recurrence risk. Furthermore, clinicians do not appear to discriminate among those at high and low risk for recurrence, and there is disagreement among clinicians on recurrence risk at the individual patient level. Clinicians reported that the perceived risk of ROKS affects the intensity of follow-up and surveillance offered to patients. Therefore, the discrepancies between risk estimates have implications for patient care. The barriers to adopting nomograms need further investigation, and prompt the need for future study of how to implement clinical support tools to integrate these tools into clinical workflow.