Assessing the Accuracy of Endoscopic Estimates of Lesion Size in Urology Using In Vitro Models of the Urinary Tract

Abstract

Background and Purpose:

Despite the frequency at which urologists endoscopically estimate lesion size, their accuracy has not been established. Our objectives were to determine the accuracy of cystoscopic and ureteroscopic estimates of lesion size using in vitro models of the urinary tract and to assess potential impacting factors.

Methods:

Eleven staff urologists and 9 urology learners performed cystoscopy on a series of pig bladders containing mock papillary and flat lesions. Each provided three sets of size estimates: two using only the cystoscope to assess intraobserver agreement and the third with the aid of a ureteral catheter as a visual reference. Similar estimates were made with a flexible ureteroscope on papillary lesions within an inorganic upper urinary tract model. Differences in mean estimates and the agreement between repeated estimates were assessed.

Results:

The level of endoscopic training did not influence the mean error of estimation (MEE) for either cystoscopy or ureteroscopy regardless of lesion size and appearance. Staff and learners consistently underestimated lesion size with median errors of 34% and 43%, with excellent (median intraclass correlation coefficient [ICC] of 0.97) and fair (median ICC of 0.56) reproducibility for cystoscopy and ureteroscopy, respectively. Use of the visual reference during cystoscopy did not improve the MEE.

Conclusions:

Urologists, regardless of their level of training, substantially underestimate lesion size by 34% to 43%. These findings are independent of lesion size and appearance, and the use of a visual reference during cystoscopy. Recognizing this tendency and adjusting estimates accordingly or improving instrumentation should improve clinical and operative decision-making.

Introduction

U rologists frequently estimate the size of urinary tract lesions endoscopically and use these estimates to yield prognostic and therapeutic information to guide patient treatment.

Risk stratification and/or predicted response to treatment of patients with either upper or lower tract urothelial cancer (UC) rely on an urologist's ability to estimate tumor size endoscopically.^1

–5 The ability to endoscopically estimate the size of calculi and other lesions throughout the urinary tract also impacts their management. Urologists estimate stone size during cystoscopy and ureteroscopy and frequently make preoperative and intraoperative adjustments to their treatment approach.^6

–9 Accurate estimates of stone size also have implications in terms of operative case planning, surgical approaches, and billing in some centers.¹⁰ Endoscopic size estimates are also an important part of the everyday descriptions of less well-defined lesions such as urethral or ureteral strictures, diverticula, and mucosal abnormalities such as patches of leukoplakia, cystitis, or carcinoma in situ.

Despite the day-to-day endoscopic estimation of lesion size in urology, the accuracy of such estimates has not been established. Objectives of this study were to determine the accuracy of cystoscopic and ureteroscopic estimates of lesion sizes and to assess their variance with the level of endoscopic training, use of a visual reference, as well as lesion size, appearance, and location.

Methods

The study was approved by our institution's research ethics board. It consisted of two parts outlined in Figure 1 and included 11 staff urologists, 9 learners (1 fellow and 8 residents) from our department. Level of training varied among the residents from 1 to 5 years of urological training. Sample size was determined by the maximum number of available participants from within the Department at the time of the study.

FIG. 1.

Summary of study methodology. Part 1 (left arm) represents the cystoscopy portion of the study. Part 2 (right arm) represents the ureteroscopy portion of the study. The thick arrows represent the measure of interest at each series of estimates.

The first part assessed the accuracy of participants' endoscopic estimates using a 22F rigid cystoscope and 30-degree lens (Karl Storz Endoscopy Canada Ltd.). Seven pig bladders and urethras were acquired from a local butcher. One papillary surrogate consisting of a ball bearing, with or without a washer representing a flat lesion, was placed in each bladder. These were of predetermined sizes (ball bearings: 3.2, 6.4, 9.5, 12.7, 15.9, 25.4, and 50.8 mm; washers: 3.2, 6.4, 9.5, 12.7, 15.9, and 25.4 mm) and were selected because they possessed no visual reference that would indicate their approximate size and represented lesion sizes typically encountered in the clinical setting. The 50.8 mm ball bearing was selected to mimic a large bladder stone. Each bladder was mounted in a Plexiglas enclosure. Participants were blinded to the true size of these lesions, which were randomly distributed among seven bladders.

Participants performed videocystoscopy on the seven bladders in three series. They provided estimates for each lesion they encountered from which a percentage error of estimation was calculated as: \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} {Error \ of \ Estimation} (\% ) = \frac {(True \ Lesion \ Size - Estimated \ Lesion \ Site)} {True \ Lesion \ Size} \times 100\end{align*} \end{document}

This was used as the measure of estimate accuracy in this study.

Participants then performed a second series of size estimates with the bladders in a different and randomized order to minimize potential recall bias. The estimates from the first two series were used to calculate the intraclass correlation coefficient (ICC) for each participant to determine the agreement between repeated estimates.

Last, the participants performed a third series of lesion size estimates with randomized bladders. During this series of estimates, participants were provided with a 5F ureteral catheter of known diameter (1.7 mm) and tip length (7 mm) to use as a visual reference to align with the lesions during cystoscopy and potentially improve their estimates. A ureteral catheter was selected because it was an inexpensive, disposable device that could be realistically used by an urologist in general practice to measure a variety of bladder lesions without the need for additional equipment. Errors in estimation for each participant were calculated for each lesion and compared with their values from the first series to identify any improvement in accuracy.

The second part assessed the accuracy of participants' endoscopic estimates using a 7.5F flexible ureteroscope (Karl Storz Endoscopy Canada Ltd.). Participants were asked to estimate the sizes of several ball bearings (i.e., papillary lesions) (ball bearings: 3.2, 6.4, 9.5, 12.7, and 15.9 mm) implanted in several locations within a three-dimensional inorganic ureteroscopy model (Cook Medical) shown in Figure 2. Three 6.4 mm ball bearings were placed within the model (one each in a renal pelvis, renal calyx, and ureter) to determine whether the confines in which a lesion was visualized impacted estimate accuracy. For practical purposes, including the limitations of a 7.5F flexible ureteroscope, as well as translatability to the in vivo setting, no visual reference was used during ureteroscopy.

FIG. 2.

Inorganic urinary tract model used to facilitate ureteroscopic size estimates. The size and location of each papillary lesion is shown numerically in the diagram above in mm. Participants performed ureteroscopy by introducing the ureteroscope in one of the three tubes at the bottom of the image.

Participants were blinded to the contents of the model and asked to provide two sets of estimates for each lesion. An error for each estimate of size was then calculated. To minimize recall bias, the participants were directed to visualize each lesion in a randomized order and to provide the first and second series of ureteroscopic estimates before and after performing Part 1 of the study, respectively. Using the estimates from the two series of ureteroscopic estimates, the ICC for each participant was again calculated to determine estimate reproducibility.

Statistical analysis was performed using the R Language for Statistical Computing (R Core Team).¹¹ Welch two sample t tests were used to compare differences in the mean error of estimation (MEE) between staff and learners for all lesions. Paired t tests were used for paired comparisons between same-sized papillary or flat lesions during cystoscopy and between same-sized papillary lesions in different locations during ureteroscopy. ICCs were also computed using the “irr” package in R.¹²

Results

All 20 participants provided a complete series of estimates. Participating residents included one first year, three third years, two fourth years, and two fifth years.

During cystoscopy both staff and learners consistently underestimated lesion size across all sizes and appearances. Overall median errors in estimation for staff and learners were 30% (range 6%–48%) and 38% (range 21%–53%), respectively, for the first series of cystoscopic estimates. Overall median ICCs for staff and learners during cystoscopy were 0.96 and 0.97, respectively, both representing excellent agreement between repeated estimates (Fig. 3).

FIG. 3.

Overall intraclass correlation coefficients for all participants, learners, and staff for cystoscopic and ureteroscopic repeated estimates of lesion size.

For all participants, and lesion sizes and appearances, the overall median error in estimation during cystoscopy was 34% (range 6%–49%) (Fig. 4), and the overall median ICC was 0.97 (Fig. 3). Figure 4 also shows no trend in the median errors with increasing lesion size for either papillary or flat lesions.

FIG. 4.

Cystoscopic error of estimation box plots for all participants based on lesion appearance and size. Ball bearings and washers represent papillary and flat bladder lesions, respectively. *Error (%)=(True Lesion Size−Estimated Lesion Size)/True Lesion Size×100. Therefore, a positive error value on the plot represents an underestimate.

Comparisons of estimates by staff and learners of identical papillary and flat lesions revealed no statistically significant differences in the MEE (Table 1). These findings were consistent after removing senior learners (ie, fourth and fifth year residents and fellow) from the analysis (data not shown).

Table 1.

Comparison of the Mean Errors of Estimation for Staff and Learners for Papillary and Flat Lesions During Cystoscopy

	Mean error in estimation (%)
	Staff	Learners	P value	95% CI for difference in mean errors
Papillary (mm)
3.2	−9.4	25.8	0.12	−81.1 to 10.8
6.4	35.4	41.4	0.65	−33.9 to 21.9
9.5	21.5	31.6	0.46	−37.9 to 17.8
12.7	29.1	39.0	0.40	−34.5 to 14.8
15.9	31.4	37.1	0.71	−37.8 to 26.4
25.4	32.7	29.6	0.83	−28.7 to 34.9
50.8	39.5	32.3	0.52	−16.3 to 30.6
Flat (mm)
3.2	−2.3	10.2	0.46	−47.0 to 22.1
6.4	19.0	23.8	0.76	−37.8 to 28.3
9.5	44.5	35.5	0.67	−38.0 to 56.0
12.7	42.0	45.4	0.83	−36.2 to 29.5
15.9	26.8	15.1	0.47	−21.7 to 45.1
25.4	31.6	31.6	1.00	−30.4 to 30.5

CI=confidence interval.

For all participants, comparisons between same-sized flat and papillary lesions using paired t tests revealed few statistically significant differences in the MEE. The MEE for the 12.7 mm papillary lesion was significantly less than the 12.7 mm flat lesion (mean difference=−9.6 mm, P=0.045, 95% confidence interval [CI] −19.1 to −0.3), and the 15.9 mm papillary lesion was significantly more than the 15.9 mm flat lesion (mean difference=12.3 mm, P=0.048, 95% CI 0.12 to 24.4). These isolated findings, however, with their wide confidence intervals did not fit into an observable trend.

Using the ureteral catheter as a visual reference had no significant effect on the MEE for any lesion regardless of size or appearance for either staff or learners (data not shown).

Staff and learners consistently underestimated lesion size during flexible ureteroscopy for all lesion sizes. The overall median errors in estimation for staff and learners were 44% (range 37%–58%) and 38% (range 6%–56%), respectively, for the first series of ureteroscopic estimates. The overall median ICCs for staff and learners during ureteroscopy were 0.61 (fair agreement) and 0.47 (fair agreement), respectively (Fig. 3). The overall MEE during ureteroscopy for all participants and observed lesions was 43% (range 37%–56%) (Fig. 5) and the overall ICC was 0.56 (fair agreement) (Fig. 3). There was an observed upward trend in the median errors with increasing lesion size.

FIG. 5.

Ureteroscopic error of estimation box plots for all participants based on lesion appearance and size. c=located in a renal calix, p=located in a renal pelvis, u=located in a ureter. *Error (%)=(True Lesion Size−Estimated Lesion Size)/True Lesion Size×100. Therefore, a positive error value on the plot represents an underestimate.

For all lesion sizes and locations (ie, ureter, calyx, and pelvis), there was no significant difference in the MEE between staff and learners (Table 2). Similar to the cystoscopy portion of the study, these findings were consistent when senior learners were removed from the analysis (data not shown). On analyzing the MEE of all participants, no significant difference was identified between same-sized lesions located within the ureter, calyx, or pelvis of the model upper urinary tract (P values for paired comparisons among the lesion locations ranged between 0.47 and 0.80).

Table 2.

Comparison of the Mean Errors of Estimation for Staff and Learners for Papillary Lesions During Ureteroscopy

	Mean error in estimation (%)
	Staff	Learners	P value	95% CI for difference in mean errors
Papillary (mm)
3.2	29.0	37.3	0.34	−26.0 to 9.3
6.4	45.7	45.3	0.96	−13.9 to 14.6
6.4	36.8	32.1	0.60	−13.4 to 22.7
6.4	27.6	24.4	0.81	−23.6 to 29.9
9.5	52.6	43.1	0.28	−8.3 to 27.4
12.7	44.5	44.9	0.96	−15.4 to 14.7
15.9	57.1	51.5	0.47	−10.3 to 21.6

CI=confidence interval.

Discussion

Endoscopic estimates of lesion size are relied upon in many urologic settings including risk stratification for UC,^1

–5 treatment selection for urinary calculus disease,^6,7 as well as assessment and description of stricture disease, sizes of bladder diverticula, and patches of carcinoma in situ. Estimation of urologic stone and tumor sizes also has implications in terms of billing practices in some centers.¹⁰ Despite the frequency of estimating lesion size during cystoscopy and ureteroscopy, our study shows that the validity of clinical decisions based on these estimates is questionable.

We conducted in vitro experiments to assess the accuracy of observers' estimates of lesion size during cystoscopy and ureteroscopy with variables including lesion size, appearance, and location, and showed that despite the level of endoscopic training and use of a visual reference, all participants consistently underestimated true lesion size. The level of endoscopic experience had no influence on the accuracy or reproducibility of estimates. Even when the most senior learners were excluded from the analysis, there was no difference in accuracy between junior learners who had performed fewer than 100 endoscopic cases, and experienced staff urologists who had performed an order of magnitude more cases; however, the sample size for junior learners was small (n=4). Similar findings have been described in the gastroenterologic literature suggesting that endoscopic expertise does not influence the consistent underestimation of lesion size.^13
–15

Lesion characteristics (eg, flat or papillary) did not impact the accuracy of the estimate, implying a loss of depth perception during endoscopy. Use of endoscopic equipment that provides three-dimensional visualization, as in the da Vinci® robotic surgical system, may overcome this problem.

Neither staff nor learners demonstrated a significant improvement in the accuracy of their estimates with the use of the ureteral catheter as a visual aid. The cystoscope used in this study, which was limited to animal-use only, lacked mobile tines to deflect and position the ureteral catheter, however. Therefore, observers struggled to align the ureteral catheter with the lesion of interest, which limited its usefulness. This led to only a partially objective measure of lesion size, a shortcoming that has also been described with the use of linear probes in the gastroenterology literature.¹⁶

All of these findings reflect the inherent difficulty of accurately estimating size by visual inspection alone, particularly with endoscopes using wide-angle lenses and two-dimensional displays that limit the user's depth perception. The wide-angle lens of a cystoscope results in nonlinear changes in the appearance of the visual field—a phenomenon inherent to endoscopes known as “barreling”—with less compression centrally and increasing compression toward the periphery of the visual field. This makes objects in the periphery of the field of view appear smaller than their actual size.^16,17 As well, objects will appear disproportionally smaller as their distance increases from the cystoscope.¹⁵ The latter point helps to explain why the visual reference used in this study, which was difficult to align in the same visual plane as the object of interest, failed to improve the estimate accuracy. Similarly, the confines in which a lesion was lying during ureteroscopy did not alter the accuracy of its estimation, because the field of view of the ureteroscope is so narrow and the surrounding landmarks used as visual references (ie, walls of ureter, calyx, and pelvis) lay at the periphery of the field of view where the barreling effect is most pronounced.

Although participants had poor overall accuracy in estimating lesion size, they demonstrated fair (median ICC 0.56) and excellent (median ICC 0.97) reproducibility on repeated estimates during ureteroscopy and cystoscopy, respectively. This can be accounted for by the observed behavior of participants visualizing individual lesions at relatively consistent viewing distances to frame the objects comfortably within the endoscopic field of view. This behavior resulted in good reproducibility but did not prevent the consistent underestimation of lesion sizes, which is influenced more substantially by the loss of depth perception and the optical distortion of referential structures lying in the peripheral field of view. Based on this finding, the argument can be made for achieving both reproducible and accurate estimates if observers adjusted for their tendency to underestimate lesion size.

A logical next step, therefore, would be to use an in vitro system of endoscopic training that would allow endoscopists to learn/be taught to correct for the individual inaccuracy of their size estimates. Such a system has been described in the gastrointestinal literature during colonscopy^15,18 but not in a urologic setting where the opportunity for further study remains. In the absence of such a system, based on our findings, it would seem reasonable for endoscopists to make their initial size estimates during cystoscopy and ureteroscopy and then increase them by a factor of 30% and 40%, respectively; however, our findings would benefit from validation in a larger study population at other centers. Further, the availability of a better standard for endoscopic measurement comparisons may be beneficial.

A limitation of this study is its in vitro design using artificial “lesions” in pig bladders and an inorganic urinary tract model. This environment does not precisely reproduce that observed in vivo, but does provide a fluid-filled setting in which endoscopic maneuvers and observations are similar to those performed in real patients. The observed lesions consisted of ball bearings and washers, which were similar in a number of ways to real lesions found in the urinary tract; they varied in size, depth, and location while possessing no surface features that could reveal their true size to an observer. Studies in the gastrointestinal endoscopy literature have used ball bearings as surrogates for colonic polyps, and although their ability to accurately represent pathologic lesions has not been established, their utility as “training lesions” to improve endoscopic accuracy has been shown.^13,15

The number of participants in this study was small and was limited by the number of staff and learners in our department. One might argue that a difference in accuracy based on level of training might have been revealed with a greater numbers of participants. Nevertheless, our findings are similar to those of studies in the gastroenterologic literature, which also found no difference in accuracy with level of training.^13
–15 Generalizing our findings to a broader audience of urologists would necessitate validating our findings with a larger population at multiple centers and would be an appropriate next step. There was the potential for recall bias with repeated estimates, but this was minimized by randomizing the observations.

Conclusion

Urologists, regardless of their level of training, substantially underestimate lesion size observed during cystoscopy and ureteroscopy, despite fair to excellent reproducibility of their estimates. These findings are independent of lesion size and appearance and the use of a visual reference during cystoscopy. Recognizing urologists' tendency to underestimate lesion size by 34% to 43% and adjusting estimates accordingly or improving instrumentation should improve clinical and operative decision-making based on these estimates.

Footnotes

Disclosure Statement

No competing financial interests exist.

Abbreviations Used

References

Babjuk

, Oosterlinck

, Sylvester

et al. EAU guidelines on non-muscle-invasive urothelial carcinoma of the bladder, the 2011 update. Eur Urol, 2011; 59:997–1008.

Hurle

, Losa

, Manzetti

, Lembo

. Intravesical bacille Calmette-Guerin in Stage T1 grade 3 bladder cancer therapy: A 7-year follow-up. Urology, 1999; 54:258–263.

Lebret

, Gaudez

, Hervé

et al. Low-dose BCG instillations in the treatment of stage T1 grade 3 bladder tumours: Recurrence, progression and success. Eur Urol, 1998; 34:67–72.

Raman

, Ng

, Boorjian

et al. Bladder cancer after managing upper urinary tract transitional cell carcinoma: Predictive factors and pathology. BJU Int, 2005; 96:1031–1035.

Matsui

, Utsunomiya

, Ichioka

et al. Risk factors for subsequent development of bladder cancer after primary transitional cell carcinoma of the upper urinary tract. Urology, 2005; 65:279–283.

Papatsoris

, Varkarakis

, Dellis

, Deliveliotis

. Bladder lithiasis: From open surgery to lithotripsy. Urol Res, 2006; 34:163–167.

Razvi

, Song

, Denstedt

. Management of vesical calculi: Comparison of lithotripsy devices. J Endourol, 1996; 10:559–563.

Bhatia

, Biyani

. A comparative study of cystolithotripsy and extracorporeal shock wave therapy for bladder stones. Int Urol Nephrol, 1994; 26:26–31.

Ener

, Agras

, Aldemir

et al.

The randomized comparison of two different endoscopic techniques in the management of large bladder stones: Transurethral use of nephroscope or cystoscope?

J Endourol, 2009; 23:1151.

10.

AMA: CPT - Current Procedural Terminology. Chicago: American Medical Association, 2012

11.

R.Core.Team: R: A Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing, 2012

12.

Matthias

, Lemon

, Fellows

et al. irr: Various Coefficients of Interrater Reliability and Agreement., 0.84 ed, p. R package. 2012

13.

Margulies

, Krevsky

, Catalano

. How accurate are endoscopic estimates of size? Gastrointest Endosc, 1994; 40:174–177.

14.

Fennerty

, Davidson

, Emerson

et al.

Are endoscopic measurements of colonic polyps reliable?

Am J Gastroenterol, 1993; 88:496–500.

15.

Schwartz

, Catalano

, Krevsky

. Endoscopic estimation of size: Improved accuracy by directed teaching. Gastrointest Endosc, 1995; 42:292–295.

16.

Gopalswamy

, Shenoy

, Choudhry

et al.

Is in vivo measurement of size of polyps during colonoscopy accurate?

Gastrointest Endosc, 1997; 46:497–502.

17.

, Nie

, Soto-Thompson

et al. Robust distortion correction of endoscope. Presented at the The International Society for Optical Engineering - Medical Imaging, San Diego, CA, April 14, 2008.

18.

Chang

, Chiu

, Wang

et al. An endoscopic training model to improve accuracy of colonic polyp size measurement. Int J Colorectal Dis, 2010; 25:655–660.