Automated Computer Software Compared with Manual Measurements for CT-Based Urinary Stone Metrics: An Evaluation Study

Introduction

The management of nephrolithiasis is largely driven by CT-based assessment of stone size and density. These factors play an important role in evaluating the probability of spontaneous stone passage, predicting success of treatment, and monitoring progression of stone disease over time. ^1,2 Of the various imaging modalities, noncontrast CT scan (NCCT) is the most sensitive and specific in diagnosing urinary stones and furthermore provides information regarding stone size, density, and patient anatomy. ^{3 –6}

Despite the widespread use of NCCTs in the evaluation of urolithiasis, a standard method of measuring stone size has not been established. ^{7 –9} Many different techniques have been described such as measuring in a single plane (axial, coronal, or sagittal), using two dimensions (length × width), or reporting the volume of a stone. ^{10 –13} However, there is a reported discrepancy between the CT measured size of stones and their actual size. ⁹ Stone volume is a way to combine measurements from multiple orientations and can be crudely estimated from manual measurements using the following ellipsoid formula (0.52 × length × width × height). ¹⁴ This formula assumes an ellipsoid shape and many stones are not ellipsoid, and will have significant differences in measurements taken in an axial view compared with a coronal or sagittal view. The differences in measured length can lead to significant differences in reported stone size and subsequent clinical management. ⁹ Indeed, data have shown that when a stone is measured in the same plane by three different radiologists, the length obtained varies by an average of 28%. ¹⁵ To standardize measurements, objective automated processes have been reported, which eliminate the interobserver variability. ¹⁶

We have shown that coronary artery CT volume software can be modified to measure urinary stone volume. ^15,16 A recent development is a novel software program that provides automated stone length, volume, and density calculations. It eliminates interobserver variability and creates a precise method of measurement compared with the variability using manual length measurements. ¹⁵ The goal of this study was to compare the automated measurements of this novel software with standard manual measurements, which currently represent the most common clinical method of stone measurement.

Methods

To test the comparability of the software measurements vs a human reader, a sample of patients was chosen who had undergone prior NCCT imaging that demonstrated urolithiasis. We selected 17 patients who had a total of 42 CT scans. From these imaging studies, 85 discrete stones were identified and subsequently analyzed. The stones used in this study represented a variety of sizes, shapes, and locations. Stones were measured manually, and then, the CT image files were imported into the stone imaging software for subsequent analysis.

All stones were measured manually by a single reader (J.R.B.) using abdominal windows with magnification. Stone length and width were obtained using axial imaging and recorded in millimeters. Coronal reconstructions were used to measure the height of the stone. The longest diameter of the stone was recorded as the longest value of the measurements obtained in either the axial or coronal images. Stone volume was calculated from the manual measurements using the formula (0.52 × length × width × height). ¹⁴ Stone density information was obtained using an elliptical region of interest (ROI). The ROI was drawn inside the stone and created to fill as much of the stone as possible. The maximum Hounsfield unit (HU) obtained from the ROI was recorded and used for analysis. All manual measurements were taken independently from and before the software measurements.

A new, experimental version of our previously described dedicated stone volume software tool was used for all automated measurements (Ziosoft, Inc., Tokyo, Japan). ^15,16 The same stones that were assessed by manual measurements were imported into the software and analyzed by the software. The software automatically detects objects that contain a CT attenuation value ≥200 HU (Fig. 1). This threshold density is set to capture the vast majority of stones while minimizing background artifact related to image noise, which is more pronounced on low-dose scans. The user then clicks on the stone of interest and the software automatically and rapidly provides the stone metrics, including the maximum stone length, mean density, and overall volume. The software provides the largest dimension of the selected stone regardless of stone orientation and independent of CT reconstruction. The stone volume is calculated by the software by adding all pixels contiguous with the selected stones that contain an HU ≥200.

OPEN IN VIEWER

FIG. 1.

Screenshots from the radiology postprocessing software are shown below. In (a), the axial and coronal CT images are shown. The bottom portion shows an automatically produced three-dimensional reconstruction showing any structure with an HU ≥200. The user clicks on the stone of interest to generate the screenshot seen in (b). (b) Shows the urinary stones that have been identified by the reader. The sidebar shows the software-generated metrics, including length, density, and volume. (c) Shows a comparison screen through which the same stone was assessed for growth over time on sequential CT images. HU = Hounsfield unit.

Results

Eighty-five unique stones were identified and analyzed. These stones represented a variety of sizes, shapes, and locations. Nine of these stones were located in the ureter, 1 was in the bladder, and the remaining 75 stones were located in the renal pelvis or intrarenal collecting system. The mean manual stone length was 8.0 mm (range 1.9–21.0 mm) and the mean automated stone length was 10.1 (range 3.5–22.5 mm). The average maximal manual density was 686 HU (range 126–1492 HU) with the automated maximal density averaging 754.8 (range 232–1526 HU). The mean manual calculated stone volume was 192 mm³ (range 2.9–2555 mm³), and the automated values averaged 182.5 mm³ (range 2.8–2668.4 mm³). There was no difference between the manual and automated measurements for density (686 HU vs 755 HU, p = 0.177) and stone volume (192 mm³ vs 183 mm³, p = 0.858). The manual stone lengths were slightly shorter compared with the automated (8 mm vs 10 mm, p < 0.003; Table 1).

The mean percent difference between the manual and automated stone density was 14.3% ± 18.3%. The mean percent difference in stone volume was 21.0% ± 15.9%, and the mean percent difference in stone length was 25.2% ± 17.0% (Fig. 2).

OPEN IN VIEWER

FIG. 2.

Graphs comparing the percent differences between manual and automated measurements. In the graphs below, the x-axis represents the stone number. Graphs are shows for the comparisons of density (a), volume (b), and length (c).

The automated software was able to report the same measurements for stones regardless of who operated the program and regardless of how many times the measurements were repeated.

Discussion

In this preliminary clinical study, we compared the manual stone measurements obtained by a single reader with the data provided by a dedicated stone analysis software. The automated stone software program rapidly assessed the length, density, and volume of urinary stones with a single click of a button while closely approximating the manual measurements obtained.

Stone size is important in the management of urolithiasis as it determines the probability of stone passage, guides the modality of stone treatment, and guides the decision to progress from active surveillance to surgical management. In all of these situations, small differences in length or volume can significantly alter the course of management. Unfortunately, current imaging practices rely on nonstandard and imprecise manual measurements. This leads to a discrepancy in clinical decision-making and research reporting. Prior work has shown that even among board-certified radiologists, there is an average percent difference of 28% between linear stone measurements. ¹⁵ Based on these data, we must conclude that our ability to manually assess stone size using CT is imprecise. This lack of standardization in measuring and reporting stone size affects our ability to properly counsel our patients on the likelihood of spontaneous passage, select the appropriate treatment modality, and monitor stone growth in patients undergoing stone surveillance. ^{7 –9} An ideal method of measuring urolithiasis would provide accurate and precise measurements, be versatile in its ability to measure stones of various shapes and sizes, have the potential to be obtained quickly, and work on extremely low-dose CT scans.

A novel solution to this problem has been to use computer software to measure urinary stones. Demehri and colleagues ¹⁷ reported that computer software can closely approximate the actual size and volumes of urinary stones, thus supporting the use of software assessment of urinary stones on CT imaging. Patel and coworkers ¹⁶ showed there was no interobserver variability in the software measurements of urinary stones using a precursor software program. We tested an updated and revised version of the dedicated postprocessing stone imaging software used by Patel and associates ^15,16 and compared these results with those obtained by a human reader using current standard methods of stone measurement.

Kishore et al. ⁹ have reported on the size discrepancy between CT measurement of urinary stones compared with the actual measurements once these stones were passed and collected. Therefore, although manual measurements are our current standard, they are not perfect. We choose to compare the manual and automated measurements and then use a percent difference calculation to quantify the differences for each stone between the measurement modalities.

The most concordant measurement in our study was the maximum density (HU) of the stone. There is typically a significant difference between the maximum density of a urinary stone and the surrounding soft tissue. This allows for greater precision between the automated and manual measurements. As long as the reader encircles the majority of the stone or even if the ROI is drawn outside the stone, the maximum stone density would vary little. We choose to compare the maximum density as this guides management decisions. Stone density has been shown to be an important predictor in the success of shockwave lithotripsy. ¹⁸ We found four examples of stones in our study that manually measured less than 1000 HU, but had automated measurements >1000 HU. We hypothesize that the difference recorded represents a sampling error during manual measurements. However, this exactly highlights the need for an automated process that can minimize or eliminate this sort of sampling error as this sort of discrepancy could alter treatment decisions.

With regard to stone volume, the automated measurements did not statistically differ from the manual measurements and overall differed by 21%. Small linear differences in three-dimensional (3D) objects can result in exponential differences in volume. As such, CT volume studies are considered comparable as long as the volumes are within 20% variation. ¹⁹ Therefore, our data show comparability between the manual and automated measurements. Stone volume may have strong clinical implications as it has been shown to better predict spontaneous stone passage, ²⁰ success of SWL, ²¹ and operative time required for ureteroscopy ²² compared with linear measurements. The use of stone volume has traditionally been limited due to the difficulties in obtaining an accurate value. Some authors have reported outlining stones in each sequential image to obtain a more accurate volume. ²¹ Others have reported using formulas to simplify the process. ¹⁴ However, the use of a formula neglects the complex geometric shapes that stones can represent. Certainly, staghorn stones do not adhere to normal geometric formulas. While outlining each CT image of a stone may more accurately portray the actual geometry, this is tedious and time-intensive and therefore limits its application to clinical practice. The volumetric differences seen in our study likely resulted from using a formula that assumed an elliptical shape when the stone geometry did not align to this shape. An automated method of measuring stone volumes may allow for further clinical outcome studies as it dramatically simplifies and standardizes the process while approximating manual volume calculations.

Stone length differed from manual measurements by 25.2%, which is similar to the measurement differences reported between radiologist readers. ¹⁵ In general, we found that the software yielded longer diameters than those obtained by the reader. This may be because the software has the advantage of being able to measure the longest distance between two points in any dimension, whereas the human reader is limited to the traditional axial, coronal, or sagittal reconstructions. We hypothesize that the software measurements may in fact be closer to the true stone size because of this, whereas manual measurements likely underestimate the true size because of CT reconstruction limitations. Our study does not allow us to answer this question as we do not know the true stone size. However, Lidén and colleagues ²³ showed that the length in 3D reconstructions more accurately represents the true length than that obtained using the traditional axial or coronal measurements.

The software measurements proved to be reproducible and therefore eliminated any variability between readers. When repeated measurements were obtained, the result was the same every time. In contrast, manual measurements are inherently variable and therefore create a poor clinical and research standard. The automated software method produces results with precision and can measure the stone's length in any orientation. Furthermore, the software was able to identify and measure stones in the ureter and bladder making this a versatile platform. The precision offered by the automated results could provide an improved measuring and reporting standard. This aspect makes the software valuable to a number of clinical specialties, including radiologists, urologists, and emergency physicians, as all parties can obtain identical and clinically useful stone metrics. Having precise measurements is especially important when monitoring stone size in patients undergoing surveillance. Our software was able to identify and assess stones in the renal collecting system, throughout the ureter and even bladder stones, allowing us to obtain metrics on stones in a wide variety of locations. Currently, the most common method of reporting stone size is with a single length measurement. This can be problematic as only viewing or representing the stone in a single plane may not properly represent the true size or geometry. One can therefore be misled into assigning the same size to two stones, which are in actuality extremely different in size. For example, stones that appear similar in a set of axial images may, on further review, be found to have different rostral–caudal lengths and hence extremely different volumes. ²¹

We recognize our study has limitations. This study was conceived and designed as a preliminary validation study to determine if the new version of the software approximated manual stone measurements. Therefore, the patients selected do not represent a prospective or randomized sample. However, we did study stones with a variety of shapes, sizes, and locations, and thus, we believe the results show that the software does indeed approximate manual measurements for a variety of stones in multiple locations. As our study utilized historic CT imaging as the source of information, we cannot speak of the true accuracy of the automated software measurements. We did not compare our measurements against stone phantoms and did not have the actual stones for comparison. Future studies using stones of known length, volume, and density certainly could further validate the accuracy of our data. However, given that urologists often do not have this information, we believed it important to compare the software with traditional manual measurements as these are the current standard.

Conclusions

Automated CT-based stone measurement software provides a promising approach to standardize the radiographic evaluation of urinary stones. This technique is highly reproducible and eliminates interobserver variability while accounting for stones of varying sizes and geometric characteristics. This technology offers future promise to improve clinical decision-making in both the surveillance and treatment of urinary stones.