Abstract
To reduce the radiation dose and the equipment cost associated with lung CT screening, in this paper we propose a mixed reality based nodule measurement method with an active shutter stereo imaging system. Without involving hundreds of projection views and subsequent image reconstruction, we generated two projections of an iteratively placed ellipsoidal volume in the field of view and merging these synthetic projections with two original CT projections. We then demonstrated the feasibility of measuring the position and size of a nodule by observing whether projections of an ellipsoidal volume and the nodule are overlapped from a human observer’s visual perception through the active shutter 3D vision glasses. The average errors of measured nodule parameters are less than 1 mm in the simulated experiment with 8 viewers. Hence, it could measure real nodules accurately in the experiments with physically measured projections.
Introduction
Lung cancer is a main cause of cancer death for both male and female in the United States [1, 2].Early detection of lung nodules/tumors is important. Currently, x-ray computed tomography is an effective tool for early screening of lung cancer [3]. The normal nodule measurement is performed by scanning the whole chest to generate an image volume of many slices. After image reconstruction, a radiologist views the images to detect and quantify nodules/tumors.
CT can generate high-quality images but it requires a high radiation dose at a high hardware and software cost. Alternatively, x-ray stereo-tomographic imaging methods could be used to quantify nodules and other features in 3D but from 2 views. In 1996, an x-ray stereo imaging study was described [4], which is in the cone-beam geometry. Also, a novel projection display system was reported based on a virtual reality enhancement environment [5]. Nodule detection was investigated with lung CT images through stereo-viewing using the distance-weighted averaging, distance-weighted maximum intensity projection and conventional maximum intensity projection methods [6]. In 2011, the relative efficiencies were compared of a stereographic display scheme and two monoscopic display schemes for CT lung nodule detection [7]. Another study was done to evaluate between stereoscopic visualization and standard posterior-anterior (PA) image display [8].
With the emergence of 3D movies [9], 3D techniques attracted more and more attention in daily life. The basic principle of 3D vision is the human’s eyes sees an object in 2 angles, and merges the two images to form a 3D perception. The popular 3D movie systems include the projector or monitor which displays 2 images and 3D glasses which make each eye see a different image. There are 3 different kinds of glasses: Red-blue glasses [10], polarized glasses [11], and active shutter glasses [12].
There are major applications of stereo imaging techniques in the biomedical field [13], such as stereoscopic endoscopy [14], stereoscopic microscopy [15], and so on. A stereoscopic endoscope exams the interior surface of a hollow organ or a cavity and generates 3D vision for laparoscopic surgery. A stereoscopic microscope uses reflected illumination and two optical paths to provide depth information of complex biological specimens.
While the popular stereo techniques deal with surfaces that are opaque, we are interested in achieving stereo vision of semi-transparent structures. More specifically, we want to focus on the x-ray stereo-tomographic methodology. Thanks to the powerful nature of x-rays, most objects become semi-transparent under x-ray illumination, and there are research opportunities along this direction. As an initial example for this study, radiographs of lungs can be used for stereo imaging. This example is good, because most part of lungs is full of air, which means that lung tissues and sizable nodules can be easily visualized through stereo imaging.
Here we propose an x-ray stereo-tomographic approach for nodule measurement. By “stereo-tomographic”, we mean positional and morphometric information will be quantitatively generated from two projection views. The key idea is to mix a virtual space with a physical space so that calibrated morphometric measures in the virtual space can be carried over into the physical space. In the virtual space, we have a world coordinate system in which known geometric objects such as an ellipsoid can be precisely positioned and adaptively changed. From these geometric objects we can synthesize a pair of projections. In the physical space, we have an individual patient’s image volume through which we have a stereo pair of x-ray radiographs or projections. Finally, the synthetic and real projections are registered, presented to an observer through stereo glasses, and mixed in his/her mind to form a unified 3D scene. If a virtual object such as an ellipsoid and a real feature such as a nodule are well overlapped, the position and size of the nodule is basically the same as that of the ellipsoid; otherwise, the position and size of the ellipsoid can be iteratively changed until a good match is obtained. The main advantage of this x-ray stereo-tomographic method is that it can offer tomographic results (position and size quantitatively) without a CT scan.
The rest of this paper is organized as follows. In the next section, we describe the principle of stereo imaging and our stereo-tomographic method as applied to nodule measurement. In the third section, we report the numerical and experiment results. In the last section, we discuss relevant issues and conclude the paper.
Material and methods
Principle of stereo imaging
The basic principle of stereo imaging is to utilize the viewing angle difference of human eyes and merge a pair of images in the brain. To induce the 3D vision, a 3D movie system casts two images on a projector or monitor synchronously or alternatively, and an observer wears 3D glasses for each eye to see a different image. Among the three kinds of 3D glasses, red-blue glasses are the cheapest but yield the lowest visual quality. On the other hand, polarized glasses and active shutter glasses are much better choices.
A polarized 3D system displays 2 views in polarized light so that each glass can take the light polarized in a unique direction. An active shutter 3D system uses the monitor to display 2 different images alternatively at a high frequency in synchrony with liquid crystal glasses which block the light when glass-specific voltage applied. The 2 glasses block the light alternatively, synchronized with the refreshing rate of the screen. The polarized stereo imaging system is the most popular for 3D movies, the polarized 3D projector system is much more expensive than the active shutter 3D system but the polarized glasses are much cheaper than active shutter glasses. For example, the 3D movie system IMAX (Image Maximum) is popular with a standard IMAX screen 22 m×16.1 m, an IMAX projector up to 1.8 tons, over 178 cm in height and 195 cm in length, costing more than 1 million dollars. While the active shutter 3D system is much cheaper than polarized 3D system, a pair of active shutter glasses costs more than 100 dollars but a pair of polarized glasses may be less than 1 dollar. Therefore, the polarized stereo system fits for theater, and the active shutter 3D system is suitable for laboratory use.
In this project, we use active shutter glasses 3D Vision 2 Wireless Glasses made by NVidia [16], the monitor ASUS VG248QE with a refreshing frequency 144 Hz or 120 Hz, and the graphic card EVGA GeForce GTX 980. Our computer uses the USB IR Emitter to communicate with the 3D Vision glasses to make sure each eye only sees an appropriate image. The stereo imaging system with active shutter glasses is shown in Fig. 1.
The software to open 3D images is shown in Fig. 1, with the two images covering the screen fully for the NVidia 3D Vision Photo Viewer. In practice, there is typically only one image displayed on the monitor for stereo vision through 3D glasses. The viewer could also open several images on the monitor to compare nodule measurements. After the system setup, we can select a viewing angle between two projections to make a stereo pair that should be consistent with the viewing angle of human eyes as shown in Fig. 2.
Let L denote the distance between eyes, θ the viewing angle between two projections, and SOD the source-to-object distance. Then, we can compute the viewing angle between two projections as follows:
After two projections are generated, we can put them in the stereo format JPS which is a stereoscopic JPEG image used for creating 3D effects from a pair of 2D images. Then, the stereo images can be displayed using the software NVidia 3D Vision Photo Viewer for an observer to extract 3D information.
In the same spirit of mixed reality, our nodule measurement method visually merge the stereo vision from a pair of real x-ray projections and the stereo vision from a virtual space in a perfect registration with the x-ray imaging coordinate system. In the virtual space, the position and size of a geometric object can be interactively changed. When the geometric object and the x-ray feature of interest are well overlapped as perceived by our vision system, the size and position of the geometric object are that of the x-ray feature such as a lung nodule.
To compute the size and position of a nodule, we first generate a virtual object described by the following equation for an ellipsoid:
The ellipsoid can be translated by changing the center coordinates, enlarged or compressed by adjusting the radii, and rotated according to the following equation
Where α, β, γ are the rotation angles with respect to the X, Y, Z axes respectively, as shown in Fig. 3.
Most of nodules in the lungs are somehow spherical, and can be imagined to be comparable to an ellipsoid ball. In the clinic practice, the fitting process can be done with either a parametric ellipsoid or another object such as a cylinder, which will be determined by a radiologist. Next, we need to specify the attenuation coefficient of the ellipsoid, synthesize stereo radiographs in the same imaging geometry as that for the two real x-ray projections, and merge the virtual and real projections into a pair of mixed-reality stereo images. In this combination, if the projections of the ellipsoid and a nodule are well overlapped respectively, then the position and size of the ellipsoid and that of the nodule must be the same. The criterion for the match between synthetic and real features is our stereo vision mechanism; that is, we rely on neurological computing instead of digital computing in this context. Our overall flowchart for nodule measurement is in Fig. 4.
Nodule measurement test
In this section, we evaluate the proposed algorithm with simulated and clinical lung CT screening data. Our software interface in Matlab is shown in Fig. 5. The nodule measurement consists of 4 steps which are nodule implantation, projection synthesis, stereo viewing, and nodule quantification.
In our study, we digitally and respectively implanted 15 nodules with different positions and sizes in the clinical chest CT volumes of 156 slices and 512 × 512 pixels per slice. The original range of HU values was normalized to [0, 1], with the air and bone values being 0 and 1 respectively. The attenuation coefficients of these nodules were set to 0.75 after normalization.
Then, we generated two projections for an appropriate stereo angle. The average pupillary distance is 6.4 cm according to Google, and the source-to-object distance is 50 cm. With Equation (1), the angle between two projection directions would be 7.3°. In this test, we empirically chose 2 pairs of projection angles (the projection angle defined as the angle between the central ray and the Y axis), which are (90°, 97.3°) and (55°, 62.3°). The object-to-detector distance was set to 50 cm. The field of view was 30 cm in diameter and 23.4 cm in height, centralized at the origin of the imaging coordinate systems. The detector width was 60 cm. The length of the detector array was 46.8 cm. With these parameters, projection data were generated with Poisson noise, empirically we assumed the average number of detected x-ray photons105.
Furthermore, we created a virtual space in a perfect registration with the real imaging space and specified a geometric object in the virtual space. We projected the geometric object into the original projections. We colored the projected object in red.
First, we determined the position of a nodule, assuming the minimum size parameters a, b, c (in this study, a, b, c were all set to 3 pixels). We stereo-viewed the projections at 90° and 97.3°, adjusted the display window to find a nodule, copied the window setting to the software interface, and estimated the position parameters Y, Z. Then, with the projections at 55° and 62.3°, we selected a sufficiently large range and a sufficient sample interval to estimate the parameterX (in this study, we mostly set the range to [100, 350] and the interval to 20 in the pixel unit). We clicked the button “Make 3D images” to synthesize a pair of projections for each candidate X, with the other parameters being the same. Through the 3D vision glasses, a volunteer viewer could estimate a small range for X. Then, we sampled the small range for X refined estimation through stereo viewing (in this stage, we mostly worked in the range of 20 pixels in length and the interval of 2 pixels). After the estimation of the parameter X, we refined the parameters Y, Z with a pair of projections at 90° and 97.3° using the above-mentioned method.
After determined the nodule position, the volume of the nodule was estimated in terms of a, b, c through stereo-viewing of projections at 55°, 62.3° (within the same range [3, 7] and interval 1 for each of the three axes). Guided by the stereo-vision, the parameters a, b, c were interactively changed until the best visual overlap was reached.
According to the above-described steps for nodule quantification, the average errors of nodule position and volume estimates were generated from data collected by 8 viewers who are our lab members, which are summarized in Figs. 6 and 7.
As seen in Figs. 6 and 7, most of the errors are less than 1 mm, with submillimeter error bars, which means that the nodule measurement was stable. The nodule measurement in Z and radius c were better than other parameters. The measurement in X and a were worse than the other parameters. Clearly, this must be due to the asymmetry of the stereo-viewing geometry.
Nodule measurement with clinical x-ray projections
With the IRB approval, we collected lung CT images of 10 patients in a low-dose CT project at Massachusetts General Hospital. Each CT volume is of 118 slices and 512×512 pixels per slice. A projection image at 75° is shown in Fig. 8, where a suspected nodule is in the red circle. Using our proposed mixed reality method, we visually matched the nodule to an ellipsoid centralized at (249, 302, 39.5) with its a, b, c equal to 6, 5, 2.5 pixels respectively. After merging the synthetic and real projections and coloring the ellipsoid in red, the projection images are in an excellent fit, as shown in Fig. 9. The clinical CT slices through the nodule in the red circle is presented in Fig. 10.
As another example, an x-ray projection at 75° is shown in Fig. 11, where a nodule was marked in the red circle. Using the same method, it was found that the nodule was centered at (273, 202, 38), with a, b, c equal to 4, 5, 3 pixels respectively. The mixed projections are in Fig. 12. The clinical CT slices through a nodule are in Fig. 13.
Discussions and conclusion
Our main contribution is a novel approach for lung nodule quantification with stereo imaging. The active shutter glasses were shown useful to superimpose an appropriate ellipsoid over a lung nodule in the projection domain for lung nodule measurement. Our preliminary data suggest the feasibility of quantitative tomographic measurements based on two or few projections.
There are some issues to be discussed. First, the projection angles for stereo imaging play a substantial role. In the numerical simulation, we used 55° and 62.3° for stereo imaging while in the clinical CT study we used 75° and 82.3° for stereo imaging. The reason for the different angular setting is that the nodule was blocked by ribs. Therefore, our stereo-imaging method would be more reliable if it is applied to projections over an angular range, such as a tomosynthetic scan.
While the stereo angle we used in this study is 7.3°, this angle could be optimized. The larger the stereo angle is, the greater the difference between paired projections, and the more accurate the classic photogrammetric computation will be. However, an overly large stereo angle would compromise our 3D visual perception. Further research is needed to optimize the viewing angle and other parameters.
Several research groups tested stereoscopic visualization methods and were not very successful; that is, being not popular in the clinical practice. Hence, it is appropriate to underline the major advantages or unique characteristics of the new method tested in this study over the previously investigated methods. Briefly speaking, the main idea of our method is to measure a nodule in the mixed reality framework so that a calibrated “ruler” is also presented to a human viewer, and the measurement could be further refined with several pairs of projections. With the concept of mixed reality, quantitative graphical modeling and qualitative psychological viewing are combined synergistically for better performance. Along this direction, we can research further to incorporate color and motion information, and hopefully will eventually bring this approach to the level practically advantageous in some applications.
An emerging technology for x-ray imaging is based on the so-called Talbot effect, enabled by x-ray gratings. This imaging mode promises to capture not only attenuation contrast but also phase contrast and scattering contrast. Most importantly, small angle scattering signals are very informative for discrimination of benign and malignant tumors. In the future, we believe that x-ray grating based radiography can be combined with our proposed stereo imaging method to improve sensitivity and specificity of lung nodule detection.
In conclusion, we have proposed a mixed reality approach for stereo tomographic imaging of lung nodules. The results have demonstrated the feasibility and utility of the method. This method can also be extended for non-destructive evaluation and other applications. Further work is underway.
Footnotes
Acknowledgments
This work was supported by NIH under Grants R01 EB016977 and U01 EB017140, and partly by the National Natural Science Foundation of China (No. 61401049), the Fundamental Research Funds for the Central Universities (No. CDJZR14125501), Chongqing strategic industry key generic technology innovation project (No. cstc2015zdcy-ztzxX0002) and Graduate Innovative Research Projects of Chongqing (No. CYB14024).