Abstract
Background and Purpose:
Existing imaging modalities of urologic pathology are limited by three-dimensional (3D) representation on a two-dimensional screen. We present 3D-holoscopic imaging as a novel method of representing Digital Imaging and Communications in Medicine data images taken from CT and MRI to produce 3D-holographic representations of anatomy without special eyewear in natural light. 3D-holoscopic technology produces images that are true optical models. This technology is based on physical principles with duplication of light fields. The 3D content is captured in real time with the content viewed by multiple viewers independently of their position, without 3D eyewear.
Methods:
We display 3D-holoscopic anatomy relevant to minimally invasive urologic surgery without the need for 3D eyewear.
Results:
The results have demonstrated that medical 3D-holoscopic content can be displayed on commercially available multiview auto-stereoscopic display.
Conclusion:
The next step is validation studies comparing 3D-Holoscopic imaging with conventional imaging.
Introduction
3D-holoscopic imaging
The principle of 3D-holoscopic imaging uses a naturally occurring process of the “fly's eye” to capture 3D images. 1,2 It is an existing technique first developed by Gabriel Lippmann in 1908. 3 –5 3D-holoscopic imaging (also referred to as integral imaging) is the closest form of holography captured in a single aperture camera setup using an array of microlenses producing images that are true optical models, in contrast to holography in which coherent light sources (e.g., lasers) are required. This technique is capable of creating and representing optical models in the form of planar intensity distribution using unique optical components. 3D-holoscopic imaging is becoming a practical display technology that is attracting much interest in the 3D arena, 1,6 –9 and is now accepted as a strong candidate for next generation 3D entertainment systems. 6
The fly's eye 3D-holoscopic image is recorded using a regularly spaced array of small lenslets closely packed together in contact with a recording device, as shown in Figure 1a. Each lenslet views the scene at a slightly different angle to its neighbor; therefore, a scene is captured from many viewpoints, and parallax information is recorded. Parallax is the difference or displacement in the apparent position of an object viewed along two different lines of sight, giving the viewer multiple views of the same object in different viewing positions.
Existing technology
Current 3D imaging technology is based on stereo vision, where left and right eye images are presented to the viewer through temporal or spatial multiplexing by wearing a pair of shutter or polarized glasses to give the effect of depth perception. To create a stereoscopic image, two images are projected superimposed onto the same screen through different polarizing filters. The viewer wears 3D-eyewear that also contains a pair of polarizing filters. Each filter allows only light that is similarly polarized to pass, ensuring each eye sees a different image. This produces the 3D effect by projecting the same scene into both eyes but depicted from slightly different perspectives.
In contrast, autostereoscopic imaging systems use a large number of paired video signals that are recorded and presented on displays that do not require glasses for 3D viewing. A number of such systems are commercially available. 9,10 Such systems tend to cause eye strain, fatigue, and headaches after prolonged viewing because users are required to focus on the screen plane (accommodation) but to converge to a point in space in a different plane (convergence), producing unnatural viewing. 11
There are a number of alternative means for capturing true 3D content; two of the most recognized are holography and holoscopic imaging. Because of the interfering of coherent light fields needed to record holograms, their use is still limited and mostly confined to research laboratories. 12,13 3D-holoscopic imaging consists of a lens array mated to a photographic film or digital sensor with each lens capturing perspective views of the scene. 4,6,14 –17 The light field does not need to be coherent, and “holoscopic” color images can be obtained with full parallax.
Role in Endourology
There has been very little work on the clinical application of this technology. 18,19 An imaging company called Visionsense has reported the use of a microscopic array of lenses in front of a single imaging sensor device on the end of the endoscope. 20 The elemental information is translated using advanced proprietary software into virtual images, with 3D content viewed using special eyewear. A group from the University of Tokyo developed a high-resolution surgical display based on integral photography for visualization of computer-generated data. 17 3D-holoscopic imaging is capable of creating and representing true volume spatial optical models of an object scene in the form of a planar intensity distribution using unique optical components.
Techniques for 3D CT reconstruction in motion and maximal intensity projection do create 3D reconstructions that overcome most issues with static 2D images. 3D CT reconstruction provides multiple projections with a preview of the surgical field and study of lesion characteristics, which can help achieve faster and safer surgery. 21 Current 3D reconstructions are an artistic impression with shadows and perspectives with the viewer interpreting this image seen as 3D. Stereoscopic vision can often lack smoothness and can exaggerate the perception of depth.
The potential advantages of 3D-holoscopic imaging over computer-generated 3D reconstructions are associated with the limitations of 3D image representation on a 2D screen. The viewer still has to use cognitive processes to interpret the 3D image and generate a mental 3D image. With 3D-holoscopic imaging, this process is more intuitive because the images are seen in natural light without having to create a mental 3D image or use 3D-eyewear.
We aim to enhance preoperative planning and intraoperative visualization in prostate and renal cancer with 3D-holoscopic images during laparoscopic/robotic surgery adding the use of “augmented reality” (AR) as previously described by Ukimura and Gill. 19,22,23
AR is a live, direct or indirect, view of a physical, real-world environment whose elements are superimposed by computer-generated video or graphics. This technology may help to facilitate the development of the next generation of endoscopes using augmented reality endoscopic systems. 24 Further application of this technology is in endourology and stone disease. Development of augmented reality 3D-holoscopic representation of patient-specific imaging has the potential to enhance image-guided percutaneous nephrolithotomy (PCNL) access and stone localization. 25
Current research
The medical data obtained through CT and MRI is exported in Digital Imaging and Communications in Medicine format. A segmentation process is then applied to manually highlight the regions of interest in each slice. Using commercially available software, a 3D representation of the medical image data from the resulting sliced images is generated. The 3D model is then passed through our developed software creating a virtual camera made of microlens array, with the data being rendered into full 3D-holoscopic medical grade images (Fig. 1b). We have developed proof of concept display models (Figs. 2 –4) using a commercially available high-definition autostereoscopic display. Figure 3 shows the holoscopic image in a number of viewing angles with the pointer representing the actual time location of the image. To view the images as a stereoscopic image, the viewer must be at least 1 meter away from the screen.
Illustrating object location in the space with a pointer with multiple views.
Parallax angles on autostereoscopic display.
The limitation of the current prototype is that the virtual 3D-holoscopic camera is made of eight orthographic cameras with the capability to generate eight orthographic images (Fig. 4). We are able to generate eight images because the high definition autostereoscopic display (1920×1080) used currently only accommodates eight horizontal views through parallel projections. The virtual 3D-holoscopic camera model has eight pixels under each lenticular (long strips of cylindrical lenses) giving a 15-degree range of parallax (Fig. 4). The quality of the images produced is limited by development and release of higher definition displays.
We aim to perform feasibility studies using the autostereoscopic 3D displays with improved viewing angles and a higher resolution (Quad Definition, 3840×2160 pixels). Software tools have been developed that allow 3D-holoscopic images to be converted into the appropriate format supported by commercial autostereoscopic displays for viewing with greater degrees of parallax.
Potential clinical applications in urology
We aim to prove that this technology has a benefit over 3D reconstruction or clinical use of 3D eyewear in a number of urologic applications: • Real-time 3D-holoscopic acquisition and display during robot-assisted laparoscopic surgery. • The use of 3D-holoscopic imaging to facilitate accurate mapping for use in template transperineal biopsy of the prostate gland. • Creation of 3D-holoscopic images for use in real-time renal or lymph node biopsy. • Creation of 3D-holoscopic images for use in minimally invasive therapies such as cryotherapy for the destruction of small renal tumors or high-intensity focused ultrasound for treatment of prostate cancer. • Creation of 3D-holoscopic image-guided PCNL access and stone localization.
Our aim is to demonstrate that 3D-holoscopic technology provides accurate representation of target organs and surrounding vasculature and location of pathology by conducting validation studies and clinical trials.
Conclusion
Our application of 3D-holoscopic technology in endourology and minimally invasive therapies may have some impact in the visualization spectrum from diagnosis to staging investigations; however, the extent of this impact is yet to be determined. We believe 3D-holoscopic technology will add a new dimension to augmented reality in tumor location during technically challenging minimally invasive urologic procedures and in diagnostic uncertainty in tumor localization or stone surgery. We aim to further develop this technology in the field of endourology and demonstrate the use of this concept with further research and trials.
Footnotes
Acknowledgments
Prof. Dasgupta acknowledges financial support from the NIHR Biomedical Research Centre and the MRC Centre for Transplantation.
Disclosure Statement
No competing financial interests exist.