Algorithm-Based Motion Magnification for Video Processing in Urological Laparoscopy

Introduction

Nowadays, numerous medical procedures can be performed in a minimally invasive manner. For some of them (such as radical nephrectomy), laparoscopy has become the first-line treatment option. The utilization of advanced instruments, high-definition video processing, and robotics has further pushed the boundaries of minimally invasive medicine and substantially extended its indication range in surgery. ¹ Surgeons performing endoscopic procedures face specific challenges, making laparoscopy technically demanding and particularly difficult for novices. ^2,3 Obstacles such as a limited degree of freedom for intracorporeal manipulation and range of motion are important to note. ⁴ Factors like quality of video processing (analog or digital technology and separate video or high definition [HD]) thus play an important role in information perception and interpretation by the surgeon. ⁵ In addition, the video signal might be displayed as a two- or three-dimensional picture. In addition, three-dimensional imaging might worsen the image quality by halving the picture resolution.

Thus, structures located beneath the visualized surface (such as blood vessels) cannot be identified manually. ⁶ The improvement of these limiting conditions is being investigated intensively and addressed by many working groups worldwide. ^{7 –10}

Our research focuses on technological validation of the open-source software Eulerian Video Magnification (EVM) for its use during endoscopic procedures in urology (e.g., identification of vessel branches during partial and nephrectomy) and to investigate whether the pulsation of vascular structures in the abdominal cavity can be sufficiently enhanced to make them easily identifiable to the surgeon. ^11,12 Furthermore, we modified this technology, developing the novel CRS iimotion Motion Magnification (CRSMM) algorithm, which is optimized for applications in laparoscopy.

Materials and Methods

Results

Initial testing using EVM showed that the pulsating motions and color fluctuations of living tissue could indeed be visually enhanced and improved so that it becomes visible to the human eye. The first examples displayed the diastolic and systolic states detected through Motion Magnification (Fig. 1).

OPEN IN VIEWER

FIG. 1.

Mathematical background of motion magnification video processing. (A) Image example describing the temporal dimension of a video, (B) intensity changes for a pixel within three different frames of a video in grayscale setting, and (C) video transformation using CRS iimotion Motion Magnification algorithm.

Without coupling with a pulse oximeter, the algorithm was not able to give a satisfactory degree of visual enhancement to the vascular structures. This must have been attributed to the fact that, in videos of surgical interventions, motion occurs within the same frequency range as a pulse without being actually caused by a pulse. This could lead to false readings and the amplification of areas within the images that are not vascular structures.

Therefore, CRSMM algorithm adapted to laparoscopy videos was utilized. In this setting, coupling to a pulse oximeter was realized that resulted in significant precision improvement for detecting vascular structures. If the algorithm had reliable data regarding the pulse, the frequency band could have been narrowed down such that tissue movement due to other reasons had not been highlighted.

Figure 2 shows the strongly enhanced visibility of vascular structures when the enhancement algorithm was applied. However, it must be noted that this amplification comes with a significant loss of image quality. In future applications, the option to switch the amplification on or off depending on the situation will be integrated.

OPEN IN VIEWER

FIG. 2.

Practical examples of the motion magnification technology. (A, B) Pulsating blood flow-related motions adjusted to heart beat rate by coupling a pulse oximeter, (C) laparoscopic view at the renal hilus during radical nephrectomy without motion enhancement, (D) after applying Eulerian Video Magnification, (E) laparoscopic view of the internal side of the abdominal wall without motion magnification, and (F) after applying CRS iimotion Motion Magnification by coupling a pulse oximeter.

Discussion

Today, minimally invasive surgical techniques play an important role in operative medicine. Their ascendency started in 1987 with the first successfully performed laparoscopy by Mouret. ^13,14 In addition, the Hopkins rod-lens system, the dramatic miniaturization of video cameras has allowed a surgeon to visualize the interior of the human body in a way that was at least equivalent to open surgery. First and foremost, avoiding large wounds has led to less postoperative pain followed by a faster mobilization of the patient with fewer complications like atelectasis or thrombosis that can be caused by immobilization. Ultimately, shorter hospitalization times can result in faster convalescence. In addition, improved cosmetics because of minor surgical scars can be considered as a further positive effect, paradoxically often the most important advantage of the treatment in the patient's perception.

However, laparoscopic surgery can still be potentially hazardous under certain circumstances. The lack of haptic feedback can cause severe restrictions in intraoperative orientation and the identification of relevant anatomic structures. ¹⁵

Given these and other points of consideration, the surgeon may be forced to convert to open surgery. The rate of conversion in laparoscopic surgery depends on the particular procedure, and this factor is important for patient counseling. However, the most common indication for conversion in laparoscopy is inadvertent vascular injury and bleeding. A substantial vascular injury is usually difficult to manage in a laparoscopic manner, therefore forcing the surgeon to rapidly terminate the intervention. The absolute number of necessary conversions per year has declined significantly over the last three decades, reaching a low of <1%. Accidental vascular injuries, which account for 38.5% of these conversions, are the main cause. ¹⁶ Furthermore, a conversion is frequently associated with serious risk to the patient. Research on technologies for the detection of vascular structures is therefore needed and might possibly minimize that risk.

The new motion magnification technology in humans was first applied in 2009 by Park and Kim. They presented a method for transforming subtle facial expressions into exaggerated movements by using motion magnifying video processing algorithms. ¹⁷

The main strength of our study at this point of development lies in the exploration of new possibilities in atraumatic marker-free visualization of vascular structures during laparoscopic surgery. This work should be considered as an example of technology transfer into a new medical application and a feasibility test.

Motion magnification technology has therefore a real potential to be used in the future as a supportive image processing tool in endoscopy/laparoscopy. It can amplify subtle motions in video sequences to visualize structures that would otherwise not be visible. ¹⁷ The ability to amplify miniscule movements in real-time and thus visualize them could make small and/or deep unfavorably located vascular structures intraoperatively visible (e.g., under fatty tissue). Such noninvasive and label-free assistance to the surgeon might have the potential to reduce both the risk of intraoperative bleeding and conversion rate as well as significantly decrease operation times. Despite all the optimistic first results, motion magnification technology is not yet mature enough to be reliably utilized in the operation room (OR).

We have developed a new algorithm, based on the freely available EVM software that is adapted for intraoperative endoscopic image processing. However, the presented work has substantial limitations.

First, patient vascular structure pulsation can be highly heterogeneous, considering the target operative field. The organs might be covered partially by blood or blood clots resulting in a strong absorption of the oscillation signal. Furthermore, the constant patient-related pulsation of the vessels can be transferred to the recording video system through simultaneous vibration of the laparoscopy ports. This phenomenon, in combination with trembling caused by the hand of the surgeon holding the camera, generates additional artefacts that have to be extracted while processing the video signal. From this point of consideration, video material recorded for motion magnification editing should be captured in a tremor-free setting. This might be only possible using a special camera-holding system attached, for example, to the operating table. However, these requirements cannot be often guaranteed under sterile conditions in the OR.

Therefore, a patient's heartbeat rate has to be considered for mathematical processing of the video signal and hence included in the algorithm. The complexity of the calculation model thus exponentially increases, resulting in unstable image reproduction conditions, artefacts, and longer processing times. The respiratory rate is another factor that can interfere with the reconstruction process of endoscopic videos. Breathing might also cause organ (micro)movements, in particular, in the upper parts of the abdominal cavity. In urology, retroperitoneal structures are accessed frequently during different treatment modalities. For this reason, the respiratory rate has to be considered when developing video processing modalities suitable for endoscopy.

During the experimental evaluation, several further engineering issues were discovered, which need to be addressed to obtain reliable results, such as a low signal-to-noise ratios, color noise, and low or fluctuating brightness. Additionally, naturally moving tissue and jiggling motions of the instruments can generate false readings. These problems require further development, but we are confident that they can be resolved. There are several possibilities that will be investigated, such as coupling the system to a pulse oximeter to determine a narrower frequency band, utilizing specialized imaging electronics and optics, and providing better lighting technology.

This research focuses on basic questions regarding (micro)movement enhancement in video processing in urologic endoscopy and provides first evidence on this technology in laparoscopy. Both extensive technical development efforts and clinical trials are needed to make this idea suitable for the OR. The clinical relevance of motion magnification in surgery also should be evaluated in the future. Nonetheless, modern algorithm-based video processing methods that operate marker-free have already achieved noticeable results. Further research in this field is therefore needed and will most likely result in novel technical modalities. We conclude that it is technologically feasible to visualize blood vessels in real-time during endoscopic surgery. This kind of digitally enhanced visualization system is expected to further reduce the risk of accidental blood vessel injury, reduce patient recovery time, and improve overall surgical efficiency.

Conclusions

Motion magnification image processing technology has the potential to gain clinical importance as a video optimizing modality in endoscopic and laparoscopic surgery. Movements barely detectable to the human eye can now be visualized using this noninvasive marker-free method. We showed that it is technologically feasible to visualize blood vessels in real-time during endoscopic surgery. In laparoscopy, this might result in reduced bleeding and conversion rates, improving the overall outcome of surgery. Despite optimistic results, the technology requires considerable further technical development and clinical tests.