Accessible Laparoscopic Instrument Tracking (“InsTrac”): Construct Validity in a Take-Home Box Simulator

Introduction

Objective performance assessment using laparoscopic simulators can guide training, skills development, and preoperative warm-up.^1–4 “Gamifying” simulation training by introducing objective performance scores and target setting has been demonstrated to improve frequency of practice and resultant performance.^5,6 Instrument movement metrics are currently the preserve of expensive virtual reality simulators or of devices requiring complex and expensive additional motion-detection hardware.^7,8 The cost-effectiveness of equipment is an important but under-reported factor in the application of simulation training. ⁹ An analogy can be drawn to the evolution of the personal computer. Surgical simulators with performance metrics are currently at the “mainframe computer” stage of development—large, expensive devices found mainly in institutions and work places. Access is limited. A recent study suggested that fewer than 42% of surgeons have access to any form of laparoscopic simulator during working hours, with the level falling to 16% outside working hours. ¹⁰ We aim to evolve performance metric-equipped simulators to the “laptop” stage. The “InsTrac” software described here combines with a take-home simulator to make a package that is portable, affordable, and suitable for use at both work and at home. In achieving this we hope to enhance the uptake of simulation training, just as laptops facilitated increased computer use.

There have been many previous studies describing techniques for tracking instrument movement in laparoscopic box simulators.^8,11–15 The problem with these systems is that they require either multiple cameras or additional and costly motion-detection hardware. The “InsTrac” software described here is different in that it works through the camera feed and requires only a standard desktop or laptop computer running either the Windows™ (Microsoft, Redmond, WA) or the Mac™ (Apple, Cupertino, CA) operating system. It is “user-ready” and available for online download (eoSurgical.com/products/InsTrac). The eoSim minimally invasive surgery simulator used in this study (eoSurgical™ Ltd., Edinburgh, United Kingdom) (regular retail price from $600 U.S.) has recently been demonstrated to have both construct validity and concurrent validity with the established but more expensive Fundamentals of Laparoscopic Surgery (FLS™) simulator (Society of American Gastrointestinal and Endoscopic Surgeons, Los Angles, CA). ¹⁶

Generating objective performance metrics in a take-home box simulator may represent the optimal combination of performance feedback, real instrument handling, and accessibility.^9,17,18 In this study we report the development and construct validity of software that aims to achieve this.

Materials and Methods

Instrument tracking software

Computer-vision technology is used in many applications in high-dependency environments such as space travel, high-fidelity industrial manufacturing, and unmanned vehicles. ¹⁹ We have developed a color-thresholding motion-tracking program that uses the “hue,” “saturation,” and “value” color classification system to track the movement of colored laparoscopic instrument tips. The Hue-Saturation-Value system is superior to red, green, blue color coding as it separates “luma” (image intensity) from “chroma” (color information). This is important in maintaining tracking during the changing light conditions caused by movement of the instruments.

With a blue band wrapped around the left instrument tip and a red one around the right, each frame from the simulator camera's image feed is analyzed. In real time every frame is thresholded for each color, and the center of the resultant thresholded area is plotted as an x- and y-axis coordinate (Fig. 1). Combining these position data with elapsed time allows several metrics to be calculated. We chose to track in two dimensions as a recent experimental study demonstrated a strong correlation between two-dimensional and three-dimensional metrics. ¹² This suggests that two-dimensional may be as effective as three-dimensional tracking in generating metrics useful to provide incentives for practice and monitor skills progression. The movement metrics that we chose to calculate are based on parameters and formulas previously shown to be able to distinguish novice from expert surgeons in studies that used virtual reality or other complex instrument tracking hardware^{7,8,11–13,20,21} (Fig. 2 and Table 1).

OPEN IN VIEWER

FIG. 1.

The process of taking a frame from the simulator camera output, thresholding for non-white, and then refining the thresholding to the specific instrument tip color.

OPEN IN VIEWER

FIG. 2.

The instrument movement metrics results screen.

OPEN IN VIEWER

Table 1.

Formulas Used to Calculate Movement Metrics

Metric	Unit	Formula/description
Time (t)	Seconds
Total distance (td)	Meters
Average speed (as)	mm/second	td/t
Average acceleration	mm/second2	as/t
Motion smoothness	mm/second3	√((t5/2)×td2×as6)
Working area	Millimeters	Average distance between instrument tips
Time off screen	%	% time instrument tip not in the operative field
Handedness	Meters	Average difference in distance

Results

Construct validation

Significant differences were observed between the novice FY group and experienced StR group in the following variables: time to complete task, StR versus novice, 2 minutes 33 seconds versus 9 minutes 53 seconds (P=.01); total distance traveled by instruments, 3.29 m versus 11.38 m, respectively (P=.01); average instrument motion smoothness, 0.15 mm/second ³ versus 0.06 mm/second ³ (P<.01); and handedness (mean difference between dominant and nondominant hand), 0.55 m versus 2.43 m (P=.03) (Fig. 3). The dominant hand was demonstrated to move further than the nondominant hand.

OPEN IN VIEWER

FIG. 3.

Significant differences in instrument movement metrics between novices (Foundation Year [FY]) and active laparoscopic surgeons (surgical trainees [StR]). Total distance represents the combined left- and right-hand instruments' path length. Motion smoothness represents a measure of the rate of change of acceleration or “jerkiness” of the instrument movement. Handedness represents the mean difference in instrument path length between dominant and nondominant hands. From the data points, mean and standard deviation values (vertical lines) were calculated. Statistical analysis was performed using the unpaired t test with Welch's correction.

There was no significant difference seen in the average distance between instrument tips, average speed of instrument tips, mean acceleration, or time out of operative field (Fig. 4).

OPEN IN VIEWER

FIG. 4.

Instrument movement metrics in novices (Foundation Year [FY]) and active laparoscopic surgeons (surgical trainees [StR]), which were not significantly different. Working area represents the average distance between the instrument tips while the task is completed. A larger average distance indicates the instruments are occupying a larger operating area within the visual field. From the data points, mean and standard deviation values (vertical lines) were calculated. Statistical analysis was performed using the unpaired t test with Welch's correction. L, left; R, right.

Discussion

This study demonstrates construct validity of an accessible performance measurement tool in a portable laparoscopic box simulator. It is an advance in the field because it is user-ready and requires only a standard desktop or laptop computer. By using the camera feed alone, it removes the need for additional cameras or expensive motion-sensing hardware. This study was completed in a classroom setting, but the compatibility with personal computers and the portability of the system make it suitable as a take-home device.

The accessibility of take-home simulation has the potential to facilitate skills acquisition free from working-time restrictions and the problem of limited access to institutional simulators.^10,18 It is known that skills development in a simulated environment helps optimize the utility of time spent in the real operating theater. ¹⁷ It is also known that simulation training is most effective when goal-directed and accompanied by performance feedback.^2,17 This software package successfully combines accessibility and objective performance feedback.

Construct validity is demonstrated by distinguishing between novices and surgical trainees with previous laparoscopic experience (using metrics of time, total distance moved by instruments, degree of ambidexterity, and motion smoothness). Distance between instrument tips, average speed of instrument tips, mean acceleration, and time out of operative field were not distinguishing metrics in this study.

The significant differences in time and total instrument distance moved to complete the task are consistent with other studies comparing instrument movement metrics between novice and experienced surgeons.^8,11,12 The greater variance of the data points in the novice group is also in keeping with other studies using both virtual reality and augmented reality systems.^12,21 The finding that novices demonstrated greater reliance on their dominant hand is consistent with previous work highlighting a greater degree of ambidexterity in trained surgeons. ²² Motion smoothness (“jerk”) is a measure of the rate of change of acceleration. There is conflicting evidence as to its usefulness as a measure of surgical performance.^23–25

The lack of a significant difference between the groups regarding distance between instrument tips, average speed of instrument tips, mean acceleration, and time out of operative field is also in keeping with the findings of some experimental studies assessing instrument tip movement, but at odds with others.^12,26,27 It is noted that most existing studies assessing such metrics are small, likely because of limited access to equipment capable of generating appropriate data. The accessibility of the eoSim simulator and “InsTrac” software package presents an opportunity to facilitate much larger, multicenter studies.

We have developed software that brings objective performance feedback to the portable laparoscopic box simulator. Construct validity has been demonstrated. Removing the need for additional motion-tracking hardware makes it affordable and accessible. It is user-ready and has potential to enhance the training benefit of portable simulators, both in the workplace and at home.