A Study on the Learning Curve of the Robotic Virtual Reality Simulator

Introduction

Many new medical technologies such as robotic systems have been developed and applied in the medical field. These technologies are sometimes more advanced than the skills of the surgeons, and surgeons are required to adjust in order to use them. However, physicians and surgeons are also pressured to minimize the occurrence of medical errors, while training on actual patients is criticized, even though it is done to keep pace with technological developments in the medical field. Recently, regulations from institutional review boards have been tightened, not only for new drugs and studies, but also for new procedures and operations.

As interest in robot-assisted surgery continues to grow across surgical disciplines, the need for a safe and effective means of training surgeons is critical. However, training during live surgery exposes the patient to the inherent risks of an inexperienced surgeon, and training using animals creates ethical issues. In this circumstance, performing robotic operations on patients without adequate training can cause serious ethical problems. While surgical skills can be learned to some degree while assisting an operator in open surgery, with the robotic operation, the surgeon performs the operation at a separate location called the “console.” Therefore, it is difficult to acquire surgical skills naturally, as in open surgery, and isolated learning tools are needed in a robotic field.

In this aspect, a virtual reality simulator can be a good training tool, and the validation results for the contents of a few simulators have been considerably encouraging.^1–4 In addition, previous results have indicated that surgical improvement through the use of a simulator can lead to an improvement in performance with the real da Vinci^® Surgical System (Intuitive Surgical, Inc.). ⁵

However, the answer would be still unclear if someone asks how many hours should the simulator be operated in order for an individual to become proficient. So, we investigated how many performances are required to gain proficiency with the robotic simulator based on a calculated learning curve and how long the efficacy of the simulator continues.

Materials and Methods

Results

Table 1 shows the “Tube 2” task results between the initial and final sessions. In the comparison of the initial and final sessions, every parameter such as mean time, collision, and critical errors was significantly improved (Table 1).

Figure 1 is a scatter plot of the operating time of all of trainees versus the subsequent task number. The trend of trainee time corresponding to the task number is similar to that of Figure 2.

OPEN IN VIEWER

FIG. 1.

A scatter plot of the operating time of all of trainees versus the subsequent task number.

OPEN IN VIEWER

FIG. 2.

S-curve trend model. Each mean time is shown as a circle, and the circles are connected by a line. The line connecting the circles is explained by this curve. MAD, mean absolute difference; MAPE, mean absolute percentage error; MSD, mean standard deviation.

Figure 2 shows that each mean time is shown as a circle, and they are connected by a line. The line connecting the circles is explained by the best-fit curve (squares). The formula of this fitting curve is Y_t=10⁴/63.2855 – (54.6375×0.940035 ^t ), and the asymmetric rate among the curve parameters is the explanation capability of this curve model (Fig. 2). The asymmetric rate was about 0.94, which means the explanation capability of the curve model is about 94%.

The Chow test was performed to find whether the training has been done adequately. We wanted to determine the point at which the “Tube 2” task is mastered, and this can be calculated based on the starting point of the plateau in the curve. The slope in each trial changed significantly until the 74th trial, indicating that this was the starting point of the plateau. Using the S-curve trend model, we determined that the mean time at the plateau was 138 seconds.

Figure 3 is the autocorrelation plot for leaning time. Autocorrelation coefficients refer to the correlation between members of a series of numbers arranged in time. It is sometimes called lagged correlation. If we have a positive autocorrelation, it could be considered a tendency to remain in the same state from one observation to the next. In Figure 3, we determined that the autocorrelation at lag 5 is significant under the null hypothesis that the series is random. The dotted lines are simple approximate confidence intervals for the autocorrelation. The significance of the autocorrelation function is considered to not have died out until lag 5.

OPEN IN VIEWER

FIG. 3.

Autocorrelation function for time shows that there is a correlation between trials and that the training is very well done.

From the result of CUSUM charting we determined that when the target mean of the learning time is set as an average of 150 seconds, the time is achieved at about the 74th trial within the confidence interval. Here, the decision interval was 4, and the average run length was 8.38, which means the task time can be exceeding average 150 seconds about one in every eight trials.

Discussion

There has been a recent rapid increase in the development of laparoscopic or robotic devices for use in surgery, which has forced doctors to adapt quickly and safely to the new techniques. However, surgical training on the da Vinci Surgical System during live surgery is risky for the patients and laboratory animals on which it is tested, as well as being costly and unethical. Using the da Vinci Surgical System to practice and train in the use of the robotic training arms is also expensive, and potential breakdown of the robot is worrisome. In these circumstances, the appropriate training simulator can be safe for the patient, and the cost will be very efficient and low cost.

Currently, most articles on robotic simulators have been associated with validation.^1–4 Sethi et al. ⁴ and Kenney et al. ¹ assessed the validity of the Mimic dV-Trainer. Sethi et al. ⁴ reported that the Mimic simulator demonstrated excellent face and content validity as well as reasonable workload parameters. Kenney et al. ¹ validated the face, content, and construct of the dV-Trainer as a virtual reality simulator. Seixas-Mikelus et al.^2,3 assessed the face and content validity of the Robotic Surgery Simulator (RoSS), another virtual reality simulator for use with the da Vinci Surgical System developed in collaboration between Roswell Park Cancer Institute and the University at Buffalo. These studies suggested that RoSS has appropriate content validity and provided suitable opportunity for robot-assisted surgical training. There have been a few reports that simulator training leads to improvement in movement or handling of the da Vinci Surgical System. Lerner et al. ⁵ reported that, after four training sessions with the simulator, significant performance improvement was achieved, and the group who underwent simulator training seemed to have significantly better results when using the real da Vinci Surgical System. They found that simulator training including both the “letter board” and “pick and place” exercises clearly translated into improved performance on the final da Vinci Surgical System session. They also suggested that the simulator provided familiarity and comfort with the console controls that translated into a dramatic improvement on the da Vinci Surgical System for those in the simulator training group. In the present study, we observed a remarkable difference in performance between the initial and final sessions with the simulator.

However, the studies to date often only performed 10 or fewer simulator trials. Balasundaram et al. ⁷ reported that 10 novices performed a series of five tasks 10 times, resulting in statistically significant improvement for each of the five tasks. Lerner et al. ⁵ also only reported results from four training sessions. As a result, the amount of training required to attain proficiency with the simulator is not clear. According to this study, most students are able to achieve proficiency for “Tube 2” after 74 trials, after which there was no longer any improvement in results. On a scale of time, this achievement requires approximately 4 hours. In other words, trainees should invest approximately 4 hours of simulator training to become proficient.

The simulator training results reflect that the da Vinci Surgical System is a well-designed system because it does not take a very long time to adapt to its use. Ahlering et al. ⁸ reported that even a laparoscopically naive open surgeon successfully transferred open surgical skills to a laparoscopic environment in 8–12 cases when using a robotic interface, and this outcome was comparable to that achieved with skilled laparoscopic surgeons. That indicates that it is very easy for surgeons to adapt to the da Vinci Surgical System, unlike pure laparoscopy. The simulator, which imitates the da Vinci robotic system, seems to be well designed, making it easy for students to use. Conversely, the role of a training tool for robotic surgery might be limited, unlike with pure laparoscopy. The time to adapt to a simple instrument adjustment could be several hours, as shown in our study. However, the training for an actual surgery cannot be resolved with a simple adaption to the instruments, but requires an understanding of anatomy by reviewing video clips or observing live operations. Then, an understanding of the step-by-step surgical process is needed. However, this does not mean that the simulator is not useful. Compared with pure laparoscopy, the time to adapt to robotic instruments may be very short, and the simulator may be very effective for preparing surgeons for their first robotic operation.

There were several limitations to this study. First, the learning curve was not used to determine how long it would take to master all of the various applications. The “Tube 2” task is more than just another exercise; the performance of one task may affect that of another, making it virtually impossible to obtain the required time for all of the tasks. So, in this study, it would be more appropriate to consider the learning curve as the curve required to master “Tube 2” instead of the curve needed to make an individual using the simulator proficient. Evidently, “Tube 2” is the complex procedure introduced in the literature among recently released programs, and the current status of the development in the simulator can be weighed by this.^4,6 Another drawback of this study is that we did not perform an additional study to determine whether the technical improvement of trainees who overcame the learning curve could be reproduced with the real da Vinci surgery. The ultimate goal of virtual reality simulator training is to improve performance with the actual robotic system, so a prospective study should be performed in the future to address the effectiveness of simulator training with respect to this goal. However, this study attempted to examine the time needed to make the individual student using the developed simulator proficient, rather than focusing on how useful the simulator is in performing Da Vinci. The fact that operating the simulator is more helpful in performing the Da Vinci Surgical System than not operating it is quite a reasonable result, which has been proved already in previous studies. However, the result that the user of the simulator could be made proficient although only a short period of time was invested was revealed for the first time through this study and is the main purpose of this study as well. We anticipate that this translates into improved performance with the da Vinci Surgical System, in agreement with the previous report.

Simulators are still in their early stages. ⁶ Although the usefulness of the simulator has been demonstrated through simple robotic tasks, it is still unclear how useful the simulator will be for beginners who are performing their first robotic surgery. The possibilities with simulators are endless, and the ultimate goal of the simulator may be a presurgical rehearsal. With the development of more applications and programs for the virtual reality simulator, presurgical rehearsal will become a reality, rather than a dream. However, more work needs to be performed, and the efficacy of the simulator will have to be maximized through the development of programs that are useful for novices as well as early-stage surgeons.

Conclusions

The robotic virtual reality simulator improves technical performance within 4 hours of training, after which the usefulness of the simulator appears to be limited. Even though the simulator improves the individual's technical performance, the development of additional applications reflecting a variety of actual operations is needed to improve and maximize the usefulness of the simulator.