Educational Role for an Advanced Suturing Task in the Pediatric Laparoscopic Surgery Simulator

Introduction

Laparoscopic models are increasingly recognized as important tools in surgical training. While validation of new tasks using box trainers is regularly published, most focus on training residents, not experts.^1–7 We hypothesize that a sufficiently difficult task could present a significant challenge to experts and provide an opportunity to further develop the latter's laparoscopic skills.

The primary objectives of this study were to establish the construct validity for a more challenging intracorporeal suturing task and to explore expanding a box trainer's use to help upskill experts. Traditional construct validity is established when performance of a task allows discrimination between participants of differing levels of expertise. This methodology is felt to be adequate in assessment for educational purposes, although it may need to be more stringent for high stake assessment such as certification. ⁸ While traditional metrics of time for task completion, precision, and ability to complete the task were explored in this study, the focus was on motion analysis, which allows real-time assessment. Analysis of motion has previously been validated in the pediatric laparoscopic surgery (PLS) simulator to assess performance of a simpler suturing task. ⁹

To assist in upskilling experts, efforts were made to define a more challenging advanced task. We wanted to gain insight into the very definition of expertise and thoughtfully address factors that may help experts further improve their technical skills. To date, expertise—for the purpose of establishing construct validity—is generally ascribed according to self-reported caseloads ⁵ or certification (medical student, resident, and staff), ¹⁰ which presents inherent problems: verification, comparability and equivalence, and relevance. Current laparoscopic training systems, such as the fundamentals of laparoscopic surgery (FLS) ¹¹ and PLS ¹² certification programs, are aimed at establishing basic laparoscopic skills. As laparoscopic training and skills advance, determination of a less biased measure of expertise may aid for purposes of self-directed learning and assessment of psychomotor expertise.

Materials and Methods

The PLS simulator, a pediatric-sized trainer measuring 18 × 14 × 11 cm with an HD webcam, was fitted with custom-built motion tracking hardware and software consisting of tracking sensors on the inner surface of trocar ports (Fig. 1). Motion was assessed in the four degrees of freedom available in traditional laparoscopic surgery: Pitch, which is moving the hand up or down around a fixed point; Yaw, moving the hand side-to-side around a fixed point; Roll, which is pronation and supination; and Surge, which is pushing the hand in and pulling the hand out. Pitch, Yaw, and Roll are rotational degrees of freedom; Surge is a translational degree of freedom.

OPEN IN VIEWER

FIG. 1.

Pediatric laparoscopic surgery box trainer.

Participants were recruited at the 2013 International Pediatric Endosurgery Group (IPEG) meeting and were stratified into novice (<10 laparoscopic cases performed per year), intermediate (10–50 laparoscopic cases per year), and expert (>50 laparoscopic cases per year) groups based on self-reported caseloads. Participants performed an advanced intracorporeal suturing task, involving suturing in a plane perpendicular to the traditional axis of sewing, that is, passing the suture from superior to inferior instead of right to left, with the penrose drain placed in a vertical station instead of a more ergonomic horizontal one (similar to suturing during a Kasai procedure or choledochal cyst excision; Fig. 2).

OPEN IN VIEWER

FIG. 2.

Traditional intracorporeal suturing task (left column pictures) and advanced intracorporeal suturing task (right column pictures).

Traditional PLS scores were calculated using time (in seconds) and precision (proximity of needle passage to target, in mm). Three motion analysis parameters (MAPs), velocity, acceleration, and 90% range of motion, were recorded (see Construct validity). Performance was analyzed for successful task completion (unsuccessful tasks involved avulsion of the Penrose drain, only throwing a single knot on the first pass, or not completing the task). These data were compared to historic results for a simpler, traditional intracorporeal suturing task also performed on the PLS simulator. ⁹

Construct validity

As a discrimination measure to determine construct validity, we defined three MAPs based on velocity, acceleration, and range of motion. ⁹ We used the number of events measured that fall outside two standard deviations of a participant's overall velocity and acceleration data, referred to as “the number of extreme velocity and acceleration motion events” (Fig. 3).

OPEN IN VIEWER

FIG. 3.

Model of extreme motion events based on acceleration or velocity output from a motion tracking sensor during an intracorporeal suturing task. The thin line represents the participant's hypothetical mean. The broken lines hypothetically delineate all data lying within two standard deviations. The arrows indicate each hypothetical “event” where the participant goes beyond two standard deviations of their mean: we refer to these events as “extreme motion events.”

Velocity and acceleration extreme motion events have already been validated for the standard intracorporeal suturing task in an adult, as well as pediatric laparoscopic box trainer.^9,13 They allow discrimination based on “jerkiness” of movement: every time an operator makes a rough uncontrolled movement, this is represented by a sudden increase in the amplitude measurement of the motion sensor. The number of these extreme velocity and acceleration events is higher in the less experienced groups and decreases with increasing level of expertise. For the purpose of this study, analysis of motion was carried out using the participants' dominant hand.

Range of motion (90%) represents the range of movement made by each participant in the four degrees of freedom available. We used the 90% interval as a measure of the space used by each participant while completing the intracorporeal suturing task. The rationale was that by removing the extreme 10% of the range of motion, we would have an overall understanding of each participant's use of the space within the box trainer and determine whether this overall use of space allowed discrimination between levels of expertise.

The MAPs described above were measured for all candidates and cross-referenced to their level of expertise.

Results

Sixty participants were recruited (8 novices, 13 intermediates, and 39 experts). Analysis of motion in the performance of an advanced task allowed discrimination between participants of varying levels of expertise in all four degrees of freedom using the velocity and acceleration (Table 1), but not using the range.

Failure rates for the advanced task were high: approximately a quarter of experts and intermediates were unable to successfully complete the task, while almost two-thirds of novices were unable to complete the task (Table 2). Given that participants who failed the task could not be assigned traditional PLS scores (decreasing the power of the PLS score analysis), no significant difference between the average PLS scores of different levels of expertise was noted with ANOVA or post hoc analysis. However, compared to the average PLS scores from the traditional PLS intracorporeal suturing task published in 2011, ¹² average PLS scores for the advanced task were lower across all levels of expertise, reaching statistical significance for the intermediate and expert groups (Table 3).

When comparing MAPs between the historic values for the traditional task to those for the more advanced suturing task, the number of extreme velocity and extreme acceleration events were significantly higher in the performance of the more advanced suturing task versus the standard task among intermediates and experts (P < .001 for all comparisons, using an unpaired two-tailed Student's t-test). This was true for velocity and acceleration in all degrees of freedom available in laparoscopic surgery (Pitch, Yaw, Roll, and Surge). This did not hold true for range of motion. For novices, there was a trend toward a greater number of extreme velocity and extreme acceleration events in the performance of the more advanced task compared to the traditional task in all degrees of freedom. These values reached statistical significance for extreme velocity events in the Pitch and Yaw, and for extreme acceleration events in the Yaw, Roll, and Surge (Appendix Table 1).

Discussion

Assessment of specific MAPs in the performance of an advanced intracorporeal suturing task allowed discrimination among novices, intermediates, and experts, thus establishing traditional construct validity for this model. The number of extreme velocity and extreme acceleration events, for all participants, in all degrees of freedom, were higher in the performance of this more advanced suturing task, indicating that experts, intermediates, and novices all struggled significantly more. The 90% range metric was not discriminatory.

The more challenging nature of this advanced task is also reflected by the fact that the average PLS scores were inferior for all three levels of expertise compared to the historic average PLS scores ¹² obtained for the standard intracorporeal suturing task (Table 3), with the difference between the scores attaining statistical significance when comparing the expert and intermediate groups. Intuitively, it makes sense that the differences for novices were less marked between performances of the traditional and the advanced task because both tasks most challenged this group of participants: novices have a limited skill set given their lack of experience. The fact that only 76.3% of experts, 76.9% of intermediates, and 37.5% of the novices successfully completed the task also speaks to the greater technical challenge.

In terms of educational value, the consensus among experts who were involved in the testing was that the model mimicked the technical challenges in the laparoscopic management of a choledochal cyst. This made the model more relevant, thus adding to its educational value.

When comparing MAPs for different degrees of freedom in the standard intracorporeal suturing task, we had noted that pronation and supination (the Roll degree of freedom) was the most discriminatory. For the advanced task, all MAPs in all degrees of freedom were found to discriminate significantly: there were more extreme velocity and extreme acceleration events in the performance of the advanced task. This likely reflects the overall more complex nature of the task in all degrees of freedom. Performance of the advanced task involves operating in a vertical field (passing the suture in a superior to inferior vertical line through the Penrose drain), with fixed instrument insertion points in an incongruous horizontal plane. As mentioned above, this mimics the challenges in the laparoscopic approach to the reconstruction after excision of a choledochal cyst, or a Kasai portoenterostomy (Fig. 2).

As only a third of novices were able to successfully complete this advanced task and that among intermediates and experts the number of extreme motion events was significantly higher for this advanced task than for previously published extreme motion events in the standard task, this advanced task may be seen as being most useful for a cohort of participants with a preestablished critical level of technical skills (i.e., experts or intermediates). The complexity of the task may offer no advantage to novices, who may benefit more from focusing on the traditional task before moving on to a more complex one. We would therefore recommend this task for intermediates and experts who have already attained a certain laparoscopic proficiency, as a means to further hone their skills.

While one of the goals in this study was to explore the educational role of an advanced suturing task within the PLS simulator, the results highlight questions and reservations many have regarding the very definition of expertise, specifically as it pertains to psychomotor skills. Traditionally, this has been based on self-reported caseloads ⁵ or certification (medical student, resident, and staff) ¹⁰ —or a combination of both.^14,15 It is intuitive that someone performing <10 minimal access surgery (MAS) cases annually is likely a novice. However, the difference between intermediates and experts is more subtle; and the threshold for calling a surgeon an expert is less clear. Some “intermediates” who perform 10–50 MAS cases annually exhibit expert MAS technique, while informal observation, video analysis, motion analysis, and PLS scores indicate that some experts perform at an intermediate level. This overlap of certain participants classified as intermediates and experts has been noted in other studies ⁵ and highlights the ongoing challenge of defining expertise.^16,17 Other groups have proposed incorporating motion analysis as a performance-based tool to help designate psychomotor expertise. ¹⁵

There are several limitations to this study. The comparison to performance parameters for the simpler task is historic. The investigators certainly observed the degree to which all participants struggled more with this new, advanced suturing task, but the metrics do not compare performances in a formal “head to head” manner. There is also no proof that performance will improve with practice of this advanced task, although mastering simple tasks through practice ¹⁸ and moving on to more complex ones is the traditional paradigm for improving psychomotor skills. Finally, we acknowledge that the methodology used for validation was traditional and not keeping with that currently recommended by many for higher stake assessment. Given that our premise is the development of a tool for formative assessment and education rather than summative assessment and certification, most experts in the field agree that our traditional methodology is sufficient. ⁸

Analysis of motion for an advanced, more complex intracorporeal suturing task has demonstrated construct validity for this model, which mimics the challenging portion of a reconstruction after excision of a choledochal cyst or a Kasai procedure in terms of suturing orientation and instrument coordination. Further study will help establish whether the model should be incorporated into a structured training and assessment program targeting technically more advanced participants.