Abstract
Background and Purpose:
The Fundamentals of Laparoscopic Surgery (FLS™) skills curriculum has validity evidence supporting use for assessing laparoscopic skills for general surgeons. As charged by the American Urological Association (AUA) Laparoscopy, Robotic, and New Surgical Technology Committee, we sought to develop and validate a urology-specific FLS, referred to as the Basic Laparoscopic Urologic Surgery (BLUS
Materials and Methods:
An animate renal artery model was designed for a clip-applying skills task. We assessed the acceptability and construct validity of using BLUS for basic laparoscopic skills assessment for urologists. A cohort of practicing urologists, fellows, residents, and medical students completed the tasks at the AUA Annual Meetings in 2010 and 2011.
Results:
All exercises were acceptable and demonstrated excellent face and content validity (>4.5/5 on a five-point Likert scale). Practicing clinical urologists (N=81) outperformed residents and medical students (N=35) in time to completion of circle cut (P<0.01) and in keeping scissor tips toward the center of the circle (P<0.01). Practicing urologists who reported >3 laparoscopic procedures per week were faster at the peg-transfer exercise (P<0.05) and the cutting exercise (P<0.01) than those reporting one to two procedures. More errors were committed for clip-applying among practicing urologists who perform one to two laparoscopic procedures (1.24) vs. those who perform >3 procedures (0.57) per week (P<0.01).
Conclusions
Introduction
FLS is also valuable specifically for training urologic surgeons.
2
In 2009, the American Urological Association (AUA) Laparoscopic, Robotic, and New Surgical Technology (LRNST) Committee reviewed the curriculum of FLS and determined that the didactic component needed to be revamped to address issues more relevant to urologic surgery. It was determined that the didactic component needed to be more urocentric and, on review of the psychomotor exercises, it was decided that Basic Laparoscopic Urologic Surgery (BLUS
The University of Minnesota Center for Research in Education and Simulation Technologies (CREST) group was tasked to work with the AUA LRNST Committee to design, assemble, and assess validity of a low-cost, disposable renal artery model associated with a skills test. The objective is to assess the psychomotor skills associated with clip-applying and division of a medium-sized renal artery. This article describes the development and validity assessment of such a model over a 2-year period.
Materials and Methods
Renal artery simulator model
A 5-mm diameter renal artery model was constructed using an organosilicate-based material recipe that was designed using human tissue property data obtained from the Human Tissue Database from the University of Minnesota CREST group. Recipes of organosilicate-based materials were created and underwent uniaxial tensile tests to obtain the viscoelastic properties of the materials. These properties, including the Young modulus, could then be compared with those of human renal artery. The resulting renal artery model is comparable to human renal artery and behaves appropriately during clipping and cutting. Aesthetic and assessment features, including color mapping and black assessment lines, were added to the model design.
The length of the model is 6 cm with two 5-mm wide black clip placement lines and a dotted center-line for division (Fig. 1a). Each renal artery model is filled with artificial blood to a mean arterial pressure of 80±2 mm Hg. The model is attached to a motorized unit that creates the effect of pulsation for the artery and cycles at 60 cycles per minute (Fig. 1b).
AUA BLUS study
This study was approved by the Institutional Review Board through the University of Minnesota, and participants in the study were primarily recruited vis-à-vis AUA e-mail and conference advertisements. Practicing clinical urologists, fellows, residents, and medical students were eligible for the study. After informed consent, AUA participants filled out a demographics form and were given an overview of the objectives of each of the FLS tasks by a study administrator. Participants were asked to complete the four FLS tasks to assess the applicability and value of the tasks for BLUS. The participants did not know which FLS tasks existed previously and which were experimental. All tasks were performed inside a minimally invasive training box.
Task 1 (transfer exercise). As per FLS, a series of six plastic rings are picked up in turn by a grasping forceps from a Peg-Board™ on the participant's left, transferred in space to a grasper in the right hand and then placed around a post on the corresponding right-sided Peg-Board. After all rings are transferred from the left to the right, the process is reversed, necessitating transfer from the right to the left hand. This task is an existing FLS exercise and was designed to develop and assess the participant's depth perception and visual-spatial perception in a monocular viewing system as well as the coordinated use of both the dominant and nondominant hands. This exercise was timed, and a penalty score was assessed whenever the ring was not transferred midair. If a ring was dropped outside the participant's view, the exercise was terminated. The overall objectives of this exercise are to assess eye-hand coordination, ambidexterity and depth perception.
Task 2 (cutting exercise). As per FLS, in this exercise, a 4×4-inch gauze was suspended by alligator clips. The participant was required to cut a precise circular pattern from the gauze along a premarked template. The template consisted of an inner and outer circle spaced approximately 5 mm apart. This task requires precision as well as the use of the nondominant hand to appropriately position and provide traction to the gauze so that the dominant hand holding laparoscopic scissors may cut the gauze accurately. Deviation of the cut from between the two circles on the template resulted in a penalty. This exercise teaches the concept of traction, use of the nondominant hand to provide a convenient working angle, and working through the constraints of fixed trocar positions. The overall objectives of this exercise are to assess the ability to cut precisely and use both hands in a complementary manner to provide traction and countertraction.
Task 3 (suturing exercise). As per FLS, in this exercise, the ability to accurately place an intracorporeal stitch was assessed. The participant was required to pick up a 2-0 silk suture on a V-20 needle and position it properly using the needle holders. Next, a stitch was placed through target points on either side of a slit Penrose drain. Accuracy of stitch placement and deviation from marks were assessed. In 2010, the knot-tying ability of the participants was not assessed, and the overall objective of the exercise was to assess the ability to place a stitch. In 2011, the knot-tying task was added and required the participant to throw three knots and cut the suture to complete the exercise with the overall objective of assessing the ability to tie three knots accurately without strangulating or avulsing the Penrose drain. Penalties were given for deviation from the marks and strangulation of the tissue.
Task 4 (clip-applying exercise). This new exercise was designed to assess the ability of the participants to efficiently, safely, and accurately place metal clips on an animate medium-sized renal arterial model and divide it with laparoscopic scissors. Participants were asked to place two clips within the shaded area on each side of the dotted center line with a self-reloading clip applier and then separate the artery along the dotted center line with laparoscopic scissors. The metrics measured were time to completion, complete coaptation of the artery by the placement of four clips (two within each shaded area and completely across the structure) (Fig. 2), accurate division of the artery along the dotted line, and presence of leakage from the two cut ends on completion. The overall objectives were to assess the ability to accurately and expeditiously place clips on the renal artery and properly perform a division.
The four tasks were graded by two persons who had achieved inter-rater reliability with a Crohnbach alpha >0.8 before the study by evaluating performance of persons at the University of Minnesota and reaching agreement as to the standards. Standards were additionally defined by the principal investigators (RS and EM) and approved by the AUA LRNST Committee. Standards were designed to be similar to existing FLS grading with the addition of grading criteria for the clip-applying task. On completion of all four exercises, participants filled out a postexercise survey to evaluate the acceptability, face, and content validity of the FLS tasks. Five-point Likert scales were used for face and content validity. Comparison between groups (chi-square test) and within groups with different experience (independent sample t test), were analyzed to assess for evidence of concurrent validity. Significance was considered with a P<0.05, and a potential trend was considered with a P<0.10.
Cost 3
The BLUS exercise nondisposables consist of a basic laparoscopic video box trainer, a pair of laparoscopic grasping forceps, a pair of laparoscopic needle drivers, a laparoscopic scissors, 10-mm clip applier with disposable cartridges (Aesculap & Microline) and a peg-transfer board. The peg-transfer board costs $85 from SAGES. For the purposes of this study, we manufactured our own peg-transfer board at a cost of $25. The official box trainer from SAGES costs approximately $2200. Other commercial boxes range in cost from $1800 to $3000 depending on the quality of the camera and materials and flexibility of the camera port placement. We used three-dimensional med boxes for this study at a cost of $2500.
For the cutting exercise, the jumbo white clip costs $15.00 per unit. and modified disposable gauze costs $0.65 per unit from SAGES. We purchased the clip for $1.00 each and manufactured it at a cost of $0.04 per unit at the University of Minnesota for the purpose of this study. For the needle-driving and knot-tying exercises, we had to obtain the 2.0 silk suture ($1.25 each). The suture block costs $30.00 per unit, and the modified Penrose drain costs $0.70 per unit from SAGES. We manufactured the suture block for $5.00 per unit and the Penrose model at a cost of $0.10 per unit at the University of Minnesota for the purpose of this study. The cost for pure materials and labor for the University of Minnesota CREST to manufacture the motorized unit is approximately $360, and it cost $3 per each handmade disposable artery unit.
Results
Demographics
A total of 116 subjects participated in the study. Subjects were ages 22 to 66 years with a mean age of 39.5 years (standard deviation [SD]=10.1). There were 81 practicing clinical urologists, 29 residents, and 6 medical students. Medical student data were excluded from the analysis of acceptability, face, and content validity because of their inherent lack of knowledge and practical experience with the needs of current laparoscopic curricula.
Acceptability
Participants were given pre- and post-task questions to assess the acceptability of laparoscopic/endoscopic simulation. Figure 3 shows the distribution of pretask opinions among participants (excluding medical students) regarding the use of laparoscopic/endoscopic simulation in different situations.
Pretask opinions on laparoscopic/endoscopic simulation training.
The acceptability of the renal artery model was established post-task by asking participants if the artery model behaved like an actual human renal artery. The model was rated an average score of 3.71 (SD=0.99) using the Likert scale from 1 (low) to 5 (high) by 110 practicing clinical urologists and residents.
Face validity
In evaluating face validity, the postexercise evaluation showed that all of the tasks were considered valuable to train urologists as rated by 110 practicing clinical urologists and residents (Fig. 4). The suturing model yielded a mean score of 4.75 using the Likert scale from 1 (low) to 5 (high). The clip-applying exercise had a mean of 4.81. The circle-cutting and transfer exercises were both found valuable with mean scores of 4.66 and 4.66, respectively.
Value of exercise for training residents; n=110 practicing clinical urologists and residents (medical students excluded).
Content validity
Participants were asked to evaluate how accurately each model assessed the objectives of each task as described in the task descriptions in the Materials and Methods section. Participants (excluding medical students) were asked about the perceived accuracy of the models using the Likert scale (Fig. 5). The transfer model received an average score of 4.31 (SD=0.71), and the cutting model received an average score of 4.38 (SD=0.64). The suturing model without knot tying had an average score of 4.38 (SD=0.71), and the suturing model with knot tying received an average score of 4.54 (SD=0.60). The clip-applying exercise was given an average rating of 4.25 (SD=0.76).
Rating of content validity (5-point Likert scale).
In addition, participants (excluding medical students) were questioned regarding the extent to which they believed the models had a place in the urology laparoscopic curriculum (Fig. 5). On a Likert scale, the transfer model received an average score of 4.37 (SD=0.63). The cutting model received an average score of 4.47 (SD=0.63). The suturing model without knot tying was rated a 4.50 (SD=0.59), and the suturing model with knot tying received an average score of 4.67 (SD=0.55). The clip-applying model had an average of 4.29 (SD=0.81).
Concurrent validity
Our data did not show a significant difference between those practicing clinical urologists who had completed an endoscopic/laparoscopic fellowship and those who had not. There were statistically significant differences within the practicing clinical urologists group, however, when comparing the number of laparoscopic cases performed per week. The practicing clinical urologists were divided into those who performed only one to two cases per week (N=41) and those who had three or more cases per week (N=28). Only 12 physicians reported completing no laparoscopic procedure per week and were not included in this comparison. As shown in Figure 6, for all exercises, the average time to completion decreased as the number of cases performed per week increased. Using an independent samples t test with unequal variance, we found that the average time to completion was significantly shorter for physicians who perform three or more laparoscopic procedures per week compared with physicians who complete only one to two laparoscopic procedures per week for all exercises except for the clip-applying exercise.
Differences in time to task within practicing clinical urologists group.
Specifically, those who complete only one to two laparoscopic procedures per week averaged 141 seconds for the transfer exercise, and those who complete three laparoscopic procedures per week or more averaged 109 seconds (P<0.01). Those who complete only one to two laparoscopic procedures per week averaged 183 seconds for the circle cut exercise, and those who complete three or more laparoscopic procedures per week averaged 136 seconds (P<0.05). Those who complete only one to two laparoscopic procedures per week averaged 187 seconds for the suturing exercise, and those who complete three or more laparoscopic procedures per week averaged 90 seconds (P<0.01). While the trend was similar for clip-applying, there was no significant difference: Those who complete one to two laparoscopic procedures per week averaged 56 seconds, and those who complete three or more laparoscopic procedures per week averaged 50 seconds (P=0.43, not significant).
We also analyzed potential differences between these two groups of physicians in the errors observed during the four exercises. Using a chi-square test, there was no significant difference between the two groups in the frequency of dropping the ring outside of the visual field for the transfer exercise. Similarly, for the cutting exercise, there was no significant difference between the two groups in the frequency of cutting outside the lines. On the other hand, using an independent samples t test, those who complete only one to two laparoscopic procedures per week averaged 3.34 cuts into the line for the circle cut exercise while those who complete three or more laparoscopic procedures per week averaged only 2.00 cuts into the line (P<0.05). For the suturing exercise, while the trend was similar for the total deviation of insertion of needle from the mark, there was no significant difference: Those who complete one to two laparoscopic procedures per week averaged a deviation of 1.06 mm, and those who complete three or more laparoscopic procedures per week averaged a deviation of 0.86 mm (P=0.58). In addition, there was no significant difference between the two groups in the frequency of pulling the drain off the block.
Combining all clip-applying errors with the exception of “cutting half way before completing the cut,” which was deemed to be a deviation from standard technique rather than a true “error,” those who complete only one to two laparoscopic procedures per week committed an average of 1.24 errors in the clip-applying exercise while those who complete three or more laparoscopic procedures per week committed only 0.57 errors on average (P<0.01). Furthermore, 68.3% of those who complete only one to two laparoscopic procedures per week committed one or more errors in the clip-applying exercise compared with only 35.7% of those who complete three or more laparoscopic procedures per week (P<0.01).
We observed significant differences between the two groups in three of the five possible errors. Using a chi-square test, we determined that 21.9% of those who complete only one to two laparoscopic procedures per week did not place two clips per side, while only 7.1% of those who complete three or more laparoscopic procedures per week did not (P=0.08). Second, 37.0% of those who complete onlyone to two laparoscopic procedures per week did not place the clips in the correct area, while only 17.8% of those who complete three or more laparoscopic procedures per week did not (P=0.09). Third, 37.2% of those who complete only one to two laparoscopic procedures per week crossed the clips, while only 17.9% of those who complete three or more laparoscopic procedures per week demonstrated crossed clips (P=0.09). For the last two possible errors in the clip applying exercise, we observed no difference between the two groups in the frequency of not cutting half way to assess leakage, nor in the frequency of leakage after cut.
A comparison among all participant groups is shown in Table 1.
Convergent validity
Similar to other cohorts studied with FLS, using both the practicing clinical urologists group and novices (residents and medical students), we were able to establish that the practicing clinical urologists outperformed the less experienced group in regard to error in exercise completion. Similar to FLS, practicing surgeons did not always have a clear advantage over residents. 4 Initial FLS studies demonstrated results similar to those that we present here, and follow-up longitudinal studies demonstrated that more experienced surgeons were able to meet criteria with fewer repetitions than less experienced laparoscopists.
Discussion
The construct we focused on validating was the use of BLUS for the assessment of basic laparoscopic skills for urologists. In this study, we were able to demonstrate strong evidence of acceptability and construct validity (face, content, concurrent and convergent validity) as summarized in Table 2.
FLS™=Fundamentals of Laparoscopic Surgery.
For decades, we have tested surgeons' cognitive abilities on standardized multiple choice question and oral examinations and relied on “surgical numbers” during a completed residency program to assess technical skill. Once in practice, urologists can perform any procedure that they are credentialed to perform by their local hospital. The FLS program is now considered the standard assessment for basic laparoscopic skill for general surgery programs, and board eligibility is determined partly by successful completion and performance on their cognitive and psychomotor tests. It only makes sense that urology should develop and implement a similar fundamental surgical skills training program.
On careful evaluation of FLS, the AUA Laparoscopic, Robotic, and New Surgical Technology Committee found value in certain aspects of FLS, but found the didactic component to be out of touch with concepts that are important to the practicing urologist. A subcommittee is in the process of developing and validating a new didactic component that is outside the focus of this article. It was also determined by the Committee that the FLS endoloop and extracorporeal knot-tying exercises, while potentially sparsely used within the field of urology, did not represent “core” skills that would be needed for successful completion of a BLUS examination. It was determined, however, that clip-applying was a critical skill that should be taught and assessed in a standardized fashion. This assumption was supported by the results of this study.
Our data would suggest that the cutting task and the clip-applying task may have value in discriminating skill on the upper end of the learning curve, because these were the only two to show that laparoscopic case frequency correlated with errors among practicing urologists. Several open-ended comments were made, however, about the lack of fidelity for the cutting model (gauze). Low-cost tissue analogues could be considered to replace gauze and increase the fidelity without compromising the goals and objectives of the skills task. Future studies focused on predictive and discriminate validity aspects of our construct should definitively address this preliminary finding.
The charge was to develop a model that used low-cost, disposable models. The renal artery model that was developed, while not commercially available at the time of this article, certainly would have a cost comparable to the surgical gauze, Penrose drains, and peg transfer blocks used in the FLS curriculum. The cost of $3 per unit, while low, is still higher than the models used for the exercises; however, the preexisting models are simple modifications of already existing products (Penrose drain, surgical gauze, etc.). At this point, the multilayered artery models are each made by hand. There are ongoing discussions with potential manufacturing partners for the clip-applying materials that should drive the $3 handmade per unit production cost down.
BLUS, like FLS, should be easily distributable across all of the residency programs. The fact that most urology residency program sites also have general surgery programs that require completion of FLS means there should be ready access to the materials for BLUS minus the clip-applying exercise materials. The purchase for the original FLS equipment and support for the testing for general surgery residency programs was originally facilitated by a spin-off company from SAGES and an educational grant from Covidien.
Discussions are ongoing between the AUA and American Association of Gynecologic Laparscopists (for gynecology) with regard to potential plans for jointly administering the program. In the meantime, the BLUS study group has developed a urologically relevant didactic component to accompany the aforementioned psychomotor tasks that need to be evaluated. The AUA LRNST Committee will be reviewing the validity evidence as it accumulates to facilitate the rapid dissemination of the overall program, with consideration for making it a requirement for all urology residents to complete before their qualifying examination, as is currently required for general surgery residents.
Conclusions
All exercises, including the clip-applying model, that were used for this study were shown to have value for training of urologic surgeons. The exercises allowed participants to demonstrate basic laparoscopic skills that are used routinely by this group, and the models have the ability to be used as testing devices in the future. The renal artery clip-applying model is low-cost and disposable and meets the needs of the training device articulated by the AUA Laparoscopic, Robotic, and New Surgical Technology Committee. A standard scoring system that distinguishes between competent and noncompetent surgeons for the clip-applying exercise needs to be established. Future longitudinal studies that examine predictive and discriminate aspects of our construct need to be performed to assess if training increases performance on the clip-applying task and if experts improve their score with less training than novices. Formal testing in combination with the newly developed didactic component of the curriculum, when it is available, should also be performed.
Footnotes
Acknowledgments
Partial funding for this project was provided through a grant from the American Urological Association Office of Education. We wish to thank 3-Dmed for providing training boxes; Covidien, Microline, and Aesculap for providing clip-appliers; the AUA Laparoscopic, Robotic, and New Surgical Technology Committee members: Steven Nakada, Louis Kavoussi, Inderbir Gill, Laurent Boisrond, Matt Dunn, Anil Kapoor, Jamie Landman, John Phillips, Ravi Munver, Erik Castle, Jeffrey Caddedu, Chandru Sundaram, Steve Shichman, Ernie Sosa, Raju Thomas, Howard Winfield, Benjamin Lee, as well as Richard Scott for their support.
Disclosure Statement
Robert Sweet, M.D., has research grant support from American Medical Systems, Olympus, Allergan, and Medtronic. He has financial interest in Red Llama Inc. None of these relationships are directly relevant to this study. Elspeth McDougall, M.D., has equipment support from Karl Storz, Intuitive Surgical, Simbionix, and Ethicon Endo-Surgical. For the remaining authors, no competing financial interests exist.