Motion Analysis–Based Skills Training and Assessment in Pediatric Laparoscopy: Construct,Concurrent,and Content Validity for the eoSim Simulator

Abstract

Purpose:

To validate the eoSim^® (eoSurgical Ltd., Edinburgh, Scotland, United Kingdom) simulator for pediatric laparoscopy.

Materials and Methods:

Participants were stratified according to their pediatric laparoscopy expertise. Three tasks were tested on the Pediatric Laparoscopic Surgery (PLS) and adapted eoSim simulators. Skill assessment was undertaken using motion analysis software for eoSim tasks and an existing validated scoring system for PLS tasks. Content validity was determined using Likert scale graded feedback responses. Construct validity was evaluated by investigating the respective abilities of the eoSim and PLS assessment tools to differentiate levels of experience. Concurrent validity was investigated by assessing the relationship between PLS and eoSim task completion times.

Results:

In total, 28 participants (8 experts, 7 intermediates, and 13 novices) were recruited. Content validity results were comparable or more favorable for the eoSim. Construct validity for motion analysis parameters was established for instrument path length (objects transfer, P = .025; suturing, P = .012), speed (suturing, P = .034), acceleration (suturing, P = .048), and smoothness (suturing, P < .001). For all tasks, there were significant differences between level of experience groups for eoSim task completion times and PLS scores (P = .038 to < .001). Significant relationships were found between eoSim and PLS task completion times for the precision cutting and suturing tasks (ρ = 0.298 and ρ = 0.435, respectively).

Conclusions:

This study demonstrates validity of the adapted eoSim simulator for training in pediatric laparoscopy. Future work should focus on implementing and evaluating the proficiency-based training curriculum that is proposed using construct validity-derived metrics.

Introduction

Surgical simulation has benefited from a surge in interest over the last decade as a result of the increasing need for a change in the traditional apprentice model of teaching surgery.¹

Simulation is now a well-established component of modern surgical training.² Major ongoing challenges in the field of surgical simulation are the standardization of performance assessment, accessibility of simulation resources, and measurable transfer of simulation-acquired skills to the operating room. A common method of structured skill assessment involves observation of task performance by an evaluator and derivation of a global score rating. Checklists are typically used as a standardization tool during this process; however, a high degree of subjectivity is inevitably involved. For a more objective and reliable method of skill assessment, some virtual reality simulators have embedded performance algorithms.^3,4 The ability to generate real-time computer data for objective assessment of skill performance using virtual reality simulators remains unavailable for pediatric laparoscopy.

A recent study by Azzie et al.⁵ validated a Pediatric Laparoscopic Surgery (PLS) simulator for conventional laparoscopic surgery (the pediatric version of Fundamentals of Laparoscopic Skills [FLS™]; Society of American Gastrointestinal and Endoscopic Surgeons, Los Angeles, CA). Currently, the PLS is the only validated system for pediatric laparoscopic training.

The eoSim^® (eoSurgical Ltd., Edinburgh, Scotland, United Kingdom) is a recently released laparoscopic box trainer that represents an affordable and potential portable training tool that when adapted can simulate the infant and neonatal environment.

The aims of this study are (1) to demonstrate construct, concurrent, and content validity for the eoSim laparoscopic simulator adapted for pediatric laparoscopic surgery and (2) to propose a proficiency-based curriculum to initiate pediatric surgeons to laparoscopic surgery.

Materials and Methods

Prior to taking part in the study, all participants read and signed an informed consent under a protocol approved by our institutional board (reference number C&W14/052).

Participants were enrolled and stratified into novice, intermediate, and expert groups according to their level of pediatric laparoscopy expertise as primary surgeon. Criteria for classification into experience-graded groups was based on participant self-reporting of personal involvement with <10, 10–50, and >50 procedures per calendar year.⁵

To take part to the study, participants were required to be qualified doctors and right-hand dominant. The latter criterion was intended to avoid handedness as a potential confounder in the motion tracking outcome data.

All participants performed three basic skill tasks on both the PLS and eoSim simulators. Three skill areas were examined: objects transfer, precision cutting, and intracorporeal suturing.

The selected PLS tasks were validated components of the PLS simulator model that is based on the McGill Inanimate System for the Training and Evaluation of Laparoscopic Skills (MISTELS).^2,5 The eoSim tasks were performed as standard according to the eoSim user guidelines⁶ (Fig. 1).

FIG. 1.

Corresponding tasks performed on the Pediatric Laparoscopic Surgery and eoSim simulators adapted for pediatric laparoscopy: (A and D) objects transfer, (B and E) intracorporeal suturing, and (C and F) precision cutting.

The eoSim box was adapted for pediatric laparoscopy, reducing the space inside the box (length, 18 cm; width, 11 cm; and height, 10 cm) to simulate a pediatric abdominal environment following the example of other authors⁷ (Fig. 2). The sizes of the blocks that tasks were mounted on were 5.5 cm × 3.5 cm (Fig. 3).

FIG. 2.

(A–D) The eoSim simulator adapted for pediatric laparoscopy.

FIG. 3.

Pediatric size blocks for the eoSim simulator.

The PLS simulator was set up in the standard configuration.⁵ Both simulators were connected to a Windows 7^® personal computer 21-inch flat screen full high-definition monitor using the LifeCam HD-6000 Webcam (Microsoft, Redmond, WA) for the PLS and the Logitech^® (Lausanne, Switzerland) high-definition Webcam C615 for the eoSim. Karl Storz^® (Tuttlingen, Germany) laparoscopic instruments 3 mm in diameter and 20 cm in length were used to perform the tasks on both simulators.

Participants were randomized to first perform tasks on either the PLS or eoSim before crossing over to perform tasks on the alternate simulator. This approach served to minimize performances bias due to training effect. The method of randomization was by sealed envelope technique. Participants were shown an instructional video for each task and permitted a single warm-up task repetition to familiarize themselves with the experimental set-up. The second run was used to measure performance and to compare results to test construct validity, as well as concurrent validity between the PLS and eoSim systems.

Motion analysis software (InsTrac^®; eoSurgical) was used to measure performance for the eoSim tasks.⁸ This software generates metrics in real time using a high-definition Webcam and labeling the tip of the instruments (blue for the left instrument and red for the right instrument). The InsTrac metrics are as follows:

• Time (in seconds)

• Total path length (in meters)

• Average speed (in mm/second) (formula = total path length/time)

• Average acceleration (in mm/second²) (formula = average speed/time)

• Motion smoothness (in mm/second³) (√[(time⁵/2) × total path length² × average speed⁶)

• Working area (in millimeters) (average distance between instrument tips)

• Time off screen (% time instrument tip not in the operative field)

• Handedness (in meters) (average difference in path length)

Standard PLS scoring metrics were used to assess performance for PLS tasks.⁵

Content validity was evaluated by providing participants with a posttest survey that asked them to respond to the question “Is this a good practice format” using a 5-point level-of-agreement Likert scale. All questions were answered in duplicate for both the eoSim and PLS simulators.

Statistical analysis

A sample size calculation was conducted based on the published data of a total composite PLS score of 75 ± 10 (mean ± standard deviation) for intermediate-level surgeons.⁵ It was estimated that to detect a 20% difference in total score PLS tasks completion between groups, with a two-sided 5% significance level and power of 80%, a minimum sample size of seven participants was required for each group.

To assess content validity (a measure of determining that all aspects of a test are included), analysis of Likert scale responses was undertaken. The Friedman test was used to compare responses for the PLS and eoSim simulators.

To assess construct validity (the degree to which a test measures what it claims and can differentiate between different levels of expertise), Kruskal–Wallis one-way analysis of variance was used to compare outcomes between levels of experience. If significant differences were determined, then the Mann–Whitney test was used to further compare outcomes between two groups, with the Bonferroni correction. In particular, outcomes between intermediate and expert groups were investigated.

To assess concurrent validity (when a test correlates well with a measure that has been previously validated), relationships between PLS and eoSim task completion times were evaluated using two-tailed Spearman's correlation tests. Additionally, relationships between experience level and task completion times were investigated to determine comparative strengths of association for the PLS and eoSim simulators.

Median and interquartile range values were calculated for all outcome measures. Statistical significance was regarded as P < .05. Statistical analysis was performed using SPSS version 22.0 software (IBM Corp., Armonk, NY).

Results

In total, 28 participants were enrolled (8 experts, 7 intermediates, and 13 novices). Participant demographic information is summarized in Table 1.

Table 1.

Demographic Data of Participants

	For total n = 28
Age (years)	32.5 ± 8.5
Sex (male:female)	19:9
Surgeon experience level
Expert	8
Intermediate	7
Novice	13

Content validity

Median aggregated 5-point level-of-agreement Likert scale scores for participant feedback responses were either comparable or more favorable for the eoSim compared with the PLS simulator. No significant differences in responses were observed between the eoSim and PLS simulators (Table 2).

Table 2.

Participant Feedback Question Responses for the Pediatric Laparoscopic Surgery and eoSim Simulators

Statement	PLS	eoSim	P
The simulator is useful for training in pediatric surgery.	4.5 (1)	5 (1)	.257
All novices should have training on the simulator prior to performing laparoscopic surgery on children.	5 (1)	5 (1)	.655
The simulator is useful in assessing the trainee's skill to perform pediatric laparoscopic surgery.	4 (1.25)	4 (1.25)	.527
The simulator is useful in assessing the progression of the trainee's pediatric laparoscopic skills.	4 (1)	4 (1)	.317
The simulator should be included in pediatric surgery training programs.	4 (1)	4.5 (1)	.763

Aggregated Likert scale score values are presented as median (interquartile range) values.

PLS, Pediatric Laparoscopic Surgery simulator.

Construct validity

Task outcome results are summarized in Tables 3 –5 for the objects transfer, precision cutting, and intracorporeal suturing tasks, respectively. Construct validity was demonstrated for numerous variables among each of the three tasks.

Table 3.

Objects Transfer Task Results by Surgeon Experience Level

	Novice	Intermediate	Expert	P
eoSim time (seconds)	113 (55)	105 (14)	78 (17)	.001^a
eoSim total path length (m)	2.04 (0.54)	1.36 (0.63)	1.17 (0.34)	.025^a
eoSim left path length (m)	1.2 (0.6)	0.7 (0.4)	0.6 (0.2)	.047^a
eoSim right path length (m)	1.0 (0.2)	1.0 (0.7)	0.6 (0.2)	.016^a
eoSim speed (mm/second)	2.91 (0.82)	2.38 (0.76)	2.32 (1.37)	.528
eoSim acceleration (mm/second²)	1.6 (0.51)	1.43 (0.41)	1.38 (0.74)	.686
eoSim smoothness (mm/second³)	0.07 (0.05)	0.09 (0.05)	0.09 (0.07)	.259
eoSim working area (cm²)	7.57 (1.25)	7.6 (1.15)	7.0 (0.67)	.614
eoSim left off-screen (%)	2 (1)	1 (1)	1 (1)	.589
eoSim right off-screen (%)	2 (3)	1 (0)	1 (0.25)	.035^a
PLS time	140 (54)	110 (31.5)	145.5 (26)	.070
PLS score	127.7 (39)	190 (31.5)	154.5 (33.2)	.015^a

Data are median (interquartile range) values.

Significant difference.

PLS, Pediatric Laparoscopic Surgery simulator.

Table 4.

Precision Cutting Task Results by Surgeon Experience Level

	Novice	Intermediate	Expert	P
eoSim time (seconds)	236 (78)	160 (45.5)	222.5 (60.5)	.009^a
eoSim total path length (m)	2.64 (0.99)	2.32 (0.91)	2.67 (1.64)	.353
eoSim left path length (m)	1.2 (0.4)	0.9 (0.6)	1.7 (1.1)	.209
eoSim right path length (m)	1.2 (0.8)	1.1 (0.5)	1.0 (0.3)	.689
eoSim speed (mm/second)	2.09 (0.86)	2.66 (1.39)	1.77 (0.32)	.664
eoSim acceleration (mm/second²)	1.34 (0.89)	1.94 (1.2)	1.5 (0.64)	.750
eoSim smoothness (mm/second³)	0.03 (0.01)	0.06 (0.035)	0.035 (0.023)	.125
eoSim working area (cm²)	5.9 (0.72)	6.61 (0.855)	6.36 (1.4)	.396
eoSim left off-screen (%)	1 (3)	2 (4)	1.5 (1.25)	.689
eoSim right off-screen (%)	1 (1)	1 (0)	1 (1.25)	.897
eoSim cutting score	4.0 (1.0)	6.0 (3.5)	6.0 (2.3)	.006^a
PLS time	188 (55)	135 (21)	140 (71.5)	.009^a
PLS score	262.9 (14.3)	291.4 (14.3)	257.1 (41.4)	.038^a

Data are median (interquartile range) values.

Significant difference.

PLS, Pediatric Laparoscopic Surgery simulator.

Table 5.

Intracorporeal Suturing Task Results by Surgeon Experience Level

	Novice	Intermediate	Expert	P
eoSim time (seconds)	339 (204)	156 (21.5)	129.5 (42.5)	<.001^a
eoSim total path length (m)	4.21 (3.47)	3.34 (1.16)	2.7 (1.24)	.012^a
eoSim left path length (m)	2.4 (2.4)	1.4 (0.5)	1.8 (0.7)	.003^a
eoSim right path length (m)	1.8 (0.7)	1.5 (0.7)	1.1 (0.7)	.065
eoSim speed (mm/second)	2.75 (0.98)	3.57 (0.53)	3.69 (1.67)	.034^a
eoSim acceleration (mm/second²)	1.64 (0.53)	2 (0.61)	2.19 (0.89)	.048^a
eoSim smoothness (mm/second³)	0.02 (0.02)	0.07 (0.04)	0.08 (0.06)	<.001^a
eoSim working area (cm²)	6.68 (0.6)	6.86 (0.79)	6.56 (0.33)	.377
eoSim left off-screen (%)	3 (3)	2 (3.5)	2.5 (4)	.878
eoSim right off-screen (%)	5 (2)	5 (4.5)	4 (1)	.137
PLS time	401 (270)	181 (44.5)	182 (72.8)	<.001^a
PLS score	184 (260)	416 (48.5)	417.5 (71.3)	<.001^a

Data are median (interquartile range) values.

Significant difference.

PLS, Pediatric Laparoscopic Surgery simulator.

For the objects transfer task, significant differences were observed between levels of experience for variables of eoSim task completion time (P = .001), total path length (P = .025), left path length (P = .047), right path length (P = .016), eoSim right-handed instrument off-screen time (P = .035), and PLS performance score (P = .015).

For the precision cutting task, significant differences were observed for eoSim task completion time (P = .009), eoSim cutting score (P = .006), PLS task completion time (P = .009), and PLS performance score (P = .038).

For the intracorporeal suturing task, significant differences were observed for eoSim task completion time (P < .001), total path length (P = .012), left path length (P = .003), eoSim instrument speed (P = .034), eoSim instrument acceleration (P = .048), eoSim instrument smoothness (P < .001), PLS task completion time (P < .001), and PLS performance score (P < .001).

Concurrent validity

Significant relationships were found between eoSim and PLS task completion times for the cutting and suturing tasks. Strength of correlation was moderate for the cutting task (ρ = 0.435, P = .21) and strong for the suture task (ρ = 0.861, P < .001) (Fig. 4). For the peg transfer task, there was a weak correlation that was not statistically significant (ρ = 0.298, P = .123).

FIG. 4.

Suture transfer task completion times for the Pediatric Laparoscopic Surgery (PLS) and eoSim simulators, categorized by level of surgeon experience.

Discussion

Our findings highlight the capability of modifying of an established adult-sized laparoscopic box trainer to be suitable for pediatric laparoscopic simulation. This adapted simulator is demonstrated to have construct, content, and concurrent validity and thus is appropriate to be endorsed for more widespread use by pediatric surgeons or surgeons-in-training. The hardware modifications are simple and achievable with basic household items. The eoSim simulator is available for international purchase at an affordable cost.⁹ It is fundamentally designed for portable use in either the home or the workplace.

Our results highlight the ability of instrument motion analysis to objectively distinguish between different levels of pediatric laparoscopic expertise. These findings empirically confirm the widely held opinion that experienced surgeons exhibit greater economy of movement, consistency, and automaticity of performanc.¹⁰ Task completion time was also a reliable discriminator of surgeon experience and was found to demonstrate construct validity for all tasks that were investigated. Our intermediate group outperformed our expert group in at least one outcome measure for each of the tasks evaluated. This suggests that experts perform at an intermediate level, and vice versa, and reinforces that there may be other factors aside from quantity of case experience that contribute to technical skil.^7,10

In terms of content and concurrent validity, training on both the eoSim and PLS simulators was identified as a good training tool for pediatric laparoscopy (Table 2). We suggest using all the metrics that showed construct validity to define the thresholds for a proficiency-based training program, following the same example of other authors for virtual reality simulator.^3,4,11 Our curriculum structure is summarized in Figure 5. The thresholds are set at the mean score plus twice the standard deviation of the experts' score.¹² Our curriculum is a proposal that needs further validation performing learning curve studies and predictive validity studies, to assess if the laparoscopic skills acquired by the learners using our training program can be transferred to the real-life environment. A hypothetical predictive validity study could be testing real-life surgical performance of surgeons who trained on the eoSim versus surgeons who did not use the training box.

FIG. 5.

Proposal for a pediatric laparoscopy proficiency-based curriculum for training and assessment purposes.

The concept of distributed training is based on validated educational theories that allow task performance and learning to be interspersed with rest periods.^13–15 This has been shown to optimize the learning process by maintaining skill acquisition for a longer period compared with mass training, as well as the avoidance of frustration and fatigue that could otherwise compromise novices' performance. We suggest performing the three tasks twice in a day in two different sessions, with each session more than 1 hour apart. On a subsequent day the novice is required to perform the three tasks, aiming to achieve the level of skills reported in Figure 5 on two consecutive sessions. We suggest practicing using the box for no more than 45 minutes at a time.^3,4,11,12 Adhering to such educationalist principles would allow a more structured approach to any training unit when designing a proficiency-based curriculum.

When the completion of predetermined training has been reached, the novice is required to undergo assessment in the presence of a supervisor. This is mainly due to the limitations of motion analysis software such as detection of tissue injury, etc. Contextualizing the results and assessing if the movements performed by the surgeon are purposeful are also essential aspects of the final stages of a validated training.¹⁶

The PLS and the FLS box trainer scoring systems require a trained proctor and do not allow for immediate feedback.^2,5 By incorporating motion analysis software and having the task metrics in real time, this approach will allow the trainee to self-direct his or her training, focusing the attention on the specific aspect of the task, as well as tracking his or her performance. The portability of this system also means that training can be done after hours and outside of clinical duty commitments.

The amount of time a novice is required to perform using a deliberate practice approach has also been examined. Ericsson¹⁷ has demonstrated how a novice may need at least 10 years of deliberate practice to become an expert in a particular field. The key fundamental aspects we have therefore considered for a structured approach to the design of a proficiency-based training program are (1) motivation to improve performance in a particular task or procedure, (2) immediate and detailed performance feedback, and (3) opportunity to repeat the task or procedure.¹⁸

The opportunity to repeatedly perform a surgical procedure is unpredictable and unreliable in clinical practic.¹⁴ Validated training tools for pediatric surgery are limited. The development and validation of training boxes, such as eoSim, permit novices to repeatedly train in a safe and reproducible environment. This study provides additional information, which may assist in a simulation-based training program or as a tool for selection criteria, as to date there is no validated standardized assessment tool for technical skills in pediatric surgery.¹⁹

Our findings also highlight the difficulties in defining the status of an expert. A surgical expert is usually defined only on the amount of procedures performed as first operator,^4,5 but the definition of surgical expert may be more complex. The classification of expert status may need to be reconsidered as based on level of surgical performance, rather than on the number of procedures or his or her professional status. A surgical expert needs to have appropriate technical and nontechnical skills and possibly natural talent to reach expert performance.¹⁰ Skill acquisition has been shown to be based on innate biological capacities, which limit the level of achievement that can be attained.¹⁰ Grantcharov and Funch-Jensen²⁰ went a step further to suggest that individuals who lacked the innate ability to acquire such technical skills should “choose an alternative professional field.”

In conclusion, the eoSim can be customized for simulation-based training and assessment in pediatric laparoscopic surgery. This modified version of the eoSim is construct, content, and concurrent valid. Our findings have permitted the development of a unique proficiency-based curriculum programme for novices to train in core skills for pediatric laparoscopic surgery. Further work is required to validate a more diverse range of basic skills tasks as well as procedure-specific tasks. Verification and measurement of simulation-acquired skill transfer into the operating room remains underinvestigated and should be a priority for future predictive validity studies.

Footnotes

Disclosure Statement

No competing financial interests exist.

References

Davies

, Khatib

, Bello

. Open surgical simulation—A review. J Surg Educ, 2013; 70:618–627.

Fried

, Feldman

, Vassiliou

, et al. Proving the value of simulation in laparoscopic surgery. Ann Surg, 2004; 240:518–525; discussion 525–528.

Aggarwal

, Grantcharov

, Eriksen

, et al. An evidence-based virtual reality training program for novice laparoscopic surgeons. Ann Surg, 2006; 244:310–314.

Aggarwal

, Crochet

, Dias

, Misra

, Ziprin

, Darzi

. Development of a virtual reality training curriculum for laparoscopic cholecystectomy. Br J Surg, 2009; 96:1086–1093.

Azzie

, Gerstle

, Nasr

, et al. Development and validation of a pediatric laparoscopic surgery simulator. J Pediatr Surg, 2011; 46:897–903.

Hennessey

, Hewett

. Construct, concurrent, and content validity of the eoSim laparoscopic simulator. J Laparoendosc Adv Surg Tech A, 2013; 23:855–860.

Nasr

, Carrillo

, Gerstle

, Azzie

. Motion analysis in the pediatric laparoscopic surgery (PLS) simulator: Validation and potential use in teaching and assessing surgical skills. J Pediatr Surg, 2014; 49:791–794.

Partridge

, Hughes

, Brennan

, Hennessey

. Accessible laparoscopic instrument tracking (“InsTrac”): Construct validity in a take-home box simulator. J Laparoendosc Adv Surg Tech A, 2014; 24:578–583.

eoSurgical. eoSim for individuals. Updated 2014. Available at www.eosurgical.com/collections/simulators-individual (accessed November 2, 2014).

10.

Sadideen

, Alvand

, Saadeddin

, Kneebone

. Surgical experts: Born or made?. Int J Surg, 2013; 11:773–778.

11.

Beyer-Berjot

, Patel

, Acharya

, et al. Surgical training: Design of a virtual care pathway approach. Surgery, 2014; 156:689–697.

12.

van Dongen

, Ahlberg

, Bonavina

, et al. European consensus on a competency-based virtual reality training program for basic endoscopic surgical psychomotor skills. Surg Endosc, 2011; 25:166–171.

13.

Mackay

, Morgan

, Datta

, Chang

, Darzi

. Practice distribution in procedural skills training: A randomized controlled trial. Surg Endosc, 2002; 16:957–961.

14.

Marcus

, Vakharia

, Kirkman

, Murphy

, Nandi

. Practice makes perfect? The role of simulation-based deliberate practice and script-based mental rehearsal in the acquisition and maintenance of operative neurosurgical skills. Neurosurgery, 2013; 72(Suppl 1):124–130.

15.

Moulton

, Dubrowski

, Macrae

, Graham

, Grober

, Reznick

. Teaching surgical skills: What kind of practice makes perfect?: A randomized, controlled trial. Ann Surg, 2006; 244:400–409.

16.

Aggarwal

, Moorthy

, Darzi

. Laparoscopic skills training and assessment. Br J Surg, 2004; 91:1549–1558.

17.

Ericsson

. Deliberate practice and acquisition of expert performance: A general overview. Acad Emerg Med, 2008; 15:988–994.

18.

Ericsson

. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med, 2004; 79(10 Suppl):S70–S81.

19.

Datta

, Chang

, Mackay

, Darzi

. The relationship between motion analysis and surgical technical assessments. Am J Surg, 2002; 184:70–73.

20.

Grantcharov

, Funch-Jensen

. Can everyone achieve proficiency with the laparoscopic technique? Learning curve patterns in technical skills acquisition. Am J Surg, 2009; 197:447–449.