The Effect of an Interval Training on Skill Retention of High-Complex Low-Volume Minimal Invasive Pediatric Surgery Skills: A Pilot Study

Abstract

Background:

Current training programs for complex pediatric minimal invasive surgery (MIS) are usually bulk training, consisting of 1- or 2-day courses. The aim of this study was to examine the effects of bulk training versus interval training on the preservation of high-complex, low-volume MIS skills.

Materials and Methods:

Medical students, without prior surgical experience, were randomly assigned to either a bulk or interval training program for complex MIS (congenital diaphragmatic hernia [CDH] and esophageal atresia [EA] repair). Both groups trained for 5 hours; the bulk group twice within 3 days and the interval groups five times in 3 weeks. Skills retention was assessed at 2 weeks, 6 weeks, and 6 months posttraining, using a composite score (0%–100%) based on the objective parameters tracked by SurgTrac.

Results:

Seventeen students completed the training sessions (bulk n = 9, interval n = 8) and were assessed accordingly. Retention of the skills for EA repair was significantly better for the interval training group than for the bulk group at 6 weeks (P = .004). However, at 6 months, both groups scored significantly worse than after the training sessions for EA repair (bulk 60 versus 67, P = .176; interval 63 versus 74, P = .028) and CDH repair (bulk 32 versus 67, P = .018; interval 47 versus 62, P = .176).

Conclusion:

This pilot study suggests superior retention of complex pediatric MIS skills after interval training, during a longer period of time, than bulk training. However, after 6 months, both groups scored significantly worse than after their training, indicating the need for continuous training.

Introduction

Minimal invasive surgery (MIS) is increasingly used for the repair of rare congenital anomalies, such as esophageal atresia (EA) and congenital diaphragmatic hernia (CDH).^1–3 To master these complex pediatric surgical procedures, advanced skills are needed. Acquisition of these skills, in turn, requires sufficient training.^2,4 Traditionally, the master-apprenticeship model has been used to acquire surgical skills, this is a training method that allows trainees to practice on patients under supervision. It is based on the assumption that trainees gain competence over time simply by exposing them to experience.⁵ However, for low-volume procedures, sufficient exposition in the clinical setting can be challenging and simulation-based training may be used.^6–9

For minimally invasive pediatric surgery, there are 1- or 2-day training programs, which are efficient to get the first skills and explanation on the specific procedures and should lead to a steeper learning curve than no training. Nevertheless, complex low-volume procedures such as congenital diaphragmatic hernia (CDH) and EA repair are subject to a steep forgetting curve due to the lack of repeated practice of the acquired skills. The forgetting curve is the rate at which skills are (partially) lost when they are not repeated (Fig. 1). Repeated exercise by interval training will, therefore, most likely result in a less steep forgetting curve.^9,10 Continuous training enables the frequent practicing that is needed to successfully retain the necessary skills and improves perioperative psychomotor skills.^11–16

FIG. 1.

Learning curve versus forgetting curve (copyright Maja Joosten). Color images are available online.

An alternative may be a different training program. Learning curve studies have shown that training spread out over multiple training days or weeks can be beneficial for skill retainment as well.⁸ This type of training is known as interval training, which is proved to be more effective in acquiring and maintaining general surgical skills compared to bulk training. Moreover, the positive effect of interval training is especially pronounced for advanced skills that are difficult to learn.^7–10,17 We hypothesize that interval training will be beneficial to acquire and retain high-complex, low-volume MIS skills as well. The aim of this study is to compare interval training and bulk training to evaluate the optimal training method for retention of skills needed for high-complex low-volume minimal invasive pediatric surgery.

Materials and Methods

Participants

Medical students of the Radboud University, Nijmegen, the Netherlands, were recruited to participate in this study in January-July 2018. Medical residents may introduce bias by practicing surgical skills in the clinical setting during the study period. Therefore, medical students without exposure to surgical training in the clinical setting, but with medical and surgical knowledge, were recruited. All participants completed an informed consent form and agreed with anonymous processing of the data. According to local law, IRB approval was waived. The participants were randomly assigned to either the bulk or interval training group. Blinding and allocation concealment was not possible due to the nature of the study, although the assessment results were processed and analyzed by researchers who were not present during the training sessions.

MIS simulator

The EoSim simulator is an Augmented Reality MIS simulator by Eosurgical ltd., Edinburgh, Scotland, United Kingdom. It consists of a validated inanimate box trainer setup with a laptop and supplied SurgTrac software, to guide the trainee and track the instrument tips.^18–22 This take-home simulator is a training tool for MIS skills, which is suitable for pediatric training as well (Fig. 2). The SurgTrac software measures and analyzes the performance by tracking the colored instrument tips (red for right instrument and blue for left instrument). The instruments used during these training sessions were 3 mm needle holders, dissector, and scissors.

FIG. 2.

The EoSim simulator. Color images are available online.

Pediatric surgical procedures

Previously developed low-budget models for EA and CDH repair were used.²³ The EA model consisted of two water balloons, one with the small end and the other with the wide end cutoff. They were attached to a suturing pad, resembling both ends of interrupted esophagus (Fig. 3). The CDH model consisted of a round see-through plastic cup without the bottom. A nonlatex surgical glove with the fingers cutoff was placed over the box, simulating the diaphragm defect (Fig. 4). Basic laparoscopic suturing was practiced using a standard suturing pad provided with the EoSim simulator.

FIG. 3.

Esophageal atresia model. Color images are available online.

FIG. 4.

Congenital diaphragmatic hernia model. Color images are available online.

Protocol

The study was designed as an open-label randomized trial to evaluate the effects of different training methods on the surgical performance over time. The participants were randomly assigned to either of the groups based on the availability of the simulators. The content of the training programs was identical except for the time schedule (Fig. 5). The bulk training resembled the current workshops and courses provided for (pediatric) MIS training, they practiced the pediatric MIS procedures twice within 2–3 days during training session of two and a half hours. The interval training consisted of five 1-hour sessions within 3 weeks.

FIG. 5.

Training protocol for the bulk versus the interval group. For the assessments, one suture was performed in both the EA and the CDH model, the maximum duration was 30 minutes per model. CDH, congenital diaphragmatic hernia; EA, esophageal atresia. Color images are available online.

The participants first performed basic tasks to get used to the simulator and the instruments. General MIS suturing was explained and video instructions were provided as a basis for the complex MIS tasks. During the training sessions, the subjects were guided and instructed by one of the researchers. These researchers were the same for both groups and they were present during training at all times. Subsequently, EA and CDH repair were trained, with focus on suturing (Fig. 5). Thereafter, an after-training baseline test was conducted to determine the skills level reached by initial training, which consisted of performing one suture in the CDH model (braided 3-0 suture) and EA model (braided 4-0 suture). Skills retention was assessed at 2 weeks, 6 weeks, and 6 months posttraining by the same test. These tests were without guidance or instructions to avoid bias in the results. The performance was assessed by time needed to perform the procedure (seconds), sum of the off-screen time for the instruments of the right and left hand (percentage), workspace (centimeters), path length (meters), handedness (percentage), speed (m/second), acceleration (m/second²), smoothness (m/second³) aided by SurgTrac software, and distance of the sutures to the set targets in the EA model (millimeters) and suture gap distance of the EA and CDH models (millimeters), measured by researchers. Parameters were combined in a composite score.²⁴ The maximum time for the tests was 30 minutes per model.

Statistics

The data were analyzed with IBM SPSS Statistics 25 (Armonk, NY: IBM Corp). A P value <.05 was considered statistically significant. Differences between the two groups were analyzed using the independent samples Mann–Whitney U test. The different time points were analyzed with a repeated measures analysis of variance (ANOVA) and a Wilcoxon signed-rank tests. Sphericity was assumed if Mauchly's Test of Sphericity >0.05. If sphericity was violated, Greenhouse-Geisser was used. To provide an overall outcome, composite scores were calculated after general linear normalization of the outcome parameters ranging from 0 to 100 with 100 being the best score. The assessment parameters of the time, working space, distance, off-screen time and smoothness were weighted equally in the calculation of the composite score. A power calculation was performed, using the composite score as guidance, in which a difference of 20% in the composite score was considered a clinically relevant difference. With a power of 0.80 and alpha of 0.05, this came to eight participants in each study group.

Results

Demographics

A total of 18 participants enrolled in the study. There was one loss to follow-up during the training session in the interval group, which was excluded from analysis, resulting in nine participants in the bulk group and eight in the interval group. The majority of the participants (67%) were female, the average age was 21 years, and most (83%) were second year medical students. There were no significant differences in demographic characteristics between the two groups.

Comparison of bulk and interval group

At the baseline test, the interval group scored higher for smoothness (0.02 versus 0.00, P = .007) for EA repair, there were no other significant differences between the performances of the two groups. After 6 weeks, however, the interval group scored significantly better for the distance from target (1.0 versus 6.5 mm, P < .001). No other significant differences were found for the separate time points, as can be seen in Tables 1 and 2.

Table 1.

Parameters Tracked with SurgTrac at the End of Training (Baseline), After 2 Weeks, 6 Weeks, and 6 Months, Stated as Median with 25th–75th Percentile Range

Variable	Test type	Group	Baseline	2 weeks	6 weeks	6 months
Time (s)	EA	Bulk	564 (459–641)	533 (342–865)	574 (314–731)	463 (289–746)
	EA	Interval	413 (242–664)	477 (395–537)	401 (389–542)	406 (363–814)
	CDH	Bulk	518 (380–824)	567 (372–872)	504 (449–800)	1674 (1241–1800)
	CDH	Interval	516 (355–936)	544 (372–872)	451 (301–527)	989 (541–1577)
Off screen (%)	EA	Bulk	4.7 (2.1–13)	5.6 (4.0–22)	5.3 (2.3–14)	6.1 (1.2–8.2)
	EA	Interval	12 (7.6–19)	5.8 (0.9–16)	11 (5.8–28)	6.1 (3.4–15)
	CDH	Bulk	7.0 (4.0–9.5)	2.6 (0.1–5.7)	3.0 (1.3–6.0)	6.3 (3.2–13)
	CDH	Interval	4.4 (1.3–11)	2.9 (0.8–7.5)	2.0 (1.4–7.8)	7.3 (6.6–15)
Workspace (cm)	EA	Bulk	4.6 (4.2–5.6)	6.1 (4.7–7.8)	4.9 (4.3–6.4)	4.2 (3.8–5.7)
	EA	Interval	6.1 (5.2–6.9)	6.4 (4.7–8.2)	4.7 (3.9–5.9)	5.5 (3.8–5.6)
	CDH	Bulk	4.5 (3.9–5.2)	4.1 (3.6–4.3)	3.8 (3.6–4.3)	4.2 (3.0–5.6)
	CDH	Interval	4.4 (3.9–6.8)	5.0 (4.5–5.1)	3.7 (3.4–3.9)	4.5 (4.2–4.6)
Path length (m)	EA	Bulk	16.8 (14–25)	15.3 (14–20)	15.8 (15–16)	9.0 (4.2–19)
	EA	Interval	12.3 (6.6–32)	15.7 (11–23)	12.3 (7.9–18)	12.4 (11–17)
	CDH	Bulk	40.0 (15–63)	13.3 (11–20)	16.2 (15–17)	27.2 (7.9–53)
	CDH	Interval	22.9 (11–33)	12.8 (11–23)	14.3 (8.8–26)	31.6 (26–41)
Right handedness (%)	EA	Bulk	61.6 (53–66)	60.9 (51–64)	59.4 (56–68)	45.4 (14–49)
	EA	Interval	57.8 (49–62)	58.1 (54–64)	56.4 (45–67)	40.9 (41–52)
	CDH	Bulk	60.1 (28–72)	61.8 (52–65)	56.2 (54–59)	45.4 (14–50)
	CDH	Interval	56.3 (55–65)	48.8 (45–57)	56.4 (47–59)	55.1 (43–61)
Speed (m/second)	EA	Bulk	3.0 (2.6–5.2)	8.2 (3.9–10)	5.9 (3.5–9.2)	2.3 (1.3–4.8)
	EA	Interval	3.7 (2.9–5.6)	6.2 (3.3–11)	4.7 (2.3–9.4)	3.1 (2.8–4.1)
	CDH	Bulk	6.1 (3.7–9.1)	6.1 (3.7–9.1)	4.4 (2.7–13)	2.8 (0.7–5.2)
	CDH	Interval	4.0 (2.7–5.6)	4.0 (2.7–5.6)	5.8 (1.9–10)	4.0 (3.5–4.6)
Acceleration (m/second²)	EA	Bulk	1.9 (1.4–2.9)	4.5 (2.3–5.2)	3.4 (1.8–5.3)	1.5 (0.8–2.7)
	EA	Interval	2.2 (1.9–2.5)	3.9 (2.0–5.7)	2.4 (1.4–4.6)	1.8 (1.6–2.5)
	CDH	Bulk	2.0 (1.6–2.8)	3.3 (1.4–5.1)	2.7 (1.7–6.4)	1.8 (1.2–2.9)
	CDH	Interval	1.9 (1.8–2.7)	2.7 (1.4–4.3)	3.1 (1.3–5.6)	2.2 (2.1–2.2)
Smoothness (m/second³)	EA	Bulk	0.0 (0.0–0.01)	0.03 (0.01–0.08)	0.03 (0.0–0.06)	0.0 (0.0–0.03)
	EA	Interval	0.02 (0.01–0.02)	0.03 (0.01–0.11)	0.05 (0.0–0.10)	0.01 (0.0–0.01)
	CDH	Bulk	1.0 (0.0–1.0)	2.0 (1.0–5.0)	1.5 (1.0–5.5)	0.0 (0.0–0.0)
	CDH	Interval	1.0 (0.0–1.0)	3.0 (1.0–4.0)	1.0 (0.0–11)	0.0 (0.0–0.0)

Interval group n = 9, bulk group n = 8.

CDH, congenital diaphragmatic hernia; EA, esophageal atresia.

Table 2.

Parameters Measured by Researchers and Composite Score for the Different Test Moments (Baseline, After 2 Weeks, After 6 Weeks, and After 6 Months)

Variable	Test type	Group	Baseline	2 weeks	6 weeks	6 months
Distance from target (mm)	EA	Bulk	5.0 (0.8–6.8)	4.5 (2.8–7.0)	6.5 (4.8–9.8)	3.0 (0.0–5.5)
		Interval	2.0 (1.0–4.0)	3.0 (0.0–4.0)	1.0 (0.0–4.0)	6.0 (4.0–6.0)
		P-value	.534	.167	<.001	.815
Suture gap distance (mm)	EA	Bulk	0.0 (0–3.8)	0.0 (0–0)	0.0 (0–5.3)	17 (3.0–21)
		Interval	0.0 (0–1.5)	0.0 (0–1.0)	0.0 (0–0)	0.0 (0.0–7.0)
		P-value	.805	.959	.606	.142
	CDH	Bulk	0.0 (0.0–2.3)	5.0 (1.0–16)	0.0 (0.0–1.5)	6.0 (1.0–10)
		Interval	0.0 (0.0–9.0)	0.0 (0.0–4.5)	0.0 (0.0–12)	0.0 (0.0–2.0)
		P-value	.181	.743	.888	.537
Knot successful (n; %)	EA	Bulk	8; 100	7; 88	8; 100	6; 75
		Interval	9; 100	9; 100	9; 100	8; 89
		P-value	1.000	.351	1.000	.485
	CDH	Bulk	8; 100	8; 100	8; 100	5; 63
		Interval	9; 100	9; 100	9; 100	6; 67
		P-value	1.000	1.000	1.000	.868
Composite score	EA	Bulk	67 (8.5)	69 (7.4)	65 (9.2)	60 (12)
		Interval	74 (5.8)	75 (5.6)	77 (5.8)	63 (19)
		P-value	.097	.072	.004	.682
	CDH	Bulk	67 (11)	57 (20)	65 (8.7)	32 (24)
		Interval	62 (12)	64 (12)	69 (16)	47 (24)
		P-value	.432	.378	.476	.219

Values are stated as median with 25th–75th percentile range, the composite score is stated as mean with standard deviation. Differences between the bulk group and the interval group are calculated with the independent samples Mann–Whitney U test on a two-tailed significance. Interval group n = 9, bulk group n = 8.

CDH, congenital diaphragmatic hernia; EA, esophageal atresia.

To provide an overall outcome, composite scores were calculated from all parameters together. The composite score of the normalized parameters per assessment moment is shown in Table 2 and Figures 6 and 7. The bulk training group seemed to have an overall steeper decrease in composite score compared to the interval group for both EA and CDH repair. The difference in skills between the groups was statistically significant for the EA repair at 6 weeks (bulk 65 versus interval 77, P = .004). However, no difference was found at 6 months (bulk 60 versus interval 63, P = .682). The CDH repair task was scored similarly by both groups without statistically significantly differences between the groups. Again, both groups lost their skills at the 6 months assessment session.

FIG. 6.

Graph of EA composite scores (mean of normalized scores: time needed to complete the procedure, distance between the instruments, path distance, off-screen time, smoothness, distance from target, and suture gap distance) at baseline, after 2 weeks, after 6 weeks, and after 6 months for the bulk group versus the interval group. EA, esophageal atresia. Color images are available online.

FIG. 7.

Graph of CDH composite score (mean of normalized scores: time needed to complete the procedure, distance between the instruments, path distance, off screen time, smoothness, and suture gap distance) at baseline, after 2 weeks, after 6 weeks, and after 6 months for the bulk group versus the interval group. CDH, congenital diaphragmatic hernia. Color images are available online.

Skill regression

For EA repair, the participants in the interval group retained their skills after the initial training session at the 2- and 6-week test. In the bulk group, there seemed to be a slight decrease in skills after 2 and 6 weeks, however, this was not statistically significant (2 weeks P = .499, 6 weeks P = .866). After 6 months, there was a decrease in skills for both groups, which was most evident in the interval group (bulk 60 versus 69, P = .063, and interval 63 versus 74, P = .028). Specific skills that were decreased after 6 months were accuracy of suturing in the interval group (2.0 mm at baseline versus 6.0 mm after 6 months, P = .046) and the suture gap distance in the bulk group (0 mm at baseline versus 17 mm after 6 months, P = .043).

For CDH repair, no significant regression in skills was found 2 weeks and 6 weeks posttraining in the bulk and the interval group. However, after 6 months, there was a significant decrease in scores in both training groups (bulk 32 versus 67, P = .018, and interval 47 versus 62, P = .028). Moreover, there was a significant increase in time needed to complete the procedure for the bulk group (518 seconds at baseline versus 1674 seconds at 6 months, P = .028) and in off-screen time for the interval group (2.0% at 6 weeks versus 7.3% at 6 months, P = .019) (Table 3, Figs. 6 and 7).

Table 3.

Differences in Composite Score Between Two Time Point (Baseline, After 2 Weeks, 6 Weeks, and 6 Months), Calculated with a Related Samples Wilcoxon

Loss in skills over time	EA		CDH
Loss in skills over time	Bulk	Interval	Bulk	Interval
δ composite score
Baseline versus 2 weeks	+3.1; 2.1 (9.3; 3.5)	+1.7; 1.2 (5.0; 1.9)	−4.9; −10 (14; 5.3)	+5.6; 2.0 (7.6; 2.9)
Baseline versus 6 weeks	−0.7; −0.4 (12; 4.7)	+4.4; 3.3 (11; 4.0)	−0.7; −2.4 (10; 3.9)	+11.4; 7.0 (14; 5.3)
Baseline versus 6 months	−4.2; −7.1 (9.0; 3.4)	−13; −11 (18; 6.7)	−31; −35 (26; 9.9)	−11; −16 (21; 7.9)
P-values
Baseline versus 2 weeks	.499	.310	.398	.128
Baseline versus 6 weeks	.866	.398	.735	.063
Baseline versus 6 months	.063	.028	.018	.028

Values are stated as difference of the median; absolute difference (standard deviation; standard error of the mean). Interval group n = 9, bulk group n = 8.

CDH, congenital diaphragmatic hernia; EA, esophageal atresia.

Discussion

Overall, the interval group seemed to have better skills retention during the first 6 weeks after the training, resulting in an even better performance over time. The graphs in Figures 6 and 7 show an evident and significant decrease in assessment outcome after 6 months, indicating an obvious forgetting curve. Previous research has demonstrated that interval-trained surgeons perform better than bulk-trained surgeons in general MIS surgery.^17,25 This study suggests that this accounts for pediatric MIS skills as well.

In the first 6 weeks after the training, there seemed to be little loss of the trained skills in both study groups. Nevertheless, the interval group showed indications of better performance on all assessed parameters throughout all test moments. Especially in the interval group, the performance seemed to be increasing at the 2- and 6-week time point for both tasks. Possibly, the test moments were in such short succession after the initial training that they acted as additional training exercises, enhancing skill retention and resulting in better performance in tests at later time points. This has previously been described by Dempster, who states that if a test shortly follows a training session, a successful test can count as a learning interval as well.¹⁹ Overall, loss of skills seemed to be less for CDH repair compared to EA repair. During the test sessions, participants performed the EA repair task first, followed by the CDH repair task. Possibly, performing the EA task acted as an additional exercise as well, resulting in skill retainment or even acquirement and a better performance on the CDH task. These two findings enforce our hypothesis that interval training is better for skill enhancement and retainment than bulk training. However, after 6 weeks, a decrease in skill retention was found in both groups, indicating that complex MIS skills should be practiced at least once every 2 weeks, to overcome the forgetting curve.

There is little consensus on the optimal time between independent training sessions (the intersession interval). A meta-analysis regarding the optimal intersession interval in training for surgical skills, comparing 11 controlled trials, found intersession intervals ranging from 5 minutes to 1 month. They recommend that the interval should be 10%–15% of the retention interval, which is the interval between the last training and the first test.⁹ A study by Dempster et al. suggests that a longer interval between sessions, ranging from 2 to 4 weeks, has a superior effect over a short interval.²⁶ While a study by Gallagher et al. showed that newly acquired surgical skills significantly degraded without use, after a mere 2 weeks. They recommend that the use of the trained skills in clinical setting should be no more than 2 weeks after the completion of the training, suggesting a smaller intersession interval for optimal acquisition of skills.¹⁷ In our study, the bulk group had training sessions spread out over 3 days. Although in some studies^17,27,28 this was actually regarded as interval training, it is suggested from our study that a longer interval was beneficial (five sessions in 3 weeks). Future studies focusing on the intersession interval are needed to adequately assess the optimal interval duration. Preferably, this would be done by designing a randomized trial, in which each group is assigned a different interval duration. Furthermore, research on how to overcome the forgetting curve could focus on continued training after an initial training session, to assess whether continued training contributes to better skill retention over time.

Limitations

This study has few limitations. Participants were not pretested on their innate abilities. Possibly, this could have given a better indication on the skill improvement over time. The inclusion of more participants would be beneficial to the results as well, however, this was tried and not possible in the current protocol of the study, due to logistic reasons. A power calculation was performed before the start of the study, indicating that eight participants should be sufficient to indicate a clinically relevant difference between the skills. This was seen in the loss of the skills in both groups, which is an important finding in this study. Another limitation is the use of medical students instead of surgical residents, because the latter are the target group. However, surgical residents are exposed to more confounders (e.g., courses on laparoscopic skills and exposure to surgical skills in the clinical setting) and, subsequently, more bias in the results. Therefore, medical students were recruited.

It was not possible to blind all researchers and participants, however, the analyses were performed independently and anonymously. The subsequent risk of bias was regarded low, due to the motivation of the participants to perform at their full ability during tests and the objectiveness of the outcome measures.

Conclusion

This study suggests superior skills retention of complex pediatric MIS skills after interval training compared to 1- or 2-day courses until 6 weeks after the training. However, after 6 months, performance of both groups significantly decreased, indicating the need for continuous practice to ensure optimal skill retention. Based on this and previous studies, an interval of 2 weeks should be a sufficient interval timing.

Informed Consent

Informed consent was obtained from all participants. According to local law and legislation Ethics Board approval was waived. All study procedures were in accordance with the Declaration of Helsinki as revised in 2013.

Footnotes

Disclosure Statement

No competing financial interests exist.

Funding Information

No funding was received for this article.

References

Borruto

, Impellizzeri

, Montalto

, et al. Thoracoscopy versus thoracotomy for esophageal atresia and tracheoesophageal fistula repair: Review of the literature and meta-analysis. Eur J Pediatr Surg, 2012; 22:415.

Teague

, Karpelowsky

. Surgical management of esophageal atresia. Paediatr Respir Rev, 2016; 19:10–15.

Wessel

, Fuchs

. Rolle U. The surgical correction of congenital deformities: The treatment of diaphragmatic hernia, esophageal atresia and small bowel atresia. Dtsch Ärztebl Int, 2015; 112:357.

Tsao

, Lally

. Minimally invasive repair of congenital diaphragmatic hernia. J Pediatr Surg, 2011; 46:1158–1164.

Evgeniou

, Loizou

. Simulation-based surgical education. AZN J Surg, 2013; 83:619–623.

McGaghie

. Mastery learning: It is time for medical education to join the 21st century. Acad Med, 2015; 90:1438–1441.

Smith

, Scarf

. Spacing repetitions over long timescales: A review and a reconsolidation explanation. Front Psychol, 2017; 8:962.

Spruit

, Band

, Hamming

. Increasing efficiency of surgical training: Effects of spacing practice on skill acquisition and retention in laparoscopy training. Surg Endosc, 2015; 29:2235–2243.

Cecilio-Fernandes

, Cnossen

, Jaarsma

, et al. Avoiding surgical skill decay: A systematic review on the spacing of training sessions. J Surg Educ, 2017; 75:471–480.

10.

Reynolds

, Glaser

. Effects of repetition and spaced review upon retention of a complex learning task. J Educ Psychol, 1964; 55:297.

11.

Abdalla

, Erin

M-A

, Chen

, et al. The effect of warm-up on surgical performance: A systematic review. Surg Endosc, 2015; 29:1259–1269.

12.

Bonett

, Wright

. Sample size requirements for estimating Pearson, Kendall and Spearman correlations. Psychometrika, 2000; 65:23–28.

13.

Calatayud

, Arora

, Aggarwal

, et al. Warm-up in a virtual reality environment improves performance in the operating room. Ann Surg, 2010; 251:1181–1185.

14.

da Cruz

JAS

, Reis

STD

, Cunha Frati

, et al. Does warm-up training in a virtual reality simulator improve surgical performance? A prospective randomized analysis. J Surg Educ, 2016; 73:974–978.

15.

Moldovanu

, Tarcoveanu

, Dimofte

, et al. Preoperative warm-up using a virtual reality simulator. J Soc Laparoendosc Surg, 2011; 15:533–538.

16.

Pike

, Pathak

, Mushtag

, et al. A systematic examination of preoperative surgery warm-up routines. Surg Endosc, 2017; 31:2202–2214.

17.

Gallagher

, Jordan-Black

, O'Sullivan

. Prospective, randomized assessment of the acquisition, maintenance, and loss of laparoscopic skills. Ann Surg, 2012; 256:387–393.

18.

Leijte

, Arts

, Witteman

, et al. Construct, content and face validity of the eoSim laparoscopic simulator on advanced suturing tasks. Surg Endosc, 2019; 33:3635–3643.

19.

Arts

, Leijte

, Witteman

, et al. Face, content, and construct validity of the take-home EoSim augmented reality laparoscopy simulator for basic laparoscopic tasks. J Laparoendosc Adv Surg Tech, 2019; 29:1419–1426.

20.

Retrosi

, Cundy

, Haddad

, et al. Motion analysis-based skills training and assessment in pediatric laparoscopy: Construct, concurrent, and content validity for the eoSim simulator. J Laparoendosc Adv Surg Tech, 2015; 25:944–950.

21.

Hennessey

IAM

, Hewett

. Construct, concurrent, and content validity of the eoSim laparoscopic simulator. J Laparoendosc Adv Surg Tech, 2013; 23:855–860.

22.

Partridge

, Hughes

, Brennan

, et al. Accessible laparoscopic instrument tracking (“InsTrac”): Construct validity in a take-home box simulator. J Laparoendosc Adv Surg Tech, 2014; 24:578–583.

23.

Bökkerink

GMJ

, Joosten

, Leijte

, et al. Validation of low cost models for minimal invasive surgery training of congenital diaphragmatic hernia and esophageal atresia. J Pediatr Surg, 2021; 56:465–470.

24.

Leijte

, Ijgosse

, De Blaauw

, et al. Assessing the concurrent validity of the EoSim laparoscopic simulator: Can this augmented reality simulator really assess the accuracy of a knot?. 2020 (in press).

25.

Breckwoldt

, Ludwig

, Plener

, et al. Differences in procedural knowledge after a “spaced” and a “massed” version of an intensive course in emergency medicine, investigating a very short spacing interval. BMC Med Educ, 2016; 16:249.

26.

Dempster

FN.

Spacing Effects and their implications for theory and practice Educ Psychol Rev. 1989; 1:309–329.

27.

Mackay

, Morgan

, Datta

, et al., Practice distribution in procedural skills training: A randomized controlled trial. Surg Endosc, 2002; 16:957–961.

28.

Willis

, Curry

, Gomez

. Practice schedules for surgical skills: The role of task characteristics and proactive interference on psychomotor skills acquisition. J Surg Educ, 2013; 70:789–795.