End-User Input into the Design and Validation of a Synthetic Thoracoscopic Esophageal Atresia/Tracheo-Esophageal Fistula Simulator

Abstract

Introduction:

Thoracoscopic repair of esophageal atresia and tracheo-esophageal fistula (EA/TEF) is challenging. We addressed this by designing a fully synthetic simulator of the procedure and described the design process and how its content validity was assessed.

Methods:

An iterative design and assessment of content validity was undertaken in three stages. Data were collected from participants who trialed the model and completed a survey of their experience (adapted from Barsness et al.).

Results:

The model was trialed by participants of varying experience. Each design refinement improved the model's fidelity and validity. For the last iteration of the simulator, the observed averages (out of a maximum of 5) were: value as a training tool 4.8, relevance 4.6, physical attributes 4.5, realism of material 4.25, realism experience 4.17, and ability to perform tasks 3.77.

Conclusion:

An iterative design process based on end-user feedback has led to a synthetic simulator that has achieved a high level of content validity. This model has advantages over other EA/TEF simulators in that it is relatively inexpensive and does not use animal tissue, thus removing ethical and procurement issues. It was rated highly for its value and relevance to training.

Introduction

Acquiring the operative skill needed for thoracoscopic repair of esophageal atresia and tracheo-esophageal fistula (EA/TEF) is challenging.^1,2 Several research EA/TEF models have been designed to enhance the surgical skills required but these are not commercially available.^3–6 Similarly, there are several animal workshops where pediatric surgeons can practice their skills but they are costly and require travel^7–9 and may not be easily accessed by those who may benefit most from them.

In recognition of the difficulties that pediatric surgical trainees and pediatric surgeons have in acquiring these skills,¹⁰ we designed and developed a fully synthetic model for simulation. This model uses a replaceable silicone three-dimensional (3D)-molded insert of the EA/TEF within a 3D-printed scale reconstruction of the neonatal chest wall derived from neonatal computed tomography (CT) imaging data.

Any new simulator must be able to demonstrate its utility. Consequently, we describe the process of development and content validation of our simulator for thoracoscopic EA/TEF repair: The same process can be applied to all such simulators.

Methods

An iterative design approach was used to develop the simulator. The component parts were the ribcage, covering skin, and EA/TEF insert. Sequential development methodology and content validation is detailed. It involved a three-stage process over 18 months, with user feedback being obtained at each stage and informing subsequent refinement of the model.

The first iteration of the simulator was assessed for content validity in November 2017, the second iteration in April 2018, and the third iteration in March 2019. Data were collected from participants who performed parts of the thoracoscopic EA/TEF repair on the simulator and completed an evaluation survey (adapted from Barsness et al.³) of their experience. The survey consisted of 17 five-point rating scales. This evaluated the simulator across several domains, including: physical attributes, realism of materials and experience, task performance, value, and relevance. Participants were rated as novice if they were a nontraining registrar, intermediate if a training registrar, and expert if they were post fellowship or consultant surgeon/attending physician.

The use of an anonymous patient's CT data was permitted by the New Zealand Health Research Ethics Committee.

Statistical analysis

The “many-facet Rasch” model was used to minimize any bias introduced into the assessment of the model based on the experience of the participant. Validity evidence relevant to test content was evaluated by using the many-facet Rasch model, consisting of three facets (participants × level of expertise × items). Three indices were analyzed: observed averages, point-measure correlation, and item outfit mean-square (MS) statistics.

Observed averages indicate the participants' average ratings for each survey item, with higher values representing higher perceived value for that particular feature of the simulator. Comparisons were also carried out between participants with different levels of expertise, and P values were derived by using the Chi-square test. Point-measure correlation uses Pearson correlation statistics to measure how consistent the scores on an item are with the averaged scores of the remaining items. A negative correlation coefficient indicates that the item may be measuring a different construct compared with the other items in the survey. A positive value is ideal, implying that the item makes a useful contribution.

Item outfit MS statistics allowed us to measure how well the data fit the model. Values over 1.5 may indicate a problematic fit to the model. The presence of >5% of items with outfit MS greater than 2.0 may be considered as a threat to content validity.

Evidence of internal structure or consistency was estimated by using Cronbach's alpha. This index helps to evaluate whether these survey items are grouped appropriately and measure the same underlying construct, giving us more confidence to make inferences from the findings.

A one-way analysis of variance test was applied for comparison of observed averages across the three models.

Analysis in this study was performed by using Facets software (version 3.80) and SAS (version 9.4).

Results

Simulator design

Model 1

The initial model was designed and made up of a ribcage, skin, and EA/TEF insert (Fig. 1). This was assessed in November 2017, and the feedback obtained was used to modify the design and increase its fidelity.

FIG. 1.

Model 1: (a) skin, (b) ribcage and fixed EA/TEF insert. EA/TEF, esophageal atresia and tracheo-esophageal fistula.

Ribcage

A randomly chosen thoracic CT scan of a neonate was used to extract calcified ribcage data. These data were manipulated by using 3D Slicer version 4.3^11,12 and Blender version 2.78¹³ modeling software to replicate and then modify the neonatal ribcage. The “cartilaginous gaps” were filled in and simplified geometric shapes used to create a standardized, scale ribcage model measuring 110mm between the mid points of the first and tenth ribs. An Up Box 3D printer (Beijing TiertimeTechnology Co. Ltd) using acrylonitrile butadiene styrene plastic was used to print the ribcage mounted on a flat base.

Skin

An injectable 3D-printed mold was designed to create a silicone skin that fitted closely over the ribs (Fig. 1a).

EA/TEF insert

An injectable 3D-printed mold was designed to create a silicone EA/TEF model. Small plastic tubes were inserted into the open ends of the inserts to serve as anchorage points for the insert to the base plate (Fig. 1b).

Model 2

The model was further developed, fitted to a force sensor and motion tracking devices, and assessed for content validity in April 2018. The second iteration allowed the silicone to be folded under the edges of the base to securely fix the skin (Fig. 2a).

FIG. 2.

Model 2: (a) silicone skin, (b) EA/TEF insert fixed onto force sensor, (c) EA/TEF double-layered insert.

One of the technical challenges that novice surgeons encounter when first performing the esophageal anastomosis in EA/TEF repair is being able to reliably include the mucosa in the suture during the all-layer repair. This difficulty arises because the mucosal layer tends to retract from the cut end, sometimes being almost out of view, and is easily missed by the needle unless particular care is taken and a good technique is used.¹⁴ The second iteration (Fig. 2c) of the model replicated the layers encountered in vivo by creating a two-layered silicone structure for both the upper esophageal pouch and the esophagus distal to the fistula. This effectively simulates the muscle layers being relatively loosely connected to the mucosal layer, better replicating what is actually encountered during surgery. The model demands the same attention to this technical detail as is required during an actual end-to-end esophageal anastomosis. The anastomosis being performed is shown in Figure 3c.

FIG. 3.

Model 3: (a) skin covering ribcage, (b) ribcage, (c) double-layered esophageal anastomosis.

Model 3

Based on feedback, we made several changes. The EA/TEF insert molds were redesigned and printed by the Stereolithography 3D printing process to give more accurate and consistent dimensions and superior finish. Further, the insert was designed with integrated location/retention parts to allow a high degree of manipulation while remaining attached to the rib cage base. The silicones used for molding EA/TEF insert and skin were also changed to give elasticity and tear resistance appropriate to the task. The rib cage and skin were also remodeled for ease of use and aesthetics. The inner “mucosal” and outer “muscular” layer is demonstrated in Figure 3c.

Validation of simulator: content validity

Model 1: November 2017—tested at the New Zealand Society of Paediatric Surgeons

Five participants (1 novice, 1 intermediate and 3 experts) performed parts of the EA/TEF repair on our original model and completed a survey of their experience (Table 1). Participants reported seeing or performing a total of 86 open and 16 thoracoscopic EA/TEF repairs. Expert participants reported performing a combined estimate of 65 open or thoracoscopic EA/TEF repairs. The observed averages in descending order were: value as a training tool 4.8, relevance to training of 4.8, physical attributes 4.5, realism of materials 4.2, ability to perform tasks 3.3, and realism of experience 3.2. The Cronbach's alpha was estimated to be 0.89, suggesting that these survey items are in general grouped appropriately.

Table 1.

Results of Content Validity Assessment NZSPS 2017 for Model 1

Domain	Observed averages	Outfit mean square	Point-measure correlation
Physical attributes	4.45
Chest circumference	4.60	0.73	0.53
Chest depth	4.60	0.54	0.74
Intercostal space	4.60	0.69	0.58
Scale of insert within the model	4.00	0.14	0.00
Realism of materials	4.15
Overall impression	4.00	0.14	0.00
Skin	4.00	0.14	0.00
Ribs	4.62	0.54	0.74
Esophagus	4.00	0.14	0.00
Realism of experience	3.20
Anatomy during EA/TEF repair	4.00	0.14	0.00
Upper pouch anatomy	3.20	0.72	0.53
Movement of the insert during procedure	3.00	0.64	0.73
Ability to perform tasks	3.31
Closure of fistula	3.50	2.38	0.66
Dissection of upper pouch	4.00	1.02	0.85
Esophageal anastomosis	2.60	0.97	0.49
Value
As a training tool	4.80	0.87	0.22
As a testing tool	3.40	2.91	0.38
Relevance
Relevance to your training	4.80	0.64	0.43

EA/TEF, esophageal atresia and tracheo-esophageal fistula; NZSPS, New Zealand Society of Paediatric Surgeons.

Model 2: April 2018—tested at the International Pediatric Endoscopic Group Congress

Fifteen participants (1 novice, 3 intermediate, and 11 experts) performed parts of the EA/TEF repair on our model and completed a survey of their experience (Table 2). Participants reported seeing or performing a total of 515 open and 152 thoracoscopic EA/TEF repairs. Expert participants reported performing a combined estimate of 299 open or thoracoscopic EA/TEF repairs.

Table 2.

Results of Content Validity Assessment at IPEG 2018 for Model 2

Domain	Observed averages	Outfit mean square	Point-measure correlation
Physical attributes	4.17
Chest circumference	3.93	0.43	0.72
Chest depth	4.07	1.18	0.77
Intercostal space	4.53	1.13	0.46
Scale of insert within the model	4.13	0.73	0.34
Realism of materials	3.63
Overall impression	3.60	1.49	0.56
Skin	3.87	1.77	0.64
Ribs	4.33	0.65	0.86
Esophagus	2.73	1.34	0.42
Realism of experience	3.58
Anatomy during EA/TEF repair	3.73	0.65	0.76
Upper pouch anatomy	3.53	1.26	0.67
Movement of the insert during procedure	3.47	0.99	0.59
Ability to perform tasks	3.10
Closure of fistula	2.86	0.48	0.67
Dissection of upper pouch	3.69	0.97	0.67
Esophageal anastomosis	2.80	0.62	0.60
Value	4.47
As a training tool	4.60	0.78	0.50
As a testing tool	4.33	1.11	0.50
Relevance	4.40
Relevance to your training	4.40	2.08	0.64

IPEG, International Pediatric Endoscopic Group.

Table 3.

Results of Content Validity Assessment PAPS 2019 for Model 3^a

Domain	Observed averages	Outfit mean square	Point-measure correlation
Physical attributes	4.50
Chest circumference	4.40	0.97	0.08
Chest depth	4.50	0.74	0.60
Intercostal space	4.60	0.93	0.35
Scale of insert within the model	4.50	0.82	0.43
Realism of materials	4.25
Overall impression	4.30	0.93	0.01
Skin	4.00	0.07	0.00
Ribs	4.50	1.25	−0.25
Esophagus	4.20	0.69	0.10
Realism of experience	4.17
Anatomy during EA/TEF repair	4.20	0.64	0.22
Upper pouch anatomy	4.30	0.70	0.38
Movement of the insert during procedure	4.00	0.55	0.67
Ability to perform tasks	3.77
Closure of fistula	3.80	1.03	0.65
Dissection of upper pouch	4.10	1.26	0.16
Esophageal anastomosis	3.40	1.74	0.22
Value	4.80
As a training tool	5.00	—	0.00
As a testing tool	4.60	1.37	0.58
Relevance	4.60
Relevance to your training	4.60	1.79	0.13

References^3,4

PAPS, Pacific Association of Pediatric Surgeons.

The observed averages were: value as a training tool 4.5, relevance 4.4, physical attributes 4.2, realism of material and experience 3.6, and ability to perform tasks 3.10. There was written and verbal feedback that the silicone used was not sufficiently elastic and made the anastomosis difficult to perform. There were no significant differences in the observed averages for level of training for novice (3.95), intermediate (3.81), and expert (3.85) (P = .8712). The Cronbach's alpha was estimated to be 0.80, suggesting that these survey items are in general grouped appropriately.

Model 3: March 2019—tested at the Pacific Association of Pediatric Surgeons Meeting

Ten participants (5 intermediate and 5 experts) performed parts of the EA/TEF repair on our model and completed a survey of their experience (Table 3). Participants reported seeing or performing a total of 265 open and 70 thoracoscopic EA/TEF repairs.

The observed averages were: value as a training tool 4.8, relevance 4.6, physical attributes 4.5, realism of material 4.25, realism of experience 4.17, and ability to perform tasks 3.77. There were no significant differences in the observed averages for level of training with intermediate 4.37 and expert 4.21 (P = .12).

The Cronbach's alpha was estimated to be 0.78, suggesting that these survey items are in general grouped appropriately.

Comparison of models

There was a trend toward an improvement for physical attributes, realism of materials, ability to perform tasks, and value of simulator between the three models, but this was not statistically significant (Table 4). Relevance was slightly decreased between model one and model three but was still high at 4.6. There was no difference in response to the model based on the assessor's level of expertise. Realism of experience could not be compared due to the first model having an extra item “amount of pressure needed to place trocars into thoracic space” that was not used in Models 2 and 3.

Table 4.

Comparison of Responses for the Three Iterations of the Model

Observed averages	PSSNZ	IPEG	Workshop	Overall P value
Physical attributes	4.45	4.17	4.50	.1874
Realism of materials	4.15	3.63	4.25	.0708
Ability to perform tasks	3.31	3.10	3.77	.1473
Value	4.10	4.47	4.80	.1197
Relevance	4.80	4.40	4.60	.6532

IPEG, International Pediatric Endoscopic Group.

Discussion

There is an emerging need for better simulators to teach and assess surgeons as they acquire the skills required to perform EA/TEF repair competently and safely. The lack of commercially available simulators to do this led to a collaboration between engineers and surgeons to develop a fully synthetic EA/TEF simulator. However, demonstrating validity is essential to proving its usefulness as a training tool—it must achieve what it sets out to do. We have detailed the process by which we have designed and developed a fully synthetic simulator for EA/TEF repair and ensured its content validity through a sequential process of end-user assessment and structured feedback. For example, our design process has enabled modification of the silicone EA/TEF insert to obtain greater fidelity. The insert was initially noted to be too thick, varied in mobility and was not able to cope with the tension applied through suturing. Implementation of feedback allowed it to be improved by redesign and the use of different materials.

Use of a questionnaire adapted from Barsness et al.³ facilitated a stepwise overall improvement in the observed averages recorded across all domains over the three iterations of the model. The latest feedback confirms that the model now achieves good content validity as a neonatal thoracoscopic simulator for EA/TEF repair (Table 3).

Our model compares favorably with both the hybrid model and fully synthetic models described by Barsness et al.^3,4 (Table 5). Our synthetic model has the great advantage in that it removes the need for procurement and preparation of the bovine fetal tissues—in contrast to the described hybrid model.³ Moreover, our third model matches in realism of experience and exceeds in all other domains that of the previously described fully synthetic model.⁴

Table 5.

Comparison of Model 3 with Hybrid and Synthetic Models^a

Domain	Observed averages
Domain	Model 3	Hybrid model³	Synthetic model⁴
Physical attributes	4.50	4.5	4.1
Chest circumference	4.40		4.3
Chest depth	4.50		4.3
Intercostal space	4.60		4.3
Scale of insert within the model	4.50		4.2
Realism of materials	4.25	4.5	4.1
Overall impression	4.30		4.0
Skin	4.00	3.7	3.9
Ribs	4.50		4.1
Esophagus	4.20		—
Realism of experience	4.17	4.5	4.2
Anatomy during EA/TEF repair	4.20		4.1
Upper pouch anatomy	4.30		4.0
Movement of the insert during procedure	4.00		—
Ability to perform tasks	3.77	4.4	3.6
Closure of fistula	3.80		3.5
Dissection of upper pouch	4.10		3.8
Esophageal anastomosis	3.40	3.8	3.2
Value	4.80	4.2	4.1
As a training tool	5.00		4.4
As a testing tool	4.60		4.1
Relevance	4.60	4.7	4.1
Relevance to your training	4.60		4.1

References^3,4

Comparison of the hybrid model of bovine tissue and synthetic ribcage and skin³ with our synthetic model revealed observed averages for esophageal anastomosis of 3.8 and 3.4, respectively. Our model is, therefore, close to the realism provided by bovine tissue but has the distinct advantage of being achieved with fully synthetic materials. Nevertheless, this measure relates to the “ability to perform the task” on the simulator and may be more a reflection of the challenge of completing the esophageal anastomosis than a reflection of the material used within the simulator.

Another simulator for EA/TEF repair comprises a neonatal chest frame made of polyvinyl alcohol with a cut to simulate the esophagus to allow for intracorporeal suturing and knot tying.⁶ This model focused on suturing the front wall of the esophageal anastomosis and did not recreate the steps of fistula ligation, upper pouch separation, and the full esophageal anastomosis. Using the Barsness³ questionnaire score, the model scored highly (mean) for usefulness of the model for training 5.0, size of ribcage 4.8 and for skills assessment 4.5, the scores for realism of the materials were lower with the esophagus scoring 3.8. The ability to perform the suturing task was 3.3, which is “difficult to perform” and which may be indicative of the difficulty of the task rather than the specific setup of the simulator model.

The EA/TEF simulator described by Maricic et al.⁵ used easily accessible, synthetic material to recreate the esophagus. Two latex tubular balloons of 5 and 3 mm diameter were used to recreate the double layers of the esophagus. There were other structures included such as the azygos vein and vagus nerve to increase fidelity. Using a similar questionnaire, they showed that the model had good similarity that was 64% or a high degree of similarity that was 28% for esophageal dimensions. On comparing the simulated task with real surgery, respondents reported that ligation of TEF 8% had some similarity, 46% good similarity, and 46% high degree of similarity. The anastomosis showed “good similarity” in 56% and “high degree of similarity” in 46% of respondents. For the acquisition of suturing and anastomosis, 54% responded “can generate great majority of skill,” 28% “can generate good amount of skills,” and 18% “can generate some skills.” The model was tested for construct validity and able to distinguish between experienced and less experienced surgeons based on number of errors, time, and quality of anastomosis.

In the described simulators of Barsness, Deie, and our model, the scores for ability to perform the task were around the 3 mark, which is “difficult to perform” and which is likely related to the experience of the participants performing the task, perhaps in itself an indicator of the importance of having a simulator for this task! Esophageal anastomosis is technically demanding in vivo, and the experience in vitro reflects this, which is why this simulator has an important role in skill acquisition. Hence, given that thoracoscopic EA/TEF repair is technically challenging, this low score may be more an indication of the surgical difficulties faced rather than any specific failings of the simulator. Deie et al.⁶ identified construct validity and were able to detect a statistically significant difference between surgeons who had performed three or more thoracoscopic EA/TEF and those who had performed fewer than three. However, this was only for the single task of intracorporeal suturing and knot tying.

Validation of our model was limited by the sample size of participants, such that our observations may not be representative of the overall pediatric surgical community. Further, much of our data were obtained in conference settings, and this can lead to participants being distracted and where the time of trials was mostly to breaks between sessions.^4,15,16 This methodology may also lead to selection bias, as those who do not believe in thoracoscopic repair of EA/TEF or the use of simulators are less likely to participate in the validation of the model. Data collected from a thoracoscopic workshop (for Model 3) may also lead to selection bias, as attendees are keen to participate in both thoracoscopic surgical techniques and simulation training.

Although this model has achieved content validity, further research is required to assess its construct validity. This is to ensure that the model can distinguish between novice and expert surgeons, and the confirmation of skill acquisition, with particular focus being placed on validation of its effectiveness for skill acquisition for esophageal anastomosis. This necessitates fidelity of the silicone EA/TEF insert, which has been enhanced by the choice of materials (sufficiently soft and elastic silicone) and our double-layered model, which better replicates the separation and retraction that occurs between the mucosa and the muscle layers of the esophagus in vivo.

Ultimately, this type of model could be incorporated into the curriculum of specialty training boards in pediatric surgery. The high scores for physical characteristics, with the size and space of the ribcage, recreate real life. Participants in our study indicated that the simulator would be a good tool for developing technical skills, particularly for thoracoscopic needle manipulation within the small space of the neonatal chest.

Conclusion

Using an iterative design process with end-user feedback, we have attained content validity for this thoracoscopic EA/TEF simulator. It has significant advantages over alternative models of EA/TEF simulation in that by being fully synthetic it is free of the ethical and procurement issues associated with using animal tissues.

Footnotes

Acknowledgment

The authors would like to acknowledge Tara Gibb from Medical Illustration, Christchurch for her help and expertise in photographing the models.

Disclosure Statement

No competing financial interests exist.

Funding Information

No funding was received or used.

References

Okuyama

, Tazuke

, Ueno

, Yamanaka

, Takama

, Saka

, et al. Learning curve for the thoracoscopic repair of esophageal atresia with tracheoesophageal fistula. Asian J Endosc Surg, 2018; 11:30–34.

van der Zee

, Tytgat

, Zwaveling

, van Herwaarden

, Vieira-Travassos

. Learning curve of thoracoscopic repair of esophageal atresia. World J Surg, 2012; 36:2093–2097.

Barsness

, Rooney

, Davis

. Collaboration in simulation: The development and initial validation of a novel thoracoscopic neonatal simulator. J Pediatr Surg, 2013; 48:1232–1238.

Barsness

, Rooney

, Davis

, O'Brien

. Evaluation of three sources of validity evidence for a synthetic thoracoscopic esophageal atresia/tracheoesophageal fistula repair simulator. J Laparoendosc Adv Surg Tech A, 2015; 25:599–604.

Maricic

, Bailez

, Rodriguez

. Validation of an inanimate low cost model for training minimal invasive surgery (MIS) of esophageal atresia with tracheoesophageal fistula (AE/TEF) repair. J Pediatr Surg, 2016; 51:1429–1435.

Deie

, Ishimaru

, Takazawa

, Harada

, Sugita

, Mitsuishi

, et al. Preliminary study of video-based pediatric endoscopic surgical skill assessment using a neonatal esophageal atresia/tracheoesophageal fistula model. J Laparoendosc Adv Surg Tech A, 2017; 27:76–81.

IRCAD. Pediatric Surgery Laparoscopy in Newborns, Infants & Children. 2019. Available from: https://www.ircad.fr/training-center/course-calendar/?type=advanced&spec=pediatric (accessed June 11, 2019).

University of Minho. Master Class Skills Lab for Live Animal Model (Pediatric Laparoscopy). 2019. Available from: https://www.med.uminho.pt/en/Post-Graduation/courses/pediatric-laparoscopy (accessed June 11, 2019).

IPEG. IPEG Mastery Learning Series Course. 2019. Available from: https://www.ipeg.org/ipeg-mastery-learning-series-course/ (accessed June 11, 2019).

10.

Schwab

, Hungness

, Barsness

, McGaghie

. The role of simulation in surgical education. J Laparoendosc Adv Surg Tech A, 2017; 27:450–454.

11.

Fedorov

, Beichel

, Kalpathy-Cramer

, Finet

, Fillion-Robin

, Pujol

, et al. 3D Slicer as an image computing platform for the Quantitative Imaging Network. Magn Reson Imaging, 2012; 30:1323–1341.

12.

Kikinis

, Pieper

, Vosburgh

. 3D Slicer: a platform for subject-specific image analysis, visualization, and clinical support. In: Jolesz

, (ed.). Intraoperative Imaging Image-Guided Therapy. New York: Springer Science + Business Media, 2014, pp. 277–289.

13.

Blender. Blender v2.78 and later. 2016. Available from: https://www.blender.org/ (accessed April 15, 2020).

14.

Spitz

. Oesophageal atresia. Orphanet J Rare Dis, 2007; 2:24.

15.

de Montbrun

, MacRae

. Simulation in surgical education. Clin Colon Rectal Surg, 2012; 25:156–165.

16.

Schout

, Hendrikx

, Scheele

, Bemelmans

, Scherpbier

. Validation and implementation of surgical simulators: a critical review of present, past, and future. Surg Endosc, 2010; 24:536–546.