Assessment of OB/Gyn Residents’ Knowledge of Basic Obstetric Sonography and Psychomotor Skills Posttraining on a Computer-Based Simulator

Abstract

Objective:

This study sought to evaluate how a high-fidelity computer-based sonography simulator (FCBSS) can be used in training obstetrics and gynecology residents and evaluate efficacy of high-fidelity CBSS in enhancing resident’s sonographic knowledge, psychomotor skills, and level of confidence in performing sonographic fetal assessments.

Materials and Methods:

Eleven postgraduate year 1 residents (PGY1s) and ten PGY2s were assessed on sonographic performance by measuring psychomotor skills and evaluating the accuracy of the sonographic images. PGY2s received traditional sonography training while PGY1s received three individualized training sessions on a high-fidelity CBSS. At the conclusion, all residents received feedback and completed a survey to establish their current confidence level in performing and interpreting sonograms.

Results:

PGY2s scored a median of 33/66 points on the assessment, while PGY1s scored a median of 64/66 points. Statistical analysis performed from motion analysis metrics between cohorts revealed a statistically significant difference, with PGY1s demonstrating higher psychomotor skills. Interquartile ranges of PGY1s’ scores were smaller than PGY2s’, revealing consistency in knowledge and skills among the PGY1 cohort.

Conclusion:

Findings suggests that high-fidelity CBSS can be used effectively in training residents. It also demonstrates that implementation of formalized sonography training, using a high-fidelity CBSS, can be achieved and potentially expedite and enhance the learning of novice learners.

Keywords

educational technology simulation training biometry obstetric imaging

Performing and interpreting sonographic images is an essential component in the practice of obstetrics and gynecology (OB/Gyn).¹ Sonography is regarded as the imaging modality of choice for OB/Gyn imaging evaluation.¹ This imaging modality offers multiple advantages, including lack of ionizing radiation, real-time evaluation, portable equipment, wide-accessibility and cost-efficiency. Nevertheless, sonography is an operator-dependent modality that heavily relies upon the user’s expertise and ability to demonstrate high psychomotor skills. Given the high demand for the use of sonography, it is imperative that OB/Gyn residency programs provide each individual with sufficient opportunity for hands-on scanning to learn and develop the necessary psychomotor skills. The American Institute of Ultrasound in Medicine’s (AIUM) official statement asserts the need to provide evidence of involvement in the performance, evaluation, interpretation, and reporting of a minimum of 300 diagnostic obstetrics (OB) sonograms and 300 diagnostic gynecologic (Gyn) sonograms in order to achieve competency in OB/Gyn sonography.^2,3 However, the Accreditation Council for Graduate Medical Education (ACGME) released a National Data Report of residents completing OB/Gyn residency programs in 2016–2017, which revealed a mean of 107 OB and 67 Gyn sonographic examinations performed during training.⁴ This would point to a possible deficiency in the number of sonograms performed at the end of an OB/Gyn residency program, for many individuals.

Currently, there is a lack of standardized sonography training among OB/Gyn residency programs. Traditionally, residents learn sonography skills within the clinical setting, which provides observation and hands-on sonographic scanning. This can result in undue stress as trainees attempt to learn basic sonography skills on real patients, while facing pressures attributed to a fast-paced clinical environment.^5,6 In addition, this approach can result in a gap of knowledge and skill set among OB/Gyn residents, depending on their clinical exposure. A study conducted by Mohammad et al found a wide range in sonographic knowledge and psychomotor skills among PGY2 residents’ within the same cohort.⁷ In the absence of a uniform approach to sonography training in the clinical environment, a potential solution could be to utilize a high-fidelity computer-based sonography simulator (CBSS) as a learning tool.

High-fidelity CBSSs have been utilized in health care education for many years, providing a safe and controlled environment for advanced learning.^8–11 Use of CBSSs also allows for appropriate educational assessments; yet, the integration of sonography simulation in residency program education is not well documented. Less than 20% of residency programs reported having experience with sonography simulation.¹² Currently, limited research exists on sonography simulation within OB/Gyn residency programs; however, the literature that does exist on simulation education reveals promising results.^5,13–18 To this end, this research evaluated the use of high-fidelity CBSS in expediting the education of residents in basic obstetric sonographic examinations. Thus, the research hypothesis was that high-fidelity CBSS would expedite and enhance an OB/Gyn resident’s sonographic knowledge and psychomotor skills in obtaining accurate fetal images as well as increase self-confidence in their sonographic skills.

Materials and Methods

Participants

Incoming postgraduate year 1 (PGY1) residents and incoming postgraduate year 2 (PGY2) residents were invited to participate in this institutional review board–approved study. A total of 11 PGY1 residents and 10 PGY2 residents consented to this study. As part of their first year of residency, all participants had been introduced to sonography through a limited lab during their orientation. This 60-minute lab included three stations where residents were provided a brief overview of knobology, fetal presentation, and were shown images of biometry views. In addition all residents rotated through multiple clinical sites throughout their first year, at different times with varying levels of exposure to sonography.

Traditionally, one of the scheduled rotations consists of a dedicated sonography rotation where residents work alongside sonographers to scan patients and learn further about the modality. All PGY2 residents completed this assigned rotation in their first year. However, in place of this rotation, PGY1 residents who participated in this study were instead scheduled a rotation for OB sonography training, using a high-fidelity CBSS (CAE Vimedix OB/Gyn simulator), during the first 7 months. This training was supervised and conducted by a sonographer registered in diagnostic medical sonography and vascular technology (author S.H.M). This sonography training was conducted within the first 7 months of the PGY1’s first year of residency.

Equipment

A CAE Vimedix OB/Gyn sonography simulator (CAE Healthcare, Quebec, Canada) was used for this research. This is a high-fidelity CBSS consisting of the following equipment: computer’s central processing unit (CPU), monitor, keyboard, mouse, curvilinear transducer, and a female manikin. Although the computer set up and system does not replicate that of an ultrasound machine, the system does allow for acquiring measurements, saving images/cine clips, and limited optimization (e.g., depth, gain, contrast). The system also includes additional features such as a split screen of a sonography and 3D augmented reality display, selection of over 50 pathology cases, and a unique feature that captures transducer motion data, called Target Cut Plane (TCP). Once the TCP function is activated, the following quantifiable kinematic metrics are recorded:

Time—Total time, in seconds, from the start of the scan until an image is acquired

Distance traveled—The sum of the total distance, in centimeters, traveled with the transducer from the start of the scan until an image is acquired

Peak movements—Number of accelerations made from rest, determined by the transducer transitioning from a speed of 0.05 cm/s to a speed greater than 0.05 cm/s, made from the start of the scan until an image is acquired

Angular movement—The sum of angular movements (degrees) made with the transducer from the start of the scan until an image is acquired

Simulation Training

Each PGY1 resident was scheduled for three, one-on-one 2-hour long sessions using a CAE Vimedix OB/Gyn simulator (CAE Healthcare, Quebec, Canada) with a registered diagnostic medical sonographer (RDMS; author S.H.M). The research team and residency program faculty set the number of training sessions based on the residency clinical rotation schedule and determined that 2 hours per session would provide residents with sufficient time to develop the foundational skills required. The training sessions were guided by the preliminary feedback of PGY2 residents. A detailed account of the PGY2 residents’ assessment findings and feedback were published earlier by Mohammad et al.⁷ A lesson plan and set of instructional materials were created by the first author (S.H.M) to provide didactic information during each session. Didactic topics included

Imaging Planes,

Transducer Orientation,

Fetal Presentation,

Fetal Situs,

Overview of Basic Fetal Anatomy,

Biometry Views and Measurements (i.e., biparietal diameter [BPD], head circumference [HC], abdominal circumference [AC], femur length [FL]),

Fetal Heart Rate (FHR),

Documentation of Upper and Lower Extremities (UE and LE), and

First Trimester Dating Scan.

Didactics were integrated with hands-on training during each session to allow for a better understanding of the educational material. For example, after reviewing the criteria needed to acquire a diagnostically accurate image for fetal biometry head measurements, the residents were then instructed to attempt to acquire this view and the corresponding measurements.

During training, direct feedback was provided to the residents either as they were working through acquiring the image or after they acquired the image. Direct feedback consisted of providing constructive information to the resident with specifics on the images captured, as well as transducer movements made during image capture. The goal was to facilitate a better understanding of fetal anatomy and improve hand-eye coordination. After each training, a debrief session was held wherein the RDMS presented open-ended questions to the resident, reviewed scanning performance (e.g., what aspects were performed well and common themes identified that needed further development), encouraged residents to reflect on their performance, and ask specific questions.

Assessment of Residents

All residents were assessed during a one-on-one 60-minute session on a CAE Vimedix simulator with a proctor (author S.H.M) present. PGY1s were assessed just prior to the start of their training and once again at the end of their third day of training to document pre- and posttraining results. As the focus of the study was comparison of PGY1s with simulation training to PGY2s with a year of clinical experience and no simulation training, only limited results will be provided to demonstrate the PGY1 cohort’s baseline abilities. All participants were assessed on their psychomotor skills and knowledge of the following: fetal presentation, fetal situs, fetal heart rate (FHR), biometry views/measurements, and documentation of bilateral upper and lower fetal extremities on a 20-week fetus, as well as acquiring crown-rump length (CRL) images/measurements on a 12-week fetus. The assessment items were based, in part, on a consensus report that provides a proposed standardized sonographic curriculum and a list of competency assessments included for PGY1-4 OB/Gyn residents.¹⁹ Before finalizing the list of assessment items for the current study, the list was reviewed by the OB/Gyn residency director and a department faculty member. The assessment list for this study was also constrained by the exams available on the CAE high-fidelity CBSS. Sonographic performance and knowledge were evaluated via a combination of kinematic metrics obtained with target cut planes (TCPs) and an assessment form. TCP metrics were used to document psychomotor skills, and an assessment form was used to document the diagnostic accuracy of images acquired as well as residents’ knowledge of fetal presentation and situs. In addition, two expert sonographers registered in diagnostic medical sonography (RDMS) were invited to scan with the TCP package activated, and extracted metrics were reviewed for face validity and saved for comparative reference metrics. Details of the TCP feature and assessment form can be found in Mohammad et al.⁷ After the assessment and feedback/debrief session, a survey was provided to the residents to establish the resident’s level of confidence in performing and interpreting a basic OB sonogram. The PGY1s also provided feedback on this particular set of teaching techniques, including the use of the high-fidelity CBSS.

Data collected on the PGY1s were compared with similar data collected on PGY2s, who did not receive this set of teaching techniques. Data on PGY2s, who just completed their first year of residency, were collected in a one-on-one data collection session that was analogous to the conditions under which the PGY1 data were collected.

Statistical Analysis

The data gathered were entered into Microsoft Excel (Microsoft, Redmond, Washington, USA), and this program was used to generate descriptive statistics. The data demonstrated nonnormal distribution; therefore, median and interquartile ratios were generated, instead of means and standard deviations. In addition, Statistical Package for the Social Sciences (SPSS) package version 26 (IBM, Armonk, NY, USA) was used to analyze the TCP metrics from residents’ 20-week assessment. For the sum of each TCP metric, Mood’s median test was used to compare PGY1 (Posttraining) and PGY2 residents’ data from the 20-week assessment. This nonparametric test was selected due to data demonstrating a nonnormal distribution and large difference in variance between PGY1 (Posttraining) and PGY2 TCP data. The a priori level for statistical significance was set at P < .05.

Results

Assessment Form Scores

Boxplots presenting template assessment scores of PGY1 residents (baseline and postsimulation training) and PGY2 residents, who received no simulation training the first year of residency, can be found in Figure 1. Limited baseline PGY1 residents’ data are provided to demonstrate the starting median values prior to simulation training. PGY1 residents’ baseline median score on the assessment template was 34 out of 66 points (interquartile range [IQR], 23–40), with the following median values per category: Fetal presentation and situs (4/7), FHR (3/3), Biometry views and measurements (16/25), documentation of bilateral extremities (7/22), and first trimester dating scan on a 12-week fetus (3/9). Posttraining, the PGY1 residents’ median score on the assessment template was 64 out of 66 points (IQR, 62–65), with the following median values per category: Fetal presentation and situs (7/7), FHR (3/3), Biometry views and measurements (24/25), documentation of bilateral extremities (22/22), and first trimester dating scan on a 12-week fetus (9/9). In comparison, PGY2 residents’ median score on the assessment template was 33 out of 66 points (IQR, 26.75–42.5), with the following median values per category: Fetal presentation and situs (3.5/7), FHR (3/3), Biometry views and measurements (15/25), documentation of bilateral extremities (7.5/22), and first trimester dating scan on a 12-week fetus (3/9). A detailed account of PGY2 residents’ assessment findings can be found in an earlier publication by Mohammad et al.⁷

Figure 1.

Boxplots presenting template assessment scores of PGY1 residents’ baseline data, PGY1 residents (posttraining) and PGY2 residents (who received no simulation training). Maximum score possible is 66. The 75th and 100th percentile for PGY1 (Post) Residents’ Scores are both 66. PGY1, postgraduate year 1 residents; PGY2, postgraduate year 2 residents.

At the start of the assessment, two fetal presentations of a 20-week fetus were provided to the residents: breech and a transverse lie. During posttraining assessment, all 11 PGY1s correctly identified the presentations and were able to identify the left side of the fetus. Furthermore, in acquiring biometry views and measurements, although the PGY1s had higher median scores compared with PGY2s (24/25 and 15/25, respectively), there were a few key anatomy structures and measurement criteria that some residents, in both cohorts, failed to capture for HC/BPD and AC images (Table 1 and Figure 2). In obtaining the BPD measurement, seven PGY2s and three PGY1s failed to obtain measurements that were perpendicular to the cerebral falx. Similar to a majority of PGY2s (6 out of 10), two PGY1s struggled to capture an AC image with a circular abdomen and demonstrate a single pair of fetal ribs. However, unlike the PGY2s all 11 PGY1 residents met the three criteria for the femur length image and obtained an accurate femur length measurement. This contrasts with findings from the PGY2 cohort where the median scores for the femur length image and measurement were 1.5/3 and 0.5/1, respectively. In addition, three PGY2 residents did not select the femur in the near field of the image, six failed to put the femur perpendicular to the ultrasound beam, four did not fully elongate the femur bone, and two imaged/measured the humerus for a femur length. None of these errors were committed by PGY1 study participants during their posttraining assessment. In addition, unlike the PGY2 cohort, all PGY1 residents were able to correctly identify, locate, and document bilateral lower and upper extremities in the long axis view. In contrast, some PGY2s had difficulty determining the left side of the fetus, distinguishing between upper and lower extremities, and distinguishing the right and left extremities.

Table 1.

Frequency of Criteria That PGY2 and PGY1 (Posttraining) OB/Gyn Residents Either Did or Did Not Meet for the Diagnostic Quality Assessment Criterion for HC/BPD Sonographic Image and BPD Measurement.

Criterion for BPD/HC Sonographic Image	PGY2, n = 10		PGY1 (Post), n =11
Criterion for BPD/HC Sonographic Image	No	Yes	No	Yes
Transverse plane of the head^a	0	10	0	11
Cavum septum pellucidum visualized	7	3	1	10
Thalamus visualized entirely	6	4	0	11
Posterior horns of the ventricles are visualized	6	4	0	11
Symmetrical appearance of both cerebral hemispheres	6	4	1	10
Cerebral falx is perpendicular to the ultrasound beam	6	4	3	8
Cerebellum is not visualized	2	1	0	11
Notch of the transducer toward the right side of the manikin	1	9	0	11
Criterion for BPD measurement	PGY2 n = 10		PGY1 (Post) n = 11
Criterion for BPD measurement	No	Yes	No	Yes
Caliper in the near field on the outer edge of cranium	8	2	0	11
Caliper in the far field on the inner edge of cranium	2	8	1	10
Measurement at the widest diameter of the cranium	5	5	0	11
Measurement is perpendicular to the cerebral falx	7	3	3	8

Abbreviations: BPD, biparietal diameter; HC, head circumference; OB/Gyn, obstetrics and gynecology; PGY1, postgraduate year 1 residents; PGY2, postgraduate year 2 residents.

Criteria must be met, otherwise a score of zero will be given for this image.

Figure 2.

(A) A fetal BPD sonogram produced by a PGY1 resident (posttraining), for a biometry measurement. The image was taken at the appropriate level for fetal head biometry measurements. Yet, the BPD measurement made on the image is not perpendicular to the cerebral falx. (B) A fetal BPD sonogram produced by a PGY2 resident for a biometry measurement. The image was not taken at the appropriate level for the fetal head biometry measurements, resulting in missing key anatomy structures. Used with permission from CAE Healthcare Inc. BPD, biparietal diameter; PGY1, postgraduate year 1 residents; PGY2, postgraduate year 2 residents.

Review of first trimester sonograms, completed for gestational age, demonstrated that all PGY1s (posttraining) acquired a minimum of three CRL measurements. However, only six PGY1 residents had all three measurements, made on a diagnostic quality 2D image, that were within the accurate range for the 12-week fetus (Figure 3A). The remaining five PGY1 residents obtained diagnostic quality 2D images with accurate measurements for two of the three CRLs images. The errors for these residents were attributed to needing to further elongate the fetus, not capturing a mid-sagittal view and/or caliper placement error (Figure 3B). In contrast, only two PGY2 residents acquired a single image of a diagnostic quality image of a midline sagittal view of the fetus. However, in both cases, the calipers were not placed at the longest axis of the fetus. The remaining images captured by PGY2 residents did not meet the criteria and zero PGY2 residents obtained accurate CRL measurements (Figure 3C).

Figure 3.

(A) Crown-rump length image and measurement produced by a PGY1 resident, posttraining. The captured image and measurement meet the required criteria for a CRL measurement. (B) Crown-rump length image and measurement produced by a PGY1 resident, posttraining. The image obtained is not of the mid-sagittal view of the fetus, which resulted in a smaller CRL measurement. (C). Crown-rump length image and measurement produced by a PGY2 resident. The image captured is an oblique sagittal view of the fetus rather than a mid-sagittal view, which resulted in a smaller CRL measurement. Used with permission from CAE Healthcare Inc. CRL, crown-rump length; PGY1, postgraduate year 1 residents; PGY2, postgraduate year 2 residents.

Target Cut Plane Metrics

Utilizing TCP metrics, researchers were able to document the total time distance traveled (cm), angular movement (degrees), and peak movements made with the transducer from start to the end of acquiring specific view(s). Table 2 provides descriptive data generated from the CAE Vimedix for the cohort of PGY1s (posttraining) and the sonography experts. Sonography expert data were collected for a benchmark threshold. The PGY1 cohorts’ TCP metrics for the AC view and left UE documentation were close to the expert sonography level. It was also consistently documented that the median total distance traveled, per view(s), were generally close to the expert level. Figures 4 and 5 provide visual comparisons of the median TCP metrics captured for the AC view by the PGY2 cohort, PGY1 cohort (posttraining), and expert sonographers.

Table 2.

Descriptive Statistics for TCP Data Generated by the CAE Vimedix for the Cohort of PGY1 Study Participants (Posttraining) and Experts.

View(s)	TCP Metrics	PGY1 (Post), n = 11				Experts (n = 2)	Median Difference (PGY1 − Expert)
View(s)	TCP Metrics	Min	Median	Max	IQR	Median	Median Difference (PGY1 − Expert)
HC/BPD	Time (seconds)	23	40	89	24	12	28
	Distance traveled (cm)	3	14	30	16	5	9
	Angular movement (degrees)	65	253	547	316	50	203
	Number of peak movements	2	19	43	13	6	14
AC	Time (seconds)	13	26	48	19	19	7
	Distance traveled (cm)	13	15	22	5	17	−2
	Angular movement (degrees)	49	100	222	93	99	1
	Number of peak movements	4	13	32	18	10	4
RLE	Time (seconds)	46	80	122	23	37	43
	Distance traveled (cm)	22	35	62	16	30	5
	Angular movement (degrees)	357	668	877	272	539	128
	Number of peak movements	21	54	95	40	30	24
LLE	Time (seconds)	52	76	187	28	38	39
	Distance traveled (cm)	23	37	70	36	28	9
	Angular movement (degrees)	469	613	1738	665	480	134
	Number of peak movements	31	51	101	56	28	23
RUE	Time (seconds)	36	51	92	44	18	33
	Distance traveled (cm)	15	20	46	12	16	5
	Angular movement (degrees)	204	402	1027	329	248	154
	Number of peak movements	16	29	96	15	13	16
LUE	Time (seconds)	37	50	101	21	24	27
	Distance traveled (cm)	14	17	26	6	17	0
	Angular movement (degrees)	256	383	759	335	317	55
	Number of peak movements	8	20	30	14	20	0

Abbreviations: AC, abdominal circumference; HC/BPD, head circumference/biparietal diameter; IQR, inter-quartile range; LLE, left lower extremity; LUE, left upper extremity; Min, minimum; Max, maximum; PGY1, postgraduate year 1 residents; RLE, right lower extremity; RUE, right upper extremity; TCP, target cut plane.

Figure 4.

Median TCP metrics on total time (seconds), total distance traveled (degrees), and number of peak movements made by PGY2, PGY1 (Posttraining) and expert sonographers in the process of acquiring the abdominal circumference view. PGY1, postgraduate year 1 residents; PGY2, postgraduate year 2 residents; TCP, target cut plane.

Figure 5.

Median TCP metrics on total angular movement (degrees) made by PGY2, PGY1 (Posttraining) and expert sonographers in the process of acquiring the abdominal circumference view. PGY1, postgraduate year 1 residents; PGY2, postgraduate year 2 residents; TCP, target cut plane.

Furthermore, comparative cohort TCP metrics (see Table 3) revealed that PGY1 residents, posttraining, had median values that tended to be lower than the PGY2s, for all views but the fetal head image. The largest differences between cohorts (PGY1 and PGY2) for median values were noted for total angular movement (degrees), with the largest difference documented for the right lower extremity (RLE) (Figure 6). Within the cohort of PGY1s, the IQR of TCP data were lower for all sonographic views, compared with PGY2s, except for metrics related to the fetal head image.

Table 3.

Comparison of Descriptive Statistics for the TCP Data Generated by the CAE Vimedix for the PGY1 (Posttraining) and PGY2 Resident Cohorts.

View(s)	TCP Metrics	PGY2, n = 10		PGY1 (Post), n = 11		Median Difference (PGY2 − PGY1)
View(s)	TCP Metrics	Median	IQR	Median	IQR	Median Difference (PGY2 − PGY1)
HC/BPD	Time (seconds)	36	38	40	24	−4
	Distance traveled (cm)	12	17	14	16	−2
	Angular movement (degrees)	193	297	253	316	−30
	Number of peak movements	21	20	19	13	2
AC	Time (seconds)	42	56	26	19	16
	Distance traveled (cm)	27	40	15	5	12
	Angular movement (degrees)	487	731	100	93	387
	Number of peak movements	48	59	13	18	35
RLE	Time (seconds)	177	142	80	23	97
	Distance traveled (cm)	104	149	35	16	69
	Angular movement (degrees)	2227	2242	668	272	1559
	Number of peak movements	183	234	54	40	129
LLE	Time (seconds)	86	84	76	28	10
	Distance traveled (cm)	48	101	37	36	11
	Angular movement (degrees)	833	1795	613	665	220
	Number of peak movements	63	112	51	56	12
RUE	Time (seconds)	70	62	51	44	19
	Distance traveled (cm)	47	65	20	12	27
	Angular movement (degrees)	859	1562	402	329	457
	Number of peak movements	78	91	29	15	49
LUE	Time (seconds)	87	80	50	21	37
	Distance traveled (cm)	48	94	17	6	31
	Angular movement (degrees)	1319	1142	383	335	936
	Number of peak movements	87	81	20	14	67

Abbreviations: AC, abdominal circumference; HC/BPD, head circumference/biparietal diameter; IQR, interquartile range; LLE, left lower extremity; LUE, left upper extremity; PGY1, postgraduate year 1 residents; PGY2, postgraduate year 2 residents; RLE, right lower extremity; RUE, right upper extremity; TCP, target cut plane.

Figure 6.

Median TCP metrics on total angular movement (degrees) made by PGY2s, PGY1s (Posttraining) and expert sonographers in the process of documenting the right lower extremity. PGY1, postgraduate year 1 residents; PGY2, postgraduate year 2 residents; TCP, target cut plane.

Statistical analysis performed on the sum of each of the TCP metrics between PGY1 (posttraining) and PGY2 cohorts revealed a statistically significant difference in each of the four metrics gathered (Table 4). The descriptive values of the sum of the four TCP metrics gathered during the 20-week assessment are also provided in Table 4 in addition to differences in median values between PGY1 (Post) versus PGY1 (Baseline), PGY1(Post) and PGY2 cohorts, PGY2 and Experts, and PGY1(Post) and Experts. PGY2s tended to have lower median values in each of the TCP metrics compared with PGY1s’ baseline assessment. However, PGY2s had significantly larger values in each of the TCP metrics compared with the PGY1s’ posttraining assessment. In addition, PGY1s’ posttraining TCP metrics were generally closer to the median values of the experts than the median values obtained by PGY2s. Furthermore, TCP metrics gathered between pre- and post-simulation training reveal general improvement in psychomotor skills in each of the four TCP metrics gathered, particularly in total angular movement.

Table 4.

Median Values Are Presented for the Sums of Each TCP Metric Gathered for PGY2, PGY1 (Baseline and Posttraining), and Experts on the 20-Week Assessment. Differences Between Median Values Are Presented: PGY1 (Post) vs PGY1 (Baseline), PGY2 vs PGY1 (Post), PGY2 vs Expert, and PGY1(Post) vs Experts. Median Test P Values.

TCP Metrics	Medians				Median Differences				P Value
TCP Metrics	PGY2	PGY1 (Baseline)	PGY1 (Post)	Experts	PGY1 (Baseline) − PGY1 (Post)	PGY2 − PGY1 (Post)	PGY2 − Experts	PGY1 (Post) − Experts	PGY1 (Post) vs PGY2
Total time (seconds)	498	612	324	148	288	174	350	176	.005*
Total distance traveled (cm)	286	359	138	113	221	148	173	25	<.001*
Total angular movement (degrees)	5918	8174	2419	1733	5755	3499	4185	686	<.001*
Number of peak movements	480	575	186	107	389	294	373	79	<.001*

Level of significance P < .05, confidence level = 95%.

Abbreviations: PGY1, postgraduate year 1 residents; PGY2, postgraduate year 2 residents; TCP, target cut plane.

Significant difference.

Survey on Confidence Level, Scanning Opportunities, and Simulation

By the conclusion of the series of three-training sessions, the PGY1s agreed that they were confident in their ability to identify and locate fetal anatomy. They also felt that they were able to obtain/identify the following without assistance: fetal biometry views and measurements, evaluate fetal situs, and obtain all fetal longs bones (based on their psychomotor skills) (Table 5). While the median rating for PGY2s’ confidence to identify, locate fetal anatomy, obtaining fetal biometry views, and measurements, was equal to the PGY1 cohort (4 = agree), PGY2s had lower median ratings in their confidence in evaluating fetal situs and their psychomotor skills in obtaining all fetal long bones (Table 5). Furthermore, both cohorts had a median rating of 4, agreeing with the statement (survey item #2) that they are confident in performing a first trimester dating scan without assistance. However, the range of ratings for PGY1s extended lower compared to that of PGY2s (Table 5). The lowest rating for survey item #2 was a neutral response for PGY1s, in contrast to the lowest rating provided by PGY2s which was “strongly disagree” (Table 5).

Table 5.

Median Values and Ranges of Values From the PGY2 and PGY Study Participants, in Response to a Post-Study Survey Assessing Participants’ Perceptions About Their Scanning Abilities and the Value of Hands-On Practice With a High-Quality CBSS.

Statements	PGY2 (n = 10)		PGY1 (n = 11)
Statements	Median	Range	Median	Range
1. I am confident in my ability to identify and locate fetal anatomy	4	[3–4]	4	[4–5]
2. At this time, I am confident that I can perform a first trimester dating scan without assistance.	4	[1–5]	4	[3–5]
3. At this time, I am confident that I can obtain fetal biometry views and measurements without assistance.	4	[4–5]	4	[4–5]
4. At this time, I am confident in my ability to evaluate fetal situs, without assistance.	3	[2–4]	5	[4–5]
5. To date, I am confident in my psychomotor skills to obtain all fetal long bones, without assistance.	3.5	[2–4]	4	[4–5]
6. To date, I have had sufficient hands-on scanning opportunities to strengthen my scanning skills.	3	[2–5]	4	[2–5]
7. To date, I feel that I’ve had the opportunity to practice OB scanning at my own pace.	3	[2–4]	4	[2–5]
8. To date, I feel that I’ve had enough opportunity to practice through trial and error.	4	[2–5]	4	[3–5]
9. The simulator is a useful tool to practice normal and abnormal anatomy.	5	[4–5]	5	[4–5]
10. I believe I would benefit from practicing on the simulator.	5	[4–5]	.	.
11. The direct feedback throughout the sessions helped enhance my learning.	.	.	5	[4–5]
12. How much did this hands-on method (simulation) contribute to your learning.	.	.	5	[5–5]

Response choices: #1–10: 1 = Strongly Disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly Agree; #11 and 12: 1 = None, 2 = A little, 3 = Some, 4 = Moderate, 5 = A lot.

Abbreviations: CBSS, computer-based sonography simulator; OB, obstetrics; PGY1, postgraduate year 1 residents; PGY2, postgraduate year 2 residents.

Most PGY1s noted that they have had sufficient hands-on scanning opportunities to strengthen scanning skills (survey items #6 and 7). This was contested by two residents who provided lower ratings, stating, “Yes in a simulated setting, just would appreciate some instruction at bedside” and another stating “Most of hands-on training [in the clinical setting] has been in triage, where time is critical” for survey item #7, and “more hands-on, supervised/instructional training would be helpful and more simulation” for survey item #6 (Table 5). Although the median ratings for survey items #6 and 7 are higher for PGY1s compared with PGY2s (4 vs. 3, respectively), a common theme expressed by residents who provided lower ratings was a lack of scanning opportunities in their busy clinical facility (Table 5).

Moreover, both PGY1s and PGY2s believed that the simulator is a useful tool to practice normal and abnormal anatomy (Table 5). PGY2s had not received high-fidelity CBSS training but were also asked to rate the benefit of practicing on the CAE simulator. The PGY2s all agreed on the benefit of the high-fidelity CBSS (Table 5). PGY1s also agreed that the direct feedback provided by the instructor enhanced their learning, with residents stating, “This was an incredible resource to have one-on-one direct feedback.” In addition, all PGY1s provided a Likert rating of 5 (“a lot”), for the degree to which this hands-on learning method (using a high-fidelity CBSS) contributed to their learning (survey item #12 in Table 5). PGY1 residents’ responses to open-ended questions regarding overall experience with sonography simulation are included in Table 6. Corresponding responses provided by PGY2 residents can be found in Mohammad et al.⁷

Table 6.

Open-Ended Survey Questions and Corresponding Responses From Postgraduate Year 1 (PGY1) Residents Who Were Trained and Assessed With the CAE Vimedix.

What did you like least about the simulations? What did you like best about the simulations?

• “I liked least that there are only 3 sessions. I wish we could have had more and that we could have reviewed more because the sessions I did have were so helpful in contributing to my learning. I love the one-on-one sessions with an ultrasound expert. Every session and question and answer were specifically tailored to me and my strengths/weaknesses which helps optimize individual growth! These were the best and I would love more sessions.”
• “Practicing how to evaluate in a systematic way and therefore knowing where you are on the fetus when your view becomes distorted. Also, practicing the biometry was amazing. Also learning the tips/tricks to adjust your view make it so much easier.”
• “It would be nice if the baby was able to move. The fact that it wasn’t able to move, however, was great for establishing baseline skills though.”
• “Just wish that the simulator had movement, I feel like the hardest part of getting growth exams in real life is with patient body habitus. It is very helpful to get to learn anatomy in a controlled setting.”

Do you believe the simulator is a useful tool to have as a resident? Why or why not?

• “I got substantially more from these sessions than the sessions where I just shadowed sonographers. The hands-on, one-on-one teaching tailored to each student and focusing on fundamentals has made me so much more confident in my ultrasound abilities as a new intern. I feel that I now, because of the simulator and one-on-one sessions, I have a SOLID foundation that I can continue to expand upon.”
• “Absolutely it reduces the fear/anxiety associated w/gaining comfort w/ motions and placement that one has on a live patient learning w/trial and error.”
• “This is a useful tool. It provides a low stress learning environment and allows for real-time feedback of image quality.”
• “Yes, yes, yes. I feel so much more confident in my ability to date pregnancies, evaluate position and biometry. This should be incorporated for all future classes of residents.”

Discussion

Traditionally, residents learn how to create and interpret sonographic images in their clinical setting, with help from faculty, peers, and sonographers. However, this can be problematic as residents may be unable to gain clinical mastery of skills due to their clinical demands, high patient volumes, and time constraints.^18,20 In this study, some residents identified time and patient volumes as factors that limited their opportunity to strengthen their sonographic skills. Another challenge in learning sonography, in the clinical environment, is the lack of a uniform approach to learning basic sonography skills and the unpredictability of cases that present in the department.²¹ As such, residents may not have a smooth transition from completing simple and/or normal sonographic cases to executing sonograms for complex cases, which require a higher level of clinical mastery.^20–22 Moreover, differences in clinical exposure can create gaps in knowledge and sonographic performance among residents. This was noted when reviewing the PGY2s’ assessment scores and TCP metrics, which had larger IQRs compared with PGY1s’.⁷ The absence of formalized sonography training creates a need for clinical preceptors to teach basic anatomy/landmarks, transducer movements and criteria for dating and biometry views/measurements.⁷ Yet this poses a challenge for preceptors at high volume sites; thus, high-fidelity computer-based simulators could be part of the solution.

The results of this study highlighted the advantages of utilizing a high-fidelity CBSS as an educational tool to help enhance and expedite a resident’s sonographic knowledge and psychomotor skills. Although PGY2s had completed their first year of residency, PGY1s, who received training on a high-fidelity CBSS and were assessed within the first 7 months of starting their residency, achieved higher assessment scores. In addition, in comparing assessment data of PGY1 residents’ pre- and posttraining, it is apparent that the knowledge within the cohort had become more consistent, as the IQRs decreased between the two assessments. In fact, the IQRs of the posttraining assessments of PGY1s were generally smaller than that of the PGY2 cohort (Table 3). The use of a high-fidelity CBSS allowed for a uniform approach to sonography training among the cohort, thus decreasing gaps in knowledge and psychomotor skills within the cohort. The PGY1 cohort also demonstrated higher psychomotor skills than the PGY2 cohort when completing a 20-week OB sonogram on a high-fidelity CBSS. Multiple studies have identified improvement in the learner’s confidence and sonographic performance with the use of high-fidelity CBSS.^5,14,23,24 Burden et al conducted research with 18 OB sonography trainees and eight certified experts. They found that trainees were able to obtain clinically meaningful biometry measurements after an average of five educational sessions with a high-fidelity CBSS.¹⁴ In addition, Bernardi et al found that medical students who trained on a CAE Vimedix simulator obtained higher scores on biometry images captured on a real patient, compared with students who did not train on the simulator (12.3 ± 1.41 and 8.9 ± 2.64, respectively).²⁴ Another study by Monsky et al provided training on an UltraSim (high-fidelity CBSS) as part of the clinical rotations and prior to radiology residents taking call. This resulted in residents’ self-assessments revealing significant improvement of knowledge and scanning ability.⁵

The ability to document kinematic metrics during the assessment of a 20-week fetal case provided a unique opportunity to objectively score psychomotor skills. This study documented overall shorter sonographic exam times and less movements made by PGY1s (posttraining), compared with baseline data and PGY2s. In utilizing the high-fidelity CBSS during sonography training, the residents had the opportunity to develop a foundational understanding of transducer manipulation that allowed them to obtain images in a shorter period of time and require less unnecessary movement. The ability to expedite the development of necessary psychomotor skills of residents by using a high-fidelity CBSS would suggest measurable value in implementing this type of simulation training. A study by Montealegre-Gallegos et al applied motion analysis with TCP to assess cardiology fellows performing transesophageal echocardiography (TEE) and documented statistically significant reduction in the total time, path length, and probe accelerations (i.e., peak movement) in posttraining assessment.²² A study by Matyal et al also gathered kinematic metrics of novice sonography learners during the course of their training. They found an improvement in psychomotor skills based on the assessment metrics gathered.²⁰ The study also provided evidence of students’ transferable skills to the clinical environment. They found that participants who were subsequently assessed on live models were able to obtain required transthoracic echocardiography (TTE) views without assistance.²⁰

In addition to higher psychomotor skills compared with PGY2s, PGY1s also acquired higher quality diagnostic images/measurements of the fetus. Lower assessment scores for the PGY2s compared with PGY1s were identified in the subcategory of biometry views/measurements, which PGY2s performed during certain clinical rotations in their first year of residency. Although deliberate practice is an essential component to developing and improving psychomotor skills, a lack of direct observation and feedback on a learner’s performance could negatively impact the learner, potentially resulting in the learner unknowingly developing bad habits and gaining unwarranted confidence.²⁵ As seen in the data of the present study, PGY2s were capturing biometry images/measurements that did not meet the criteria for diagnostic quality images/measurements; yet, PGY2s noted they were confident in obtaining biometry views/measurements. If the behavior is not observed and immediately corrected through direct feedback and continued deliberate practice, then residents are left unaware of their mistakes. Alrahmani et al found that 34.8% of residents reported a lack of feedback on scanning performance.¹ As such, the present study used the learning model described using the Cycle of Practice and Feedback, which consists of three main components: observed performance, allowing for targeted feedback, which guides further practice, in which the cycle continues.²⁵ Based on PGY1s’ surveys, it appears that receiving direct and individualized feedback throughout the sessions helped enhance the residents’ learning. To this end, the value of constructive feedback should not be underrated and should be incorporated into sonography training. Moreover, final assessment scores revealed opportunities for further growth; some PGY1s could benefit from further deliberate practice and feedback on certain images/measurements, particularly first trimester gestational age scans.

There were limitations to utilizing a high-fidelity CBSS for sonography training. First, the fetus is static with only dynamic movement of the fetal heart. A static representation of the fetus is ideal for beginners as they develop foundational skills (i.e., linking movements made by the transducer to the effect it has on the ultrasound image); however, as learners gain the basic skills, they need to be challenged with fetal movement. Another limitation of using a high-fidelity CBSS is the inability to educate learners on knobology and various optimization settings that can be manipulated to acquire a higher quality image. In addition, the high-fidelity CBSS used in this study included only three fetal presentations (breech, cephalic, and transverse with head maternal right). Last, in using the “capture” feature on the CAE simulation equipment to save images, there were multiple instances where images did not save.

Study Limitations

This study had a small sample size, and participants all came from the same residency program. As such, findings of this study may not be representative of other residents and residency programs. In addition, only immediate follow-up assessments were performed posttraining of PGY1 residents. As such, retention of learned knowledge and skill were not tested. Furthermore, due to uncertainty of whether images captured by residents, on the simulator, would be archived for later review, the decision was made to evaluate residents with the assessment form while residents live-scanned on the simulator. In addition, the assessments were proctored by the sonographer/instructor. The author recognizes this method could introduce bias. To address this potential concern, when planning the study, the researchers worked to create a detailed breakdown of the assessment scores to provide an objective approach to scoring by clearly defining criteria required to achieve summative assessment points. Another important limitation to note is that high-fidelity CBSS cannot replace clinical experience and examining an actual patient, due to limitations of simulation equipment and skills lab environment.⁵

Conclusion

Findings of this study suggest that a high-fidelity CBSS can be used effectively in training OB/Gyn residents early in their residency. It also demonstrates that the implementation of formalized sonography training, using a high-fidelity CBSS, can be achieved and could potentially expedite and enhance the learning of novice learners. Simulation can provide residents with the opportunity to develop basic sonography skills outside of clinical, then refine and build upon those skills with preceptors. In addition, the value of incorporating a continuous cycle of deliberate practice and feedback, during training sessions, was valued by the learners as well as the research team. Moreover, the research team recognizes that further research should assess the transferability of the skills demonstrated during the assessment.

Footnotes

Acknowledgements

The authors thank CAE Healthcare Inc. for providing access to Vimedix simulators, and to Pamela Foy, MS, RDMS, FSDMS, and Phillip Samuels, MD, for their contribution to this research project.

Editor’s Note

The JDMS follows a scholarly, double-anonymized peer review process. The peer review of this manuscript was managed by members of the JDMS Editorial Board and excluded the Editor(s) listed as an author.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The primary author thanks the SDMS Foundation and Inteleos for their financial support.

ORCID iD

Sundus H. Mohammad

References

Alrahmani

Codsi

Borowski

: The current state of ultrasound training in obstetrics and gynecology residency programs. J Ultrasound Med. 2018;37(9):2201–2207. doi:10.1002/jum.14570.

American Institute of Ultrasound in Medicine: Training guidelines for physicians who evaluate and interpret diagnostic ultrasound examinations of the female pelvis. https://www.aium.org/officialStatements/58. Published 2015. Accessed March 24, 2020.

American Institute of Ultrasound in Medicine: Training guidelines for physicians who evaluate and interpret diagnostic obstetric ultrasound examinations. https://www.aium.org/officialStatements/59. Published 2015. Accessed March 24, 2020.

Accreditation Council for Graduate Medical Education: Obstetrics and gynecology case logs national data report, 2017. https://www.acgme.org/globalassets/PDFs/220_National_Report_Program_Version_2017-2018.pdf.

Monsky

Levine

Mehta

, et al: Using a sonographic simulator to assess residents before overnight call. AJR Am J Roentgenol. 2002;178(1):35–39. doi:10.2214/ajr.178.1.1780035.

Osborne

Parange

Thoirs

: The effectiveness of the use of high fidelity simulators in obstetric ultrasound training: a systematic review. Australas J Ultrasound Med. 2015;18(3):107–111. doi:10.1002/j.2205-0140.2015.tb00209.x.

Mohammad

Sommerich

Butwin

Evans

: Assessment of obstetrics and gynecology residents’ knowledge of basic obstetric sonography and psychomotor evaluation with motion analysis using a computer-based simulator. J Diagnostic Med Sonogr. 2020;36(4):312–321. doi:10.1177/8756479320919069.

Qurash

Yaacob

Azuan

Khaleel

Zakaria

: Special ultrasound phantom for interventional training: construction, advantages, and application. J Med Ultrasound. 2018;26(4):210–214. doi:10.4103/JMU.JMU_40_18.

Lewiss

Hoffmann

Beaulieu

Phelan

: Point-of-care ultrasound education. J Ultrasound Med. 2014;33(1):27–32. doi:10.7863/ultra.33.1.27.

10.

Hughes

Anderson

Patterson

O’Prey

: Introducing an obstetric emergency training strategy into a simulated environment. Br J Midwifery. 2014;22(3):201–207. doi:10.12968/bjom.2014.22.3.201.

11.

Young

Williamson

Egan

: Students’ reflections on the relationships between safe learning environments, learning challenge and positive experiences of learning in a simulated GP clinic. Adv Health Sci Educ Theory Pract. 2016;21(1):63–77. doi:10.1007/s10459-015-9611-3.

12.

Doyle

: Ultrasound training in obstetrics and gynecology residency. Obstet Gynecol. 2018;132:45S. doi:10.1097/01.aog.0000546631.73422.e0.

13.

Mohammad

Evans

Bahner

Sommerich

: An interprofessional approach to targeted sonography training using computer-based simulation in medical students’ clinical training: a case series. J Diagnostic Med Sonogr. 2020;36(2):153–161. doi:10.1177/8756479319897891.

14.

Burden

Preshaw

White

Draycott

Grant

Fox

: Usability of virtual-reality simulation training in obstetric ultrasonography: a prospective cohort study. Ultrasound Obstet Gynecol. 2013;42(2):213–217. doi:10.1002/uog.12394.

15.

Gibbs

: The role of ultrasound simulators in education: an investigation into sonography student experiences and clinical mentor perceptions. Ultrasound. 2015;23(4):204–211. doi:10.1177/1742271X15604665.

16.

Madsen

Konge

Nørgaard

, et al: Assessment of performance measures and learning curves for use of a virtual-reality ultrasound simulator in transvaginal ultrasound examination. Ultrasound Obstet Gynecol. 2014;44(6):693–699. doi:10.1002/uog.13400.

17.

Kusunose

Yamada

Suzukawa

, et al: CORRESPONDENCE effects of transthoracic echocardiographic simulator training on performance and satisfaction in medical students. J Am Soc Echocardiogr. 2016;29:375–377. doi:10.1016/j.echo.2015.12.002.

18.

Rosen

Windrim

Lee

Gotha

Perelman

Ronzoni

: Simulator based obstetric ultrasound training: a prospective, randomized single-blinded study. J Obstet Gynaecol Can. 2017;39(3):166–173. doi:10.1016/j.jogc.2016.10.009.

19.

Abuhamad

Minton

Benson

, et al: Obstetric and gynecologic ultrasound curriculum and competency assessment in residency training programs: consensus report. J Ultrasound Med. 2018;37(1):19–50. doi:10.1002/jum.14519.

20.

Matyal

Mitchell

Hess

, et al: Simulator-based transesophageal echocardiographic training with motion analysis: a curriculum-based approach. Anesthesiology. 2014;121(2):389–399. doi:10.1097/ALN.0000000000000234.

21.

Shakil

Mahmood

Matyal

: Simulation in echocardiography: an ever-expanding frontier. J Cardiothorac Vasc Anesth. 2012;26(3):476–485. doi:10.1053/j.jvca.2012.01.019.

22.

Montealegre-Gallegos

Mahmood

Kim

, et al: Imaging skills for transthoracic echocardiography in cardiology fellows: the value of motion metrics. Ann Card Anaesth. 2016;19(2):245–250. doi:10.4103/0971-9784.179595.

23.

Maul

Scharf

Baier

, et al: Ultrasound simulators: experience with the SonoTrainer and comparative review of other training systems. Ultrasound Obstet Gynecol. 2004;24(5):581–585. doi:10.1002/uog.1119.

24.

Bernardi

Benzina

Hajal

Chalouhi

Salomon

Ville

: OP20.08: obstetrical ultrasound simulator as a tool for improving teaching strategies for beginners. Ultrasound Obstet Gynecol. 2013;42(s1):107. doi:10.1002/uog.12897.

25.

Ambrose

Bridges

Dipietro

Lovett

Norman

Mayer

: How Learning Works: Seven Research-Based Principles for Smart Teaching, 2010. https://firstliteracy.org/wp-content/uploads/2015/07/How-Learning-Works.pdf.