Impact of an Online Question Bank on Resident In-Training Exam Performance

Introduction

Obstetrics and gynecology (OBGYN) residents complete an annual in-training examination (ITE) through a nationwide exam issued by the Council on Resident Education in Obstetrics and Gynecology (CREOG). The ITE provides a measure to evaluate cognitive knowledge and assess a program's didactic curriculum. A strong performance on ITEs has been demonstrated to predict a passing score on the American Board of Obstetrics and Gynecology (ABOG) Qualifying Exam which in turn is used by ACGME to help inform accreditation decisions.^1,2 Given this, OBGYN residency training programs can use these exams to identify gaps in resident education to ensure resident success on qualifying board exams.

Currently, there is no standardized OBGYN residency education curriculum and no consensus in the literature on tools to improve performance on ITE. ³ The CREOG Educational Objectives provide a framework for resident instruction, but didactic curriculum vary between residency programs. ⁴ Retrieval methods, including the use of a question bank, have been demonstrated to be effective strategies for learning but the optimal implementation during residency is unknown.^5–8 This study aimed to improve resident performance on the CREOG ITE through the implementation of a question bank and mock ITE.

We hypothesize that universal access to a question bank and mock ITE will improve resident performance on the CREOG ITE. Furthermore, we hypothesize that the extent of question bank usage as measured by number of questions completed will positively correlate with the magnitude of ITE improvement.

Methods

We performed a pre–post intervention study of the CREOG ITE score reports from 2017 to 2019 for all OBGYN residents at Indiana University. The Indiana University Institutional Review Board (IRB) reviewed the protocol and determined the study to be IRB-exempt with waived informed consent as the study fell into research conducted in common educational settings involving normal educational practices (IRB# 1903206729). CREOG ITE examinations are given during January of each year. Beginning in July 2018, residents were given unlimited access to the general TrueLearn question bank. Didactic content and reviews were unchanged during the study and dedicated time was not provided for use of the question bank. Residents had 3 h of dedicated didactic time per week with traditional lectures provided by faculty and fellows. The schedule adheres to topics provided in the CREOG Educational Objectives.

The TrueLearn question bank is a commercial product with over 1,200 unique questions available for self-study and access both online and through a mobile device. ⁹ Administrators were able to track the use of the question bank, but use was not mandatory. The total number of practice questions performed up to the ITE test date was collected. As a part of the implementation, a simulated ITE through the question bank provider was administered two months prior to the CREOG ITE. This simulated ITE utilized 150 novel questions and was provided in a timed manner during protected didactic time. The questions were selected by TrueLearn and covered a blueprint similar to the CREOG ITE. Performance on the simulated ITE was not measured, but all residents completed the mock ITE.

CREOG ITE performance was evaluated as individual improvements, assessed by the percent of correct questions answered in the 2018 ITE compared to the 2019 ITE. This year-to-year change was calculated for each individual resident and mean changes were reported for each post-graduate year (PGY), termed postintervention. The preintervention cohort consisted of the average improvement from one PGY year to the next (PGY 1 to 2, PGY 2 to 3, and PGY 3 to 4) from 2017 to 2018. The postintervention cohort consisted of the average improvement from one PGY year to the next from 2018 to 2019.

Statistical Analysis

A dependent t-test was conducted to compare the average annual improvement in CREOG ITE scores before and after question bank implementation. Correlation between question bank usage (defined by the number of questions performed) and improvement on the 2019 ITE (defined by percent increase) were calculated for each individual using Pearson correlation testing and a linear regression was used to determine their association. We used SPSS Statistics 26 ¹⁰ to perform all statistical analyses.

This study was submitted to Indiana University IRB and found to be exempt. All scores on the CREOG ITE and the question bank were deidentified.

Results

The Indiana University OBGYN Residency Program has 10 residents per year. Over the course of the study, there were a total of 51 OBGYN residents (one class had a temporary increase to 11). Those who did not have an ITE score report prior to the implementation of the question bank and those who did not sit for all eligible exams were excluded from this study. A total of 13 residents were excluded, leaving 38 OBGYN residents in the analysis. The overall mean ITE for all PGY levels was 61.4% (SD 6.9) prior to the implementation of the TrueLearn question bank and the overall mean ITE was 64.5% (SD 4.9, 3.1% increase, P = .02) postimplementation.

Prior to the implementation of the question bank, the mean ITE improvement was 0.6% (SD = 3.1) for the PGY1 to 2 cohort and 1.0% (SD = 3.3) for the PGY 2 to 3 cohort. The residents in the PGY3 to PGY4 cohort demonstrated a mean decrease in ITE performance of 1.6% (SD = 2.6) prior to access to this question bank.

Following the implementation of the question bank, there was a statistically significant improvement in the CREOG ITE for all levels of residency training. The PGY 1 to 2 showed a mean improvement of 6.6% (SD = 3.1), with an absolute increase of 6.0% with the question bank, P = .018, 95% CI [2.17–9.72] (Figure 1). Residents in the PGY 2 to 3 cohort showed a postintervention improvement of 9.0% (SD = 4.7), demonstrating an absolute increase of 8.0% with the question bank, P < .01, 95% CI [4.0–11.96] (Figure 2). The PGY 3 to 4 cohort demonstrated a mean improvement of 7.2% (SD = 3.6), with an absolute increase of 8.9%, P < .01, 95% CI [5.56–12.14] (Figure 3).

OPEN IN VIEWER

Figure 1.

Mean improvement on ITE performance from PGY 1 to 2 before (0.60 [SD 5.0]) and after (6.6 [SD 3.1]) implementing TrueLearn (P < .01).

OPEN IN VIEWER

Figure 2.

Mean improvement on ITE performance from PGY 2 to 3 before (1.0 [SD 3.3]) and after (9.0 [SD 4.7]) implementing TrueLearn. (P < .01).

OPEN IN VIEWER

Figure 3.

Mean improvement on ITE performance from PGY 3 to 4 before (−1.6 [SD 2.6]) and after (7.2 [SD 3.6]) implementing TrueLearn (P < .01).

Pearson correlation coefficients were calculated to assess the relationship between the extent of question bank usage, as measured by the number of practice questions completed, and percentage improvement on the ITE. Overall, there was a positive moderate correlation between the number of questions accessed in the question bank and the percentage point improvement on the CREOG ITE; r = .361, P = .046, though this was not significant for any individual year (Table 1). A scatter plot demonstrates a linear correlation between the number of questions answered in the question bank and the CREOG ITE result (Figure 4).

OPEN IN VIEWER

Figure 4.

Correlation between individual TrueLearn usage and their percentage point improvement on CREOG ITE; r = .361, P = .046.

OPEN IN VIEWER

Table 1.

Correlation of TrueLearn Usage and Improvement on CREOG ITE.

	Mean post-TrueLearn improvement	Average number of questions completed	Pearson correlation coefficient	P-value
PGY 3-4	7.2	397	0.29	.41
PGY 2-3	9.0	965	0.50	.14
PGY 1-2	6.6	503	−.019	.58
Total	7.6	618	0.361	.046 a

CREOG: Council on Resident Education in Obstetrics and Gynecology; ITE: in-training exam; PGY: post-graduate year.

^a Significance P < .05.

Discussion

Our study demonstrates a statistically significant increase in year-to-year improvement on CREOG ITE after the implementation of a question bank and mock ITE. When assessing independent question bank usage, our data demonstrates a positive correlation with ITE performance. The focus of our study was to discern if universal access to an online question bank and mock ITE proved beneficial to the global performance of the CREOG ITE at our institution. These findings support our hypotheses.

The positive effects of practice question use on scores on the United States Medical Licensing Examination (USMLE) Step 1 examination have previously been demonstrated.^11–13 Similarly, this benefit has been shown in performance on the American Board of Surgery In-Training Examination (ABSITE), the CREOG ITE equivalent for general surgery residents.^14,15 Imran et al. have identified that the use of the TrueLearn question bank specifically is associated with a greater percentage correct on ABSITE. ¹⁶

Our findings differ from a separate evaluation of the same question bank. ¹⁷ Both studies found a positive correlation between question bank usage and CREOG scores, however, the authors did not find a statistically significant increase in CREOG performance with question bank implementation. Though our study also provided unlimited access to the question bank, participants in our study differed as they had access to a mock ITE. The opportunity to simulate exam conditions may have helped to reduce anxiety or improve time management for some test takers, leading to the difference in scores observed in our study.

Additionally, we chose to report mean improvement from one PGY year to the next, with the control group representing the same PGY transition year, as resident knowledge is naturally expected to change throughout this time. Using the average improvement in percent correct over a year rather than absolute ITE scores allows for individual feedback for residents and mitigates potential outliers for individual scores that might skew the mean. Scores on the CREOG are relatively stable over time. In 2019, the average CREOG percent correct was 55% for PGY1, 60% for PGY2, 64% for PGY3, and 66% for PGY4. This represents a national average increase from PGY 1 to 2 of 5%, PGY 2 to 3 of 4% and PGY 3 to 4 of 2%. Our results in the post-TrueLearn implementation group of 6.6%, 9.0%, and 7.2% exceeded these thresholds.

Our study has several limitations. First, our sample size was small with no a priori size calculation and limited to a single institution. Although the resident class at Indiana University is large at 10 persons per year, it cannot be concluded whether the question bank and mock ITE filled a gap in resident education specific to the Indiana University residency program, or if this service would be beneficial across all OBGYN programs. Our R square of 0.13 suggests that only 13% of the variability in percent improvement on the CREOG ITE was explained by TrueLearn usage therefore other factors could explain the improvement aside from the question bank access. Additionally, it is unknown from this study whether any peer-reviewed question bank would yield similar improvements, or if this benefit is limited to TrueLearn only.

One potential area of concern for many residency programs could be the cost of providing universal access to a question bank for its residents. A one-year subscription to the OBGYN ITE TrueLearn question bank is $320 per resident. ⁹ This may be prohibitive for many programs, particularly those with larger resident cohorts. A potentially less cost-prohibitive option may be to use shorter subscription plans, as TrueLearn offers one-, three- and six-month options. Other question banks exist with different payment structures, such as Board Vitals or Rosh Review, with the latter offering a mock ITE in addition to question bank access.

Historically, standardized multiple choice exams, particularly the USMLE Step 1, have sometimes been used broadly as a screening metric by which OBGYN residencies offer interviews. ¹⁸ One reason for this is that those who score highly on STEP 1 tend to perform well on the CREOG ITE and ABOG qualifying exams.^19,20 However, with the implementation of Pass/Fail grading on the USMLE Step 1 exam, residency programs will no longer have this metric during the recruitment process. This study suggests that with additional resources, including a question bank and simulated ITE, resident performance can be positively affected by retrieval practice. Future studies will continue to refine the optimal implementation of these resources.

Conclusions

Our study demonstrated that access to a question bank and mock ITE significantly improved the CREOG ITE performance of OBGYN residents at Indiana University.