Notes from the Field

Introduction

The effectiveness of simulation-based learning is well established and increasingly utilized for learner assessments (Cass, Crofts, & Draycott, 2011). The Test of Integrated Professional Skills (TIPS) is an observed structured clinical examination (OSCE)-simulation hybrid exam using standardized patient interactions and skill performance to assess a range of skills in order to generate personalized learning plans. Compared to standard evaluations, TIPS can better distinguish between high- and low-performing residents (Winkel et al., 2014). Although evidence suggests the high value of simulation-based learning, little is known about the learners’ perception of this type of assessment.

Method

Using focus groups interviews following the administration of the TIPS exam, reflective interviewing techniques explored residents’ perceptions of the exam. Each focus group took place directly after the exam was administered. A convenience sample of 11 obstetrics and gynecology (OBGYN) residents comprised two focus groups, one with five junior residents and one with six senior residents. The sample was selected from a group of 29 residents (14 senior residents and 15 junior residents). Focus groups, led by a trained facilitator, were recorded and later transcribed. Institutional Review Board (IRB) approval was obtained from New York University School of Medicine.

Using NVivo qualitative coding software, Version 10, two authors (Winkel and Niles), used a three-step coding process with a focus on thematic analysis (Campbell, Quincy, Osserman, & Pedersen, 2013). Transcripts were first deconstructed and independently coded. Codes were then merged to create a single codebook, generating 13 distinct codes with high intercoder agreement. Final coding scheme was reapplied to transcripts to ensure that relevant themes were captured during analysis.

Findings

Several valuable themes came from this analysis. Challenges of the simulated testing environment emerged as a dominant theme, with residents reporting anxiety when objectives or instructions were unclear. One resident stated, “It throws off the scenario when you’re trying to figure out the technical parts…it limits your ability to do anything.” Realism was also crucial for learners in complex simulations. This contrasts with evidence that there is a minimal relationship between simulation fidelity and transfer of learning observed in a teaching setting, where low-fidelity simulations are effective (Norman, Dore, & Grierson, 2012).

Distinctions between skill types was evident in residents comments about interpersonal versus cognitive versus technical skills. They valued decision making and technical skills above interpersonal ability. As one resident stated, “going through the technical steps of things…Those are things that I think are important.” Another said, “I’m not saying that interpersonal skills are not important. But…[feedback about technical skills is] what I’m more interested in.” The senior residents reported tremendous anxiety that their performance would not reflect the mastery they felt they had attained, and many worried that it was not clear how to achieve mastery on the TIPS.

The role of assessment in learning emerged as a theme in resident comments suggesting that the TIPS exam was more beneficial for the teacher than the learner. As one participant reported, “It’s to make it easy for them to grade us.” They made a clear distinction between a “learning opportunity” and “this stressful situation where we feel like we’re being graded.” In fact, TIPS results serve to generate personalized learning plans, not for summative evaluation.

The value of feedback was a theme that generated rich discussions in both groups and perceived to be extremely valuable to learning. The TIPS environment was seen as an effective way for junior residents to receive supportive feedback. One senior resident stated, “they have a chance to practice and get some feedback in a way that is not threatening.” Although feedback in the clinical environment was felt to be more genuine than the staged encounter of the TIPS, it was acknowledged that feedback is limited in clinical practice due to competing demands and was valuable in any capacity.

Discussion

Analysis of these reactions to TIPS illustrates how trainees engage with performance-based assessment. The valuation of cognitive and technical skills above interpersonal skills reflects the traditional training environment of OBGYN and underscores the importance of including interpersonal skills assessment into learning paradigms. The participants’ concerns about realism and test performance may reflect residents’ priorities as learners and encourages teachers to orient them to broader values of simulation. Residents must be encouraged to see the opportunity to identify weaknesses as a critical to enhancing learning (Shepard, 2000).

Three priorities emerge from our findings. First, include clear direction and instructions for students in design and implementation of simulation-based assessments. Second, scenarios must be designed to maintain a high level of realism, especially with complex case management. Finally, engage the learner in a better understanding of how the simulation experience may benefit their growth and development as a novice clinician. This study has several limitations, most prominently the small sample size due to the constraints of exam. By doing focus groups after the exam, cognitive set bias may have been introduced. The emergence of common themes between the two groups despite the lack of other data does, however, suggests that the themes discussed here are representative.