Notes From the Field

Video-recorded performances are frequently used to assess trainee performances. Unlike direct observations where the assessor must be present during the performance of the procedure, ratings by video can be done at a time and location convenient to the assessors and allow them to rewind and fast-forward segments of the performances as necessary. Further, depending on the camera angles, video-recorded performances allow blinded assessments by not including identifying features of the trainees, thereby eliminating examiner familiarity as a source of bias in the rating of performances. However, ratings by video involve additional equipment and audiovisual storage needs, which add to the cost and complexity of the assessment process.

Despite the frequent use of video recordings for assessment purposes, with few notable exceptions (Kneebone et al., 2006; Scott et al., 2000), technical issues are seldom discussed in the literature. We report the results of our study comparing 18 performances of central venous catheterization on simulators performed under direct observations with assessments on video-recorded performances, describe the discrepancies, and in so doing, highlight some technical issues in video recordings that are relevant to the assessment of procedural skills.

For both modalities, two faculty members assessed for procedural competence using a 9-item global rating scale and a 10-item checklist (Cronbach’s α .79 and .67, respectively; Ma et al., 2012) on each of the 18 performances. Video recording was done in a static fashion with participant blinding (recording only the participant’s gloved hands and gowned arms, the simulator and the procedural kits). The camera was placed on a tripod, recording from the mannequin’s right-hand side irrespective of the participant’s handedness. Audio information was captured, as candidates were asked to talk through the procedure. Two methods were chosen to minimize assessor recall: First, video-recorded performances were assessed a minimum of 6 months after the directly observed performances, and second, each assessor evaluated 34 video-recorded performances in random order, rather than only the 18 performances in this study.

Overall, the sound quality was deemed adequate in all video recordings and the camera view did not capture participant identifying features. However, recording issues led to failure to capture the beginning of the video in two (11%) assessments and the end in one (6%) assessment. There were difficulties capturing the entire wire-handling process in 13 (72%) of the videos, with the wire out of view for a median of 15.3 s for all videos (interquartile range, IQR 12.2–19.2 s). Inability to visualize the entire wire was felt to be consequential for the assessment of wire handling in only five of the videos (38%) because hand positioning was easily ascertained in the remaining eight (62%). Because of the close proximity of the drape handling to the participants, handling of the drapes was frequently out of view (n = 17; 94%). The drape was out of view for a median of 7.7 s (IQR 4.5–10.0 s). Nine of these 17 videos (53%) were felt to have potential impact on scoring. For example, impact on scoring was deemed to be of potential consequence to the assessment if the assessors felt that the portion of drape handling that was out of view may have revealed breaches of sterility that were otherwise not observed in the video. Impact on scoring was deemed unlikely to be of consequence if the assessors were able to ascertain that the portion of the drape that was out of view was unlikely to have made contact with nonsterile equipment or surfaces.

Analysis of 26 scoring discrepancies between direct observation and ratings by video identified three discrepancies (12%) as being due to inattention during video review, while one (4%) discrepancy was due to inattention during direct observation. The remaining discrepancies were attributable to video recording technical issues.

Overall, there were no differences in mean summary scores in global scores (out of five) between video-recorded assessment and direct observation (2.7 ± 0.9 vs. 2.8 ± 0.8, p = .40, respectively), nor were there significant differences in the mean checklist scores (91.6 ± 11.5% for video-recorded assessment vs. 90.6 ± 11.7% for direct observation, p = .28). There were no significant differences in the pass/fail decisions (McNemar’s S = 2.78; p = .10). Interrater reliability for both assessment modalities was high (intraclass correlation range .71–.88).

In summary, our results suggest that overall scores and pass/fail decisions did not significantly differ between direct observation versus ratings by video, and interrater reliability for both modalities were acceptable and comparable. However, adequacy of video recording may significantly impact on individual items of assessment. Specifically, if blinding is required, portions of the procedure close to the participants may be difficult to capture. To overcome these difficulties, we recommend capturing hand positioning with the use of dynamic recording in the assessment of procedural skills.

Overall, the advantages of assessment by video recordings include the ability for blinding participant identity, convenient rating schedules by the assessor, but at a cost of potentially missing important segments for rating and the potential for video review inattention. The advantages of direct observation include the ability to view the entire procedure, but at a cost of scheduling difficulties, potential for bias due to examiner familiarity with the participants, and the inability to rewind to review portions of the procedures missed by the assessor. The use of dynamic recording may minimize the disadvantages associated with ratings by video.