Interdisciplinary video review: Assessing milestones and providing feedback

Background

Residency faculty members train and evaluate residents in the six core competencies that the accrediting organization (i.e., Accreditation Council for Graduate Medical Education (ACGME)) has identified. ¹ These core competencies are medical knowledge, patient care, systems-based practice, interpersonal and communication skills, practice-based learning and improvement, and professionalism (see Table 1). ² Although most of the core competencies can be assessed in various ways, there are some competencies and subcompetencies that are best assessed through directly observing a resident’s skills. When nonverbal and communication skills are observed either in real-time or through recorded encounters, faculty can provide assessment of the actual skills residents displayed.

OPEN IN VIEWER

Table 1.

Core competencies and video review of residents.

Core competency	Family medicine description	Recommend video review2
Patient care	Comprehensive and coordinated care provided using a biopsychosocial framework	Yes
Medical knowledge	Broad clinical reasoning and adequate knowledge to care for diverse population	Yes
Systems-based practice	High-quality access to care, with physician advocacy for improved care	No
Practice-based learning and Improvement	Evaluate patient care, assess scientific evidence, and continuously improve patient care	No
Interpersonal and communication Skills	Demonstration of skills that exhibit trust and collaboration with patients, families, other healthcare providers, and the public	Yes
Professionalism	Skills include competence, integrity, identifying conflicts of interest, and taking responsibility for learning and ethical practice	Yes

When faculty members observe residents’ interactions with patients during clinical care, the focus can often be on taking care of the needs of the patient; it may be difficult to find time to discuss any specific feedback to residents regarding patient care, medical knowledge, or communication skills. ³ An approach to providing residents with more detailed feedback through reviewing their recorded patient encounters is the focus of the article. For context, our residency is in an urban, Midwestern city. The Family Medicine residency has been accredited since the 2000–2001 academic year. Over the last 10 years, we have had an average of 13 residents per resident cohort. During that time, more than half of our graduates practice within the state the year after graduating.

Curriculum development

Per the ACGME requirements for Family Medicine (section V.A.2¹), each program must include a formative evaluation. This assessment is of residents’ procedural skills, medical knowledge, professionalism, and patient care. Part of this assessment is defined by direct observation, as well as evaluations from peers, clinical staff, and patients. Each program is required to develop their own process and requirements for direct observation/video review. Similarly, for Internal Medicine and Pediatrics, there is no mention of how observation should be performed. ACGME does require that Psychiatry residents have the ability to review themselves on film to assess techniques and strategies for evaluating patients (section II.D.5¹).

Direct observation has played an integral role throughout training for many residencies, including most family medicine residencies.^4,5 At the Indiana University-Methodist Family Medicine Residency, residents’ outpatient encounters are routinely recorded to be reviewed by faculty members. Faculty members do not receive any specific training on recording review. Depending on the clinic schedule, some faculty preceptors will view resident encounters live in clinic. This allows the faculty preceptor to gather relevant medical history about the patient and observe the resident’s interactions with the patient. Since this format is not done on a regular basis and is used as an efficient, nuanced precepting approach, it is not the dominant way that our faculty use direct observation to evaluate residents. The indirect viewing that takes place without a secondary provider in the room creates a more realistic encounter for evaluation by both staff and the trainee themselves.

A few different iterations of observing residents' recorded patient encounters are described in Table 2. Special attention was paid to choosing recordings that would allow for feedback in multiple areas as outlined by the ACGME milestones (see Table 3 for sample evaluation form). Assigned faculty members completed the direct observation evaluation form online. These evaluations were often not available to residents until after the resident had the opportunity to review their own recording. An advantage of the current format is that it does not interfere with residents’ requirements for completing 1650 ambulatory patient encounters. The recorded encounters occur during regular clinic encounters at the main residency clinic. Therefore, the patient encounters that residents have at other clinical sites (e.g., off-site obstetrics clinic, transgender clinic, or student-run free clinic) are not options for recording visits to review. This limits the range of visits for faculty evaluations. Although this has not always been the case, currently residents do not observe their peers’ recorded video encounters in a context in which formal evaluations are completed. Multiple faculty members view and complete evaluations, which are then used in biennial Clinical Competency Committee (CCC) reviews of residents.

OPEN IN VIEWER

Table 2.

Video review/direct observation timeline.

Start date	Format	Pros and Cons
Prior to 2015	Behavioral Science faculty used PCOF in the exam room with resident and patient.	All responsibility and feedback only from Behavioral Science faculty. Video equipment no longer functioning, so must do in-room observation. Can see nuances of interactions.
2015/2016 Academic Year	Recording equipment installed in 5 exam rooms and 1 Counseling Room; 4 rooms can be recorded simultaneously. Behavioral Science faculty member in charge of scheduling residents to be recorded.	Resident schedules often changed; only up to one faculty to review videos and have groups of residents review own/peers in small groups.
2016/2017 Academic Year	Physician faculty member in charge of designating residents to be recorded. These encounters would be reviewed by Behavioral Science and 2 to 3 physician faculty. Evaluation form filled out by all faculty. Later, groups of residents who had evaluations would watch their videos in groups and review faculty feedback with one faculty member.	All patients provided assent in Consent for Treatment.Physician faculty members started providing feedback for resident–patient encounters. Some residents reported that feedback sessions were anxiety-provoking. Coordinating multiple faculty/resident schedules. Limited recorded encounters were observed due to scheduling/reviewing/feedback challenges. Equipment in rooms is unobtrusive, mounted on the ceiling. Recordings are stored on secure, cloud-based system accessed only by password-protected logon.
2017/2018 Academic Year	One physician faculty is responsible for identifying which residents should be recorded each week. A clinical staff assigns all residents/providers to exam rooms. Residents are assigned to recording rooms on a rotating basis, with at least four recordings per resident per academic year. If a concern about a resident’s clinical performance is identified, more encounters can be recorded for the resident’s advisor to review. Physician and Behavioral Science faculty members review each resident at least twice per year in small groups, completing evaluation forms. Physician faculty in charge of assigning which residents should be recorded each week also completes real-time observations when precepting to expedite review of patient cases.	All patients must give verbal assent for recording; signed assent included in health system Consent for Treatment. Resident peer review only occurs during Behavioral Science with focus on interpersonal/behavior change skills. Timing of video review and batched evaluations can delay feedback to residents. Coordinating schedules for faculty can be challenging.

PCOF: Patient Centered Observation Form. See https://depts.washington.edu/fammed/wp-content/uploads/2016/11/Clinician-PCOF-2016-version.pdf.

OPEN IN VIEWER

Table 3.

IU-Methodist Family Medicine Sample Direct Observation Resident Evaluation Form.

Identify the resident’s performance level in the following areas.
	Subcompetency	Level 1 Milestones (beginning Year 1)	Level 3 Milestones (beginning of Year 3)
1	Elicits and documents an accurate and appropriate history.	Recognizes need for appropriate history.	All relevant questions to CC; 4–8 elements of HPI but lacks efficiency.
2	Performs and documents accurate and appropriate physical exams	Recognizes the need for an appropriate physical exam.	PE complete, accurate and relevant to CC; lacks efficiency. Documentation complete and identification of ddx.
3	Identifies and address psychosocial problems	Identifies potential psychosocial issues, but minimizes or does not further address them.	Appropriately addresses psychosocial issues; needs to improve efficiency or integration with biomedical care.
4	Collaboratively sets agenda with patient	Focuses on physician’s agenda.	Asks for initial concern and, later in visit, for other concerns; may not follow up on agenda when closing visit.
5	Uses questions appropriately	Uses primarily closed-ended questions.	Uses open-ended and follow-up questions appropriately.
6	Confirms understanding and agreement with the diagnosis and plan	Plan provided without acknowledgement of patient understanding.	Checks for general understanding and agreement during visit; utilizes teach-back.

There are 10 levels of assessment that can be identified. This includes “does not meet Level 1,” “Not applicable,” and an aspirational “Level 5,” which performance would be expected of a physician practicing for one or more years post-graduation. Levels 1, 2, 3 correspond to each year of residency and have two assessment points within each year—roughly corresponding to the beginning and end of the training year. CC = Chief Complaint. HPI = History of Present Illness. PE = Physical Exam.

Curriculum goals and objectives

One of the responsibilities of the video-review-designated faculty member is to identify resident–patient encounters for a group of three other faculty to observe together during scheduled times. The designated faculty member in charge initially reviews recorded encounters, identifies encounters for the faculty group to review, and tracks that each resident’s encounters are reviewed at least twice an academic year. When the group of faculty members review the recorded encounters, each faculty member completes an evaluation in our online resident evaluation system. When completed, these evaluations are available for each resident to review. The faculty members’ goal has been to be clear with residents what skills and competencies we are evaluating during our review of direct observation recordings, as well as how these evaluations will impact their overall training and progression towards promotion and graduation. ⁶ These evaluations are part of the collated data of each resident’s progress for that six-month period for the members of the CCC to review. These ACGME required reviews are completed with residents’ input (educational development plan) and confidentially discussed with each resident twice an academic year.

Resident perspectives

Video observation can be a valuable tool ⁷ as well as a source of frustration for residents. The advantages include viewing one’s own encounter in the third person, the viewing of colleague encounters, and receiving direct feedback from faculty members. On the other hand, the particular recorded patient encounters may not provide a true representation of a real-world patient encounter as the resident and patient are aware that they are being recorded. At times, some residents may feel anxious about their resident colleagues and faculty members observing their patient encounters and providing feedback.

Video review allows a resident physician to view a patient encounter from a third-person perspective. For example, the resident can assess his or her body language (e.g., Did I really just stare at the computer screen the entire time?). The resident may also be unaware that he or she missed certain interpersonal or non-verbal cues.

Residents usually interact with a patient on their own. The resident then discusses the case with faculty members, who may provide a critique of the medical decision making and treatment plan. The patient–resident interaction is rarely if ever evaluated. Inpatient rounding does provide a limited opportunity to observe peers and faculty members interacting with patients. However, these interactions are generally brief. Therefore, video review provides a more robust evaluation of resident–patient interaction. This evaluation is provided in an outpatient setting where the majority of residents will work after their residency program.

Direct observations may be improved from a residents’ perspective by having more frequent direct observations throughout the residency program at planned intervals. Developing a standardized direct observation evaluation form that can be used over time to determine successes and failures from prior direct observations would help residents identify areas of growth or continued weakness (see Table 3 for sample form). Furthermore, determining a way to customize the direct observation program based off of resident comfort level would alleviate the stress that some residents may experience. ⁸ For example, if residents were able to opt out or select who they would like to evaluate their encounter.

Although residents at times may feel uncomfortable or nervous due to visits under direct observation, overall they are considered to be beneficial for learning purposes. A review of outpatient pediatric residency encounters revealed 63% of residents favor direct observation as a teaching resource. ⁹ In addition, 94% of residents concluded that direct or video observation aided them in honing their skills of history taking and physical examination. ⁹

Future directions

With the variation of residencies’ resources, one approach would not work for all programs. Through the limited review of direct observation examples discussed here, some suggestions will be discussed that have assisted in consistently evaluating residents’ patient encounters and providing feedback to them. First, determine a method that works in terms of faculty responsibility and documentation. Direct observation of resident-patient recorded encounters has occurred much more frequently since faculty members are scheduled for 2-h sessions to meet together and view videos that one designated faculty member has identified. Of the approximately six faculty development half-day sessions the residency has, one of these sessions each year has been focused on resident evaluation and direct/video observation to ensure consistency across faculty members. This designated faculty member has time blocked to briefly review recordings. The minimum number of video reviews for each resident is two per academic year (one in each half of the academic year). A second aspect that has helped was the development of a focused direct observation evaluation form that is tied to the milestones. The current form we use is the third version we have used. In addition, this form feeds directly into the overall resident evaluation system. If any deficiencies are identified, the resident and advisor have access to the evaluation, and these concerns are addressed as other patient care or medical knowledge concerns would be documented and addressed. Therefore, the direct observation evaluations, including any written, qualitative feedback faculty members include, are used to evaluate residents’ progress towards promotion and graduation. Other changes that have occurred over time is that an administrative support person has the responsibility of assisting the main faculty member with sending requests to the designated clinic staff to schedule residents in video rooms. This person is also responsible for editing the videos so that they only include the patient–physician interaction, as well as saving the files with the resident’s name and date. The clinic staff designee schedules all providers in rooms for each week, while also scheduling requested residents in video rooms. Developing a system of recording resident–patient encounters, a schedule of reviewing and evaluating the skills residents demonstrate during those encounters, as well as opportunities for real-time observation and feedback are essential for residents to develop competence as physicians.