A level playing field? Evaluation of the virtual Objective Structured Clinical Examination in Psychiatry and Addiction Medicine: A mixed methods study

Abstract

Objective

The Objective Structured Clinical Examination (OSCE) has been used for clinical assessment of a broad range of medical student competencies in Psychiatry and Addiction Medicine. However, there has been little research into online assessments. We investigated the virtual OSCE (v-OSCE) from the user perspective to better understand its acceptability, usefulness, benefits, challenges and potential improvements.

Methods

At the conclusion of the v-OSCE, all participants (medical students, examiners and simulated patients) were invited to participate in a brief online survey, based on the Technology and Acceptance Model. Freeform qualitative feedback was also obtained to explore participants’ experiences and attitudes.

Results

Participants reported the v-OSCE was acceptable, efficient, convenient and easy to use. It was perceived as useful for demonstrating students’ interviewing skills and interacting with the simulated patient. Benefits included greater convenience, reduced stress and travel time. Challenges were similar to those experienced in ‘real world’ telepsychiatry, primarily related to assessment of non-verbal cues and emotional prosody.

Conclusions

Our findings inform recommendations for improving online examinations. These include increased preparation, practice and professionalism, to better simulate the in-person experience. Study credibility was strengthened by the triangulation of qualitative, quantitative and psychometric data.

Keywords

psychiatry medical students objective structured clinical examination virtual objective structured clinical examination qualitative

The summative assessment of our final year medical students in Psychiatry and Addiction Medicine requires students to role-play an interview with a simulated patient (SP) to demonstrate their interviewing, assessment, and communication skills. The first OSCE was described by Harden et al.,¹ providing an objective, structured approach to the evaluation of medical student clinical skills and behaviours. A growing body of literature supports its effectiveness in evaluating psychiatry competencies.² However, robust evaluation of the virtual OSCE is a critically important, yet under-researched, area.^3,4 Contemporaneous feedback on the user experience and attitudes is needed to better understand its future usefulness. Research is also needed that triangulates findings using multiple methods to enhance the credibility of results.⁵

Methods

In this mixed-method study, we collected quantitative and qualitative data from exam participants via an anonymous, voluntary online questionnaire. Our primary objective was to assess the level of organisation, ease of use, usefulness and overall acceptance of the v-OSCE. Qualitative data explored contexts related to participants’ perceptions and views.

Setting

The Academic Unit of Psychiatry and Addiction Medicine is based in Canberra, ACT. Due to COVID-19 restrictions, final (4^th) year students attended online lectures and interactive workshops. Assessments were conducted using Zoom,⁶ a secure, video-conferencing platform. The assessment consisted of a 20-min role-play interview where the students obtained a relevant history from the simulated patient. To ensure vertical equity, potentially disadvantaged students could use the medical school’s dedicated high-speed internet and computer facilities.

Participants and procedures

In early 2022, medical students, examiners and simulated patients (paid actors) were informed of online, summative v-OSCEs.⁷ After the examinations (October 2022), all three groups were sent an email invitation to participate in the voluntary survey, study information sheet and a survey link (see Supplemental Table S1). Return of the survey was approved by the Ethics Committee to constitute informed consent.

Survey questions measured user perceptions, acceptance and attitudes towards the v-OSCE, as well as open-ended questions related to beneficial aspects, challenging aspects and improvements (Table 1).

Table 1.

10-item survey measuring user perceptions, acceptance and attitudes towards v-OSCE

Survey items	Response type
Level of organisation	5-point Likert type scale^a:
Perceived ease of use of technology	1 = ‘not at all’,
Perceived usefulness for demonstrating student’s competencies	2 = ‘a little’, 3 = ‘moderately well’
Perceived usefulness for interacting with all participants	4 = ‘very well’
Perceived acceptance of v-OSCE	5 = ‘extremely well’.
Beneficial aspects	Open ended text responses
Challenging aspects
Suggestions for improving future v-OSCEs
Gender	Nominal scale
Age bracket	Ordinal scale
Past examination experience (examiners only)	Ordinal scale

^aResponses were dichotomised into unfavourable perceptions (‘not at all’/‘a little’) and favourable perceptions (‘moderately well’/‘very well’/‘extremely well’).

Quantitative analysis

We computed summary descriptive statistics of survey responses using Microsoft Excel. Likert-type ordinal data were dichotomised into 0 = none or a little, and 1 = moderately, very or extremely (with respect to sample strata of the relevant items). Gender and age bracket string data were transformed into numeric data with missing data excluded. We present results as frequencies and proportions of respondents.

Qualitative analysis

Thematic analysis of the qualitative online survey responses was performed by two of the researchers, according to Braun and Clarke’s six-phase process.⁸ Researcher one led phases 2-6. Data familiarisation involved independently reading transcripts, making notes of meaningful and relevant aspects of the data. Initial codes were assigned and collated to produce a shared coding framework, using NVivo 12.0.⁹ Coding was driven by data and semantics, recognising the contribution of each researchers’ subjectivity of data interpretation.¹⁰ In phase 3, the two coders applied this framework to the transcripts, working independently to search for appropriate themes and sub-themes. Conferencing achieved a ‘thematic map’. During Phases 5 and 6, themes were further explored iteratively by both coders, who reached a final framework of four overarching themes by consensus.

Reflexivity statement

Researcher one is a senior research coordinator with the academic unit. Researcher two is a consultant psychiatrist with experience in conducting and examining OSCEs. Thus, the results of the qualitative analysis may have been influenced by our own judgements, practices and belief systems.

Ethics approval

The project was approved by the ACT Health Human Research Ethics Committee (Protocol 2022/504).

Results

Of the participants in the 2022 v-OSCE, 47.5% (n = 47) of medical students, 77% (n = 10) of examiners, and 100% of simulated patients (n = 5) returned the survey. Table 2 summarises participant demographic data.

Table 2.

Demographic data: Medical students, examiners and simulated patients

		Medical students n = (%)	Examiners n = (%)	Simulated patients n = (%)
Gender	Female	24 (57.1%)	1 (12.5%)	0 (0.0%)
	Male	16 (38.1%)	6 (75.0%)	5 (100%)
	Other	0 (0.0%)	0 (0.0%)	0 (0.0%)
	Prefer not to say	2 (4.8%)	1 (12.5%)	0 (0.0%)
	Total	42^a	8^a	5
Age bracket	20–24	2 (4.8%)	0 (0.0%)	0 (0.0%)
	25–29	34 (80.1%)	1 (12.5%)	0 (0.0%)
	30–34	6 (14.3%)	1 (12.0%)	1 (20.0%)
	35–39	0 (0.0%)	0 (0.0%)	1 (20.0%)
	40–44	0 (0.0%)	1 (12.5%)	0 (0.0%)
	45+	0 (0.0%)	5 (62.5%)	3 (60.0%)
	Total	42^a	8^a	5
Examiner experience	No prior experience		0 (0.0%)
	1–3 examinations		2 (22.2%)
	4–7 examinations		1 (11.1%
	8+ examinations		6 (66.7%)

^aReduced n = due to missing data.

Quantitative results

Most participants (93.5%) rated the level of organisation of the v-OSCE favourably, and its ease of use and usefulness as at least moderately favourably, (ease of use = 98.4%, usefulness for demonstrating interviewing skills = 86.6%, and usefulness for interacting with other participants = 83.9%). Overall, 82.3% of participants reported the v-OSCE format was at least moderately acceptable (students = 83%, examiners = 80%, and simulated patients = 80% (see Figure 1).

Figure 1.

Participant responses to the v-OSCE survey questions (%). *at least moderately agreed.

Qualitative freeform feedback – thematic analysis

The results are supported by quotations from study participants (S = student, E = examiner, SP = simulated patient). Thematic analysis identified four main themes extracted from the survey.

Theme 1: Easy to use, efficient and convenient

Students reported that the online OSCE was easy to use: ‘everything was set out well and very easy to understand/follow’ (S7). Others highlighted its ‘smooth organisation and explanation of the OSCE structure and expectations’ (S38). The technology supported them ‘to adequately demonstrate all aspects of psychiatric interview’ (S43). The examiners appreciated the ‘efficient transition from one student to the next’ (E6), observing the ‘students were mostly relaxed which may have been contributed by being in their own familiar and comfortable environment’ (E6). The convenience of working from home and reduction in stress levels were common themes. Participants highlighted, ‘it is much less anxiety provoking and I feel I can demonstrate what I’m capable of much more easily’ (S7) and (working) ‘from home was a huge advantage, both in terms of preparation and low stress on the day’ (SP5).

Theme 2: Challenges of online OSCE

Challenges to the online exam were experienced by some. Several students reported that the online technology made it ‘more difficult to immerse yourself in the scenario’ (S27) and ‘difficult to read expression, body language, and responses’ (S37), making it harder ‘to smoothly conduct the interview’ (S13). One student observed:

It was difficult to pick up on non-verbal cues from the patient, which would have made the scenario more realistic. It was hard to demonstrate rapport and engagement with the patient, which we’ve been taught are important aspects of managing psychiatric patients (S20).

One examiner commented that the online technology did not allow them ‘to view the student’s facial expression when the actor is portraying distress or seeking empathy’ (E6). Although this problem could be overcome by adjusting the software settings in future, examiners observed that ‘the virtual space does not allow the same degree of professional intimacy and connectedness as face-to-face setting’ (E6). Examiners and simulated patients also commented on some distractions and variations in students’ home environments.

Theme 3: Technical and connectivity issues

Technical problems were reported, including poor sound quality, interrupted internet connection and bandwidth issues. One student stated the ‘audio broke out a few times interrupting the flow of a difficult discussion [making it] hard to stay “in character”’ (S8). Others found they ‘would often speak over the top of each other due to some lag time with the audio’ (S30). One student expressed concern about the standardization and fairness of v-OSCE and its impact on exam performance:

[It’s] hard to have a fair playing field. Some people have different home setups and availability to sit them at home, in person means we all are doing them in the same environment… Its just overall its more difficult to really shine over Zoom. It’s harder to interact with patients and the examiners (S35).

According to one examiner, the ‘occasional Wi-Fi drop-out disrupted the continuity of the interview’ (E6), however, others described them as ‘minor technical issues’ (E5) that ‘can be adjusted for with backups and flexible timing’ (E3). One simulated patient observed, ‘the students all handled any of those moments well’ (SP5). Despite these challenges, students provided positive feedback: ‘I really enjoyed the vignette which included everything and gave me enough time to really think about it’ (S15) including that Psychiatry and Addiction Medicine stations: ‘are, in my opinion, the only stations that can be completed appropriately online’ (S24).

Theme 4: Improvements to the OSCE

The detailed feedback from participants provided helpful suggestions for improving the online OSCE experience. A consistent recommendation was the use of better-quality microphones, sound quality and internet connection. Examiners recommended participants be offered a practice telehealth session which includes advice on the ideal, professional set up. One simulated patient appreciated the reserve actor who rapidly stepped in during some technical difficulties. Others described ‘it worked about as seamlessly as could be expected’ (SP5), feeling ‘impressed with how these online OSCE’s are organized and how generally they run very smoothly’ (SP3). On the other hand, some students (6/47) and examiners (3/10) overtly stated they preferred face-to face examinations. Table 3 summarises our recommendations for conducting v-OSCEs.

Table 3.

Recommendations for conducting virtual OSCEs

University investment in high-quality software

• Microphones: Ensure high quality head-sets and microphones are used to improve sound quality

• Computers: Use of hard-wired connectivity, rather than less reliable Wi-Fi

Prepare all participants

• Ensure participants join from a quiet room where they will not be disturbed

• Test their own equipment is reliable including adequate bandwidth for videoconferencing

• Emphasise the importance of professional/neutral backgrounds and clothing to replicate the in- person experience

Prepare students

• Tutorials on telehealth skills

• Workshop: Mock OSCE role-play practice using the online software

Prepare simulated patients

• Provide instructions on setting up the ideal telehealth session

• Establish good eye contact with the camera (not the screen)

• Ideal distance from computer to allow student observations of facial expressions, non-verbal cues and body language etc.

Improve exam efficiencies

• Plan for extra breaks between students (2–5 min) to reduce examiner fatigue

• Create spare examination slots at strategic points to cater for technical or connectivity problems

• Ensure there are back-up simulated patient and examiners

Evaluate and gather feedback

• Continuous quality improvement through debriefing and surveys about the exam experiences

Discussion

The majority of study participants rated the online technology as an acceptable format for the exam. Most perceived the v-OSCE as well organised, easy to use and useful for demonstrating their skills and interacting with others. The main challenges reported were difficulties in assessing body language and emotional (affective) prosody, defined as the non-verbal aspects of language that allow people to recognise or convey emotions.¹¹ This included a lack of access to the full range of facial expressions, posture, gesticulations, and nonverbal cues which would be accessible with an in-person interaction. Similar constraints have been reported in telepsychiatry¹² and further research could investigate how various v-OSCE configurations (e.g. station setup, close ups) impact on student examination performance. There were also internet connectivity challenges that de-synchronised the interaction and negatively impacted on participant immersion in the role-play examination.

According to feedback, immersion could be improved by using high-quality headsets and microphones as well as wired internet access. Contingency plans are recommended for unavailability of examiners and simulated patients and for connectivity issues (e.g. rescheduling and back-up access options). Students indicated they lacked practice opportunities to develop their digital and interpersonal skills with telehealth, referred to as one’s ‘webside manner’.¹³ We recommend establishing telepsychiatry tutorials and peer-assisted mock v-OSCE sessions in the curriculum to enhance student preparedness and confidence with online assessments.

Credibility refers to the believability of research findings, enriched by the convergence of multiple sources of data (triangulation). Wilkes et al. recently published an evaluation of student summative OSCE results for each year of the pandemic (compared to the 3 years prior).¹⁴ Results showed the technology was effective for assessing examination competence, with a high degree of internal consistency, reliability and validity. Our recommendations for practical measures to improve the examination process concord with the findings from recent systematic reviews of health professional and medical student online examination adaptations during the pandemic.^3–5,13 Despite the limitations of the virtual examination, we conclude that it is an acceptable, efficient and useful modality for medical student examinations.

Limitations and future considerations

Our mixed-methods study surveyed a small sample of medical students, examiners and simulated patients, which may limit the generalisability of the findings. Although the e-OSCE was a response to pandemic restrictions and not an initiative of the authors, it’s important to note we were investigating our own examination processes. Future research could investigate whether improvements to the teaching curriculum (e.g. e-learning modules, tutorials and/or peer-assisted mock-OSCEs) could enhance student exam confidence and performance. A controlled trial of face-to-face compared to v-OSCE is warranted.

Conclusion

Medical students, examiners and simulated patients found that v-OSCE examinations are convenient, easy to use, efficient and acceptable. To move towards a level playing field, we need to ensure familiarity with professional telehealth interviewing, access to high quality videoconferencing and high-speed internet connections. This is likely to better simulate the in-person experience and, therefore, acceptance of online technologies. Lastly, telepsychiatry is increasingly being normalized, becoming an important tool for psychiatrists to delivery health care. This necessitates that medical schools teach and assess telehealth skills. The integration of telehealth competencies and assessment tools in medical schools is essential to equip students with the required skills for their future careers.

Supplemental Material

Supplemental Material - A level playing field? Evaluation of the virtual Objective Structured Clinical Examination in Psychiatry and Addiction Medicine: A mixed methods study

Supplemental Material for A level playing field? Evaluation of the virtual Objective Structured Clinical Examination in Psychiatry and Addiction Medicine: A mixed methods study by Rebecca E Reay, Paul A Maguire and Jeffrey CL Looi in Australasian Psychiatry.

Footnotes

Author’s note

We confirm that all authors gave final approval to the submitted paper.

Acknowledgements

We express our gratitude to all study participants for their contributions to this research. We wish thank Dr Fiona Wilkes for her helpful feedback on the manuscript.

Disclosure

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Ethics statement

ORCID iDs

Rebecca E Reay

Jeffrey CL Looi

Paul Maguire

Data availability statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.*

Supplemental Material

Supplemental material for this article is available online.

References

Harden

Stevenson

Downie

, et al. Assessment of clinical competence using objective structured examinations. BMJ 1975; 22: 447–451.

Plakiotis

. Objective structured clinical examination (OSCE) in psychiatry education: a review of its role in competency-based assessment. Adv Exp Med Biol 2017; 988: 159–180.

Kunutsor

Metcalf

Westacott

, et al. Are remote clinical assessments a feasible and acceptable method of assessment? A systematic review. Med Teach 2022; 44: 300–308.

Felthun

Taylor

Shulruf

, et al. Assessment methods and the validity and reliability of measurement tools in online objective structured clinical examinations: a systematic scoping review. J Educ Eval Health Prof 2021; 18: 11.

Chan

SCC

Choa

Kelly

, et al. Implementation of virtual OSCE in health professions education: a systematic review. Med Educ 2023; 57: 833–843.

Zoom Video Communications Inc . Zoom version 5.16.10. (2023) San Jose, CA: Zoom Video Communications Inc. https://zoom.us/, (accessed 19 October 2022).

Looi

JCL

Bonner

Maguire

. Medical education in psychiatry and addiction medicine: pandemic education as usual or the new normal? Australas Psychiatr 2022; 30: 669. DOI: 10.1177/10398562221106062.

Braun

Clarke

. Using thematic analysis in psychology. Qual Res Psychol 2006; 3: 77–101.

Richards

. Using Nvivo in qualitative research. London: Sage, 1999.

10.

Braun

Clarke

. Can I use TA? should I use TA? should I not use TA? comparing reflexive thematic analysis and other pattern-based qualitative analytic approaches. Counsell Psychother Res J 2021; 21: 37–47.

11.

Leentjens

AFG

Wielaert

van Harskamp

, et al. Disturbances of affective prosody in patients with schizophrenia; a crosssectional study. J Neurol Neurosurg Psychiatry 1998; 64: 375–378.

12.

Looi

Pring

. Private metropolitan telepsychiatry in Australia during Covid-19: current practice and future developments. Australas Psychiatr 2020; 28: 508–510.

13.

Lim

GHT

Gera

Hany Kamel

, et al. “We need more practice”: evaluating the role of virtual mock OSCES in the undergraduate programme during the COVID pandemic. Adv Med Educ Pract 2023; 14: 157–166.

14.

Wilkes

FAL

Looi

JCL

Maguire

, et al. (2024) Online medical student OSCE examinations during the first three years of the COVID-19 pandemic compared to three years pre-pandemic: an Australian experience in psychiatry and addiction medicine. Med Teach. doi: 10.1080/0142159X.2023.2279918, In press.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.46 MB