Comparability of Objective Structured Clinical Examinations (OSCEs) and Written Tests for Assessing Medical School Students’ Competencies: A Scoping Review

Abstract

Objective Structured Clinical Examinations (OSCEs) and written tests are commonly used to assess health professional students, but it remains unclear whether the additional human resources and expenses required for OSCEs, both in-person and online, are worthwhile for assessing competencies. This scoping review summarized literature identified by searching MEDLINE and EMBASE comparing 1) OSCEs and written tests and 2) in-person and online OSCEs, for assessing health professional trainees’ competencies. For Q1, 21 studies satisfied inclusion criteria. The most examined health profession was medical trainees (19, 90.5%), the comparison was most frequently OSCEs versus multiple-choice questions (MCQs) (18, 85.7%), and 18 (87.5%) examined the same competency domain. Most (77.5%) total score correlation coefficients between testing methods were weak (r < 0.40). For Q2, 13 articles were included. In-person and online OSCEs were most used for medical trainees (9, 69.2%), checklists were the most prevalent evaluation scheme (7, 63.6%), and 14/17 overall score comparisons were not statistically significantly different. Generally low correlations exist between MCQ and OSCE scores, providing insufficient evidence as to whether OSCEs provide sufficient value to be worth their additional cost. Online OSCEs may be a viable alternative to in-person OSCEs for certain competencies where technical challenges can be met.

Keywords

clinical competence curriculum COVID-19 distance education professional education

Introduction

Multiple-choice questions (MCQs), written answer questions, and Objective Structured Clinical Examinations (OSCEs) form the current backbone of medical and other health professional education assessments internationally. In Canada, trainees are usually evaluated using the CanMEDS framework with its seven domains—Medical Expert, Communicator, Collaborator, Leader, Health Advocate, Scholar, and Professional (Royal College of Physicians and Surgeons of Canada, 2015). Other countries have a similar set of core competencies that are organized within their own respective framework names (Accreditation Council for Graduate Medical Education, 2022; General Medical Council, 2017; Medical Board of Australia, 2021).

MCQs with single best answer have become the dominant testing method but have well-known limitations – mainly the cueing effect, testing the ability to recognize an answer rather than to generate an answer, and difficulty representing many important areas of medicine where no appropriate distractors can be created (Sam et al., 2016). Long answer written questions are used sparingly as they are time-consuming to mark.

OSCEs were first developed in the 1970s to improve the assessment of clinical skills by creating a more objective setting in which examiners and patients could be standardized so that all candidates would be assessed in a similar way to reduce bias (Harden et al., 1975). Since its advent, OSCEs have been incorporated into all medical curricula and most certification requirements in Canada, the United States, and many countries abroad (Accreditation Council for Graduate Medical Education, 2022; Australian Medical Council Limited, 2021; General Medical Council, 2023; Medical Council of Canada, 2023). OSCEs have been found to be useful in preparing students to manage conditions both common and rare as well as perform interventions while incorporating competency domains (Dizon et al., 2021; Jutant et al., 2022; Pugh et al., 2015).

Understanding the relative merits of both testing methods in assessing health professional trainees is necessary to ensure resources, including scarce faculty, staff, and simulated patient time and costs are used wisely. Costs associated with running OSCEs can easily surpass $600 USD per student, with some institutions spending upwards of $900 USD per student (Grand’Maison et al., 1992; Sudan et al., 2015; Walsh et al., 2009) with extensive time commitments to recruit, train, set up, equip, operate, and clean up. Written testing methods do not require the same time and resources, so they are the logical assessment method unless OSCEs can be shown to be superior for testing certain competencies.

During the COVID-19 pandemic when in-person OSCE assessments were not possible, many health professional schools rapidly implemented online OSCEs as a substitute (Hytönen et al., 2021; Jamil et al., 2022; Luke et al., 2021; Nusanti et al., 2021). While online OSCEs are modeled after in-person ones, the online format itself is a major change, and the number of stations or competencies tested may be altered; therefore, its validity and reliability are uncertain.

Our effort to develop efficient assessment methods for Clinical Pharmacology and Toxicology (CPT) curricula in medical education with its focus on therapeutics and toxicology knowledge, patient communication, and safe prescribing in a rapidly evolving discipline, led to the importance of evaluating the relative merits of OSCEs versus written exams or online OSCEs (Holbrook et al., 2019; Liu et al., 2018; Qayyum et al., 2012; Wu et al., 2015; Zhang et al., 2019). Understanding the value of various testing methods is an important undertaking for every health professional training program worldwide with limited resources.

The objective for this scoping review was to characterize the evidence on the comparability of OSCEs versus written exams and in-person versus online OSCEs for assessing competencies of health professional trainees.

Method

The protocol for this study was registered on OSF registries (https://osf.io/6u8fy/) (Holbrook et al., 2021). We conducted this scoping review in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR) framework (Tricco et al., 2018).

Research questions

The first research question is as follows: Are OSCEs a comparable evaluation method to multiple-choice or short/long answer written questions in assessing health professional trainees and, if so, for which competency domains (these seven competency domains were used: Medical Expert – the integrating role, Communicator, Collaborator, Leader, Health Advocate, Scholar, and Professional)?

The second research question asked: Are online OSCEs (examinee and examiner, with or without simulated patient, all online and at distance from each other) superior to in-person OSCEs (examinee and examiner, with or without simulated patient in the same location) in terms of logistics, feasibility, resource intensity, for assessing health professional trainees, and if so, for which competency domains?

Eligibility criteria

Our inclusion criteria are highlighted as follows within the PICOTS framework (Guyatt et al., 2015). For Q1 (OSCEs vs. written tests), the Population was health professional trainees (e.g., students or residents), excluding veterinary trainees; the Intervention was OSCE stations; the Comparator was written or typed testing methods (e.g., multiple-choice questions, short/long form answers); the Outcomes were overall score comparisons and individual competency domain score comparisons, as measured by the study itself; the Timeframe included studies published between 1946 and 2021; and the Study design/type included studies that had to have a control/comparison group (e.g., RCTs, cohort studies, case–control studies, historical controls). Included studies also had to report primary data and be written in English.

For Q2 (online vs. in-person OSCEs), the Population was health professional trainees (e.g., students or residents), excluding veterinary trainees; the Intervention was online OSCE stations (test taker communicates information through an online platform); the Comparator was in-person OSCE stations (test taker communicates information in a face-to-face, in-person setting); the Outcomes were overall score comparisons, individual competency domain score comparisons, and logistics, feasibility, and resource intensity, as measured by the study itself; the Timeframe included studies published between 2000 and 2021 (timeframe was reduced as online OSCEs are a recent phenomenon); and the Study design/type included studies that had to have a control/comparison group (e.g., RCTs, cohort studies, case–control studies, historical controls). Included studies also had to report primary data and be written in English. For both research questions, editorials, commentaries, and guidelines were excluded. Conference abstracts were considered since they may have relevant information or data.

Literature search

We conducted searches in Ovid MEDLINE (Epub Ahead of Print, In-Process, In-Data-Review and Other Non-Indexed Citations and Daily, Ovid MEDLINE(R) Daily and Ovid MEDLINE(R) 1946 to Present) and Ovid EMBASE. The search strategy was created with the assistance of university and hospital research librarians and can be found in Supplemental Appendix 1.

We searched databases from inception (1946 for MEDLINE and 1974 for EMBASE) to August 3, 2021 for Q1, and from 2000 to August 8, 2021 for Q2. No restrictions were placed for a study’s country of origin. Only these two bibliographic databases were searched given the time-sensitive nature of the study and because there is evidence that searching MEDLINE and EMBASE provides a sufficient proportion of relevant studies (93%) compared to searching all databases (Bramer et al., 2017; Rice et al., 2016; van Enst et al., 2014).

Study selection

Screening

Several authors performed title and abstract screening in duplicate and independently on the online platform Covidence (https://www.covidence.org/). Disagreements in screening were resolved by consensus. Studies that passed initial screening underwent full-text screening in Covidence (Supplemental Appendix 2, which exhibits process flow). We conducted full-text screening in the same manner and recorded reasons for exclusion.

Data extraction

Paired and independent data extraction was completed using Microsoft Excel. A standardized form was used by the team to extract data from the studies. We extracted information on study characteristics (e.g., publication year, study design, setting, sample size), participant characteristics (e.g., health profession, level of education/training), assessment outcomes including competency domains, and statistics on the results. In anticipation of varying competency domain descriptors used in different countries, each domain description was reconciled to a CanMEDS domain for consistency. For example, if an OSCE station assessed a trainee’s ability to provide patient-friendly education, this station would be classified as assessing the Communicator domain. As this study was a scoping review, a formal risk of bias assessment was not performed (Peters et al., 2015).

Data synthesis

The analyses were descriptive and reported on 1) comparisons between OSCEs and written testing methods (through total scores and competency domain scores), and 2) comparisons between in-person and online OSCEs (through total scores and competency domain scores).

Results

Q1 – OSCEs versus written testing methods

The search yielded 1782 eligible citations, with 21 relevant articles remaining after title, abstract, and full-text screening (Andrades et al., 2017; Auewarakul et al., 2005; Butler et al., 2017; Chibnall & Blaskiewicz, 2008; Couto et al., 2019; Dennehy et al., 2008; Dong et al., 2014, 2017; Eftekhar et al., 2012; Gillette et al., 2017; Gilson et al., 1998; Hull et al., 1995; Huwendiek et al., 2017; Jameel et al., 2015; Kelly et al., 2013; Nuovo et al., 2006; Schleicher et al., 2017; Schoeman & Chandratilake, 2012; Schwartz et al., 1995; Simon et al., 2002, 2007). The PRISMA flow diagram for this research question can be found in Figure 1 (Page et al., 2021). Detailed study characteristics are shown in Table 1. No studies used a randomized trial (including a randomized crossover) design, but 19 studies employed a weak before-and-after study design, with one each of prediction analysis and cross-sectional study. None ensured that outcome assessors were blinded to group allocation during grading. Twelve of the 21 studies were conducted in the United States, and the remaining were conducted in other countries (e.g., Iran, Pakistan, Thailand). Medical trainees were the focus of 19 studies, and 16 examined students (rather than residents or fellows). The most common written comparison method was MCQs, which were used in 18 studies, and the mean sample size was 258.1 (SD 196.4). More than 4645 participants were included, with no sample size information given by three studies. Characteristics of individual studies can be found in Supplemental Appendix 3.

Figure 1.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR) flowchart for the identification and selection of studies comparing a) Objective Structured Clinical Examinations (OSCEs) with written testing methods and b) in-person and online OSCEs.

Table 1.

OSCEs versus Written Assessments – Study Characteristics and Summary Results.

Study Characteristics	All StudiesN = 21 (100%)
Publication year
2015 and onwards	8 (38.1)
Before 2015	13 (61.9)
Country of origin of participants
USA	12 (57.1)
Other (Brazil, Britain, Germany, Iran, Ireland, Pakistan, Thailand)	9 (42.9)
Study design
Before-and-after	19 (90.5)
Other	2 (9.5)
Number of OSCE stations^a
<10	6 (37.5)
10-19	7 (43.8)
>19	3 (18.8)
In-person OSCE stations^a	20 (100.0)
Outcome assessors blinded to group allocation during grading	0 (0.0)
Single medical school/training program	20 (95.2)
Type of health profession
Medicine	19 (90.5)
Dentistry	1 (4.8)
Pharmacy	1 (4.8)
Level of training
Student	16 (76.2)
Resident	5 (23.8)
Written comparison method(s)^b
Multiple-choice question	18 (85.7)
Short answer	4 (19.0)
Long answer	3 (14.3)
Total participant size at beginning of study^a
<100	5 (27.8)
100-199	4 (22.2)
200-299	0 (0.00)
300-399	5 (27.8)
>399	4 (22.2)
Summary results	Number (%) studies
Studies assessing each competency domain
Medical Expert	18 (85.7)
Communicator	15 (71.4)
Collaborator	0 (0.0)
Leader	0 (0.0)
Health Advocate	0 (0.0)
Scholar	0 (0.0)
Professional	8 (38.1)
Unclear	1 (4.8)
OSCE evaluation scheme^a
Checklist	8 (61.5)
Rubric	1 (7.7)
Other	4 (30.8)
Same competency domains compared^a,c	18 (85.7)
Total score correlation coefficient magnitude^a: n comparisons with written assessments
>0.79	0 (0.0)
0.40–0.79	9 (22.5)
<0.40	31 (77.5)
Competency domain score correlation coefficient magnitude^a,d: n comparisons with written assessments
>0.79	0 (0.0)
0.40–0.79	2 (6.5)
<0.40	29 (93.5)
‘Gold standard’ comparison included^a,e	10 (66.7)

Notes. OSCE, Observed Structured Clinical Examination;

^aInformation not available for all studies;

^bMore than one answer possible;

^cAssessments all compared Medical Expert;

^dCompetency domain score included 25 Medical Expert, four Communication, two Professionalism;

^ee.g., national licensing exams or in-training exams.

The Medical Expert and Communicator roles were most commonly assessed, with 18 and 15 studies, respectively. No studies examined the Collaborator, Leader, Health Advocate, or Scholar roles. A majority of studies used an OSCE checklist as the evaluation method and 10 (66.7%) had a ‘gold standard’ comparison (e.g., national licensing exam or in-training exam). Analyses were based on correlation methods. Of the 40 total score correlations between OSCEs and written tests, 31 comparisons had a correlation coefficient (r) magnitude less than 0.40, indicating low correlation. Similarly, 29 of the 31 competency domain score correlations between testing methods yielded an r value less than 0.40. In 18 studies, the same competency domains were compared, but only 14 also reported total score correlations (a total of 32 comparisons of which 25 were low (r < 0.40)) (Andrades et al., 2017; Auewarakul et al., 2005; Butler et al., 2017; Chibnall & Blaskiewicz, 2008; Couto et al., 2019; Dennehy et al., 2008; Dong et al., 2014, 2017; Eftekhar et al., 2012; Gilson et al., 1998; Hull et al., 1995; Huwendiek et al., 2017; Kelly et al., 2013; Nuovo et al., 2006; Schleicher et al., 2017; Schwartz et al., 1995; Simon et al., 2002, 2007; Schoeman & Chandratilake, 2012). Of the four studies where the Communicator competency domain was compared to written assessments, all correlations were low (Gillette et al., 2017; Kelly et al., 2013; Simon et al., 2002, 2007). Detailed results are reported in Table 1.

Q2 – in-person versus online OSCEs

Our search yielded 306 potentially relevant articles with 13 articles remaining after title, abstract, and full-text screening (Arrogante et al., 2021; Biolik et al., 2018; Boardman et al., 2021; Darr et al., 2021; Farrell et al., 2021; Hsia et al., 2021; Lara et al., 2020; Martinez et al., 2020; Nackman et al., 2006; Novack et al., 2002; Oliven et al., 2011; Scoular et al., 2021; Sihombing et al., 2021). The PRISMA flow diagram for this research question can be found in Figure 1 (Page et al., 2021). Ten of 13 included studies used a weak before-and-after study design, two were randomized crossover trials, and one was a prospective cohort with concurrent control. No outcome assessors were blinded to group allocation during grading. Nine studies were conducted in the United States, and the remaining four were carried out in Germany, Indonesia, Israel, and Spain. Nine studies focused on medical trainees, and 11 studies examined students (as opposed to residents or fellows). Most included four or more OSCE stations for online and in-person formats, and the mean sample size was 158.8 (SD 84.4). A total of 2064 trainees participated, but two studies reported only the number of trainees who participated in the online OSCE. Full characteristics of included studies can be found in Table 2, and characteristics of individual studies can be found in Supplemental Appendix 4.

Table 2.

Online versus In-person OSCEs – Study Characteristics and Summary Results.

Study Characteristics	All StudiesN = 13 (100%)
Publication year
2021	7 (53.8)
Before 2021	6 (46.2)
Country of origin of participants
USA	9 (69.2)
Other (Spain, Germany, Israel, Indonesia)	4 (30.8)
Study design
Randomized crossover trial	2 (15.4)
Prospective cohort with concurrent control	1 (7.7)
Before-and-after	10 (76.9)
Number of in-person OSCE stations^a
<4	3 (20.0)
4-6	2 (13.3)
>6	2 (13.3)
Number of online OSCE stations^a
<4	4 (40.0)
4-6	3 (30.0)
>6	3 (30.0)
Outcome assessors were blinded to group allocation during grading	0 (0.0)
Single medical school/training program	12 (92.3)
Type of health profession
Medicine	9 (69.2)
Nursing	1 (7.7)
Pharmacy	3 (23.1)
Level of training
Student	11 (84.6)
Resident	2 (15.4)
Online OSCEs have separate examiner, examinee, and SP: Yes	8 (46.7)
Total participant size at beginning of study (in-person and online)^a
<50	1 (9.1)
50-99	0 (0.0)
100-149	2 (18.2)
150-200	3 (27.3)
>200	5 (45.5)
Summary results	Number (%) studies
Studies assessing each competency domain
Medical Expert	9 (69.2)
Communicator	9 (69.2)
Collaborator	2 (15.4)
Leader	0 (0.0)
Health Advocate	2 (15.4)
Scholar	0 (0.0)
Professional	3 (23.1)
Unclear	1 (7.7)
Evaluation scheme^a
Checklist	7 (63.6)
Rubric	2 (18.2)
Other	2 (18.2)
Overall score: n comparisons
No statistically significant differences	14 (82.4)
In-person statistically significantly higher	0 (0.0)
Online statistically significantly higher	3 (17.6)
Medical expert competency domain score^b: n comparisons
No statistically significant differences	8 (66.7)
In-person statistically significantly higher	3 (25.0)
Online statistically significantly higher	1 (8.3)
Communication competency domain score: n comparisons
No statistically significant differences	10 (55.6)
In-person statistically significantly higher	3 (16.7)
Online statistically significantly higher	5 (27.8)
Collaborator competency domain score: n comparisons
No statistically significant differences	1 (50.0)
In-person statistically significantly higher	1 (50.0)
Online statistically significantly higher	0 (0.0)
Health Advocate competency domain score: n comparisons
No statistically significant differences	1 (50.0)
In-person statistically significantly higher	1 (50.0)
Online statistically significantly higher	0 (0.0)
Professional competency domain score: n comparisons
No statistically significant differences	2 (50.0)
In-person statistically significantly higher	2 (50.0)
Online statistically significantly higher	0 (0.0)

Notes. OSCE, Objective Structured Clinical Examination; SP, standardized patient;

^aInformation not available for all studies;

^bNo studies addressed the Leader or Scholar role.

The most commonly assessed competency domains were Medical Expert and Communicator, with nine studies each. None of the studies assessed the Leader or Scholar roles. Most studies also used an OSCE checklist as the evaluation method. Of the 17 comparisons of an overall OSCE score derived from summation of individual station scores, 14 of them found no statistically significant difference between in-person and online OSCEs, while three found online OSCE scores were statistically significantly higher. When breaking down scores by competency domains, most studies also found no difference between scores. However, the Communicator competency domain was the most varied, where 10 comparisons found no difference, three found in-person scores were higher, and five found online scores were higher. For the Professional competency domain, two comparisons found no score differences between in-person and online OSCEs, while two found in-person scores to be higher. Overall, it appears that online OSCEs and in-person OSCEs are comparable for assessing the Communicator and Medical Expert competency domains. Results are summarized in more detail in Table 2.

Discussion

OSCEs have been a standard and popular health professional assessment method for several decades now, particularly for clinical skills, despite the large cost of OSCEs due to the extensive faculty organization and involvement required (Rushforth, 2007; Walsh et al., 2009; Zayyan, 2011). It was therefore surprising not to find any systematic or scoping reviews assessing OSCE comparability to written methods for the assessment of health professional trainee competencies in our searches. Two previous literature reviews did report generally low to moderate correlations between testing methods but focused more on the validity and reliability of OSCEs rather than comparability (Alunno et al., 2020; Walsh et al., 2009). Alunno et al. (2020) reviewed 36 studies on the assessment of resident competencies, finding low score correlation with MCQs but good inter-rater reliability and internal consistency. In contrast, Walsh et al. (2009) found that OSCEs had been inadequately studied for nursing competencies, including their expense, which was prohibitive in some studies.

For our research question on OSCEs versus written testing methods, we found most total score and competency domain score correlations between the formats were low. A previous review found similar results in that most correlation coefficients between standardized patient (SP)-based tests and other measures, including MCQs, short answer tests, clinical ratings, and National Board of Medical Examiners tests were ≤ 0.50 (Turner & Dankoski, 2008; van der Vleuten & Swanson, 1990). Low correlations between the testing methods may not be surprising if OSCEs are truly testing different constructs than written exams or, given the commonly used before-after designs, the two types of assessments are completed at different phases of trainee expertise. Alternatively, it has been suggested that low correlations arise from limited OSCE validity (Turner & Dankoski, 2008). For example, one study had lay volunteers review a total of 100 medical student encounters and found the volunteers’ perceptions of effective communication were not well correlated with OSCE communication checklist scores completed by trained raters in four of five stations (r = 0.03, 0.02, 0.13, and 0.33) (Mazor et al., 2005). An analysis of various OSCE communication checklists found their psychometric properties to be lacking or uncertain in many areas (e.g., content validity, agreement, criterion validity, responsiveness, etc.) (Cömert et al., 2016). These shortcomings are not limited to communication OSCEs and may be an indication of the general problems related to studies on the development and psychometric testing of OSCEs, such as missing data and absence of structure in reporting (Patrício et al., 2009).

Some of the best evidence on the value of OSCEs and written exams come from two studies that correlated exam results to later quality of care. Both certification exams (involving OSCEs and written components) for family medicine in Quebec and the Medical Council of Canada Qualifying Examinations (MCCQE) (written exam) were predictive of the quality of preventative care and disease management during the subsequent first decade of practice (Tamblyn et al., 2002). The second study found that low scores on Part I (written components) of the MCCQE predicted subpar quality-of-care in regulatory college peer assessments and inclusion of Part II results (OSCEs) did not statistically significantly improve predictive power (Wenghofer et al., 2009). These findings indicate a need for further investigation into the added value of including OSCEs in examinations.

The overall consensus from the 13 included studies indicated that online OSCEs were comparable to in-person ones in ability to assess the Medical Expert and Communicator competency domains. In particular, the two studies that employed the most robust study designs (randomized crossover trials) found no statistically significant differences in scores between OSCE formats, providing stronger evidence for their comparability (Biolik et al., 2018; Oliven et al., 2011). However, due to the lack of or small number of comparisons for the other competencies (e.g., Collaborator, Leader, Health Advocate, Scholar, and Professional), more studies will need to be conducted before a definitive conclusion can be reached as to whether online OSCEs are comparable to in-person ones for assessing these domains. Considering that online OSCEs were not widely used pre-pandemic, this rapid transition to online testing has been remarkable, given the negative impact that the pandemic has had on in-person learning. Online OSCEs have also been well-received by trainees, faculty, as well as SPs (Farrell et al., 2021; Hsia et al., 2021; Kelly et al., 2022; Martinez et al., 2020). Thus, online OSCEs seem to be a viable alternative for in-person ones for many station types. However, the advantages in terms of convenience and social distancing are counteracted by the technical difficulties of running an online OSCE as well as the lack of ability to fully test physical exam skills (Kupfer, 2020). The format of choice may ultimately rest upon which competencies are to be evaluated and the availability of in-person faculty evaluators and SPs versus the technical infrastructure and troubleshooting required for an online format. Unfortunately, we are unable to comment on relative costs, logistics, or feasibility, as no data were provided in the studies.

Strengths and limitations

This present study has several strengths. To our knowledge, this study is the first scoping review to summarize evidence on OSCEs compared to written tests as well as to compare in-person and online OSCEs for assessing clinical competence in health professional trainees. Moreover, we were able to gather enough evidence to provide an evaluation for more than one competency domain. However, there are limitations to our scoping review being able to come to strong conclusions. One is the lack of high-quality studies investigating the assessment abilities of OSCEs. Based on the design of the studies, the quality of evidence is low, but as this was a scoping review, we did not formally assess the included studies’ risk of bias. Moreover, articles not published in English were not included, so it is possible that a small number of relevant studies were missed. Similarly, searching more exhaustively in other bibliographic databases might have identified a few additional articles.

Implications

Based on the findings of this review, we decided that there was insufficient evidence to support the superiority of OSCEs in assessing competencies compared to written exams. As the initial impetus of the study was to determine whether to mount a CPT OSCE – either in-person or online, the focus of our larger study has shifted towards developing and validating computerized marking of prescription writing. We have a severe shortage of faculty to supervise OSCEs, and written exams, while limited for some competencies, are more feasible at this time. Other health professional education programs will continue to assess on a regular basis whether they require OSCEs, either in-person or online, depending on their faculty and financial resources.

Future studies should compare in-person OSCEs, online OSCEs, and written testing methods for the same competency domains, using concurrent comparison groups. Moreover, the validity and reliability of OSCEs require further work, although this undertaking would require the difficult job of producing a ‘diagnostic standard’ for threshold of competency beyond expert judgment. Additionally, future explorations comparing in-person and online OSCEs should examine other competency domains besides the Medical Expert and Communicator roles in order to provide evidence for comparability in testing the other competencies. Greater focus should also be placed on understanding the logistics, costs, and feasibility of OSCEs.

Conclusion

This scoping review was unable to confirm the value for additional resources and costs required for OSCEs compared to written exams. We found generally low correlations between scores on written assessments and OSCEs for medical trainees for different competency domains. Online OSCEs appear to be comparable to in-person OSCEs in assessing the Medical Expert and Communicator roles, although no conclusion could be arrived at for the remaining competencies due to a lack of data. These results indicate the need for further research to understand why correlations were low between testing methods and whether there are competency domains for which OSCEs provide sufficient value to be worth their additional cost.

Supplemental Material

Comparability of Objective Structured Clinical Examinations (OSCEs) and Written Tests for Assessing Medical School Students’ Competencies: A Scoping Review

Comparability of Objective Structured Clinical Examinations (OSCEs) and Written Tests for Assessing Medical School Students’ Competencies: A Scoping Review by Oswin Chang, Anne M Holbrook, Simran Lohit, Jiawen Deng, Janice Xu, Munil Lee, Alan Cheng in Evaluation & the Health Professions

Supplemental Material: Appendices

Supplemental Material - Comparability of Objective Structured Clinical Examinations (OSCEs) and Written Tests for Assessing Medical School Students’ Competencies: A Scoping Review

Supplemental Material for Comparability of Objective Structured Clinical Examinations (OSCEs) and Written Tests for Assessing Medical School Students’ Competencies: A Scoping Review by Oswin Chang, Anne M Holbrook, Simran Lohit, Jiawen Deng, Janice Xu, Munil Lee, Alan Cheng in Evaluation & the Health Professions

Footnotes

Acknowledgments

The authors wish to thank librarians Susanna Galbraith and Kaitryn Campbell for their assistance with creating the search strategy as well as Cavizshajan Skanthan for his assistance with screening articles.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by a Virtual Learning Strategy Grant from eCampusOntario to Dr. Holbrook.

ORCID iDs

Oswin Chang

Jiawen Deng

Supplemental Material

Supplemental material for this article is available online.

References

Accreditation Council for Graduate Medical Education . (2022, July 1). Common Program Requirements. https://www.acgme.org/what-we-do/accreditation/common-program-requirements/

Alunno

Najm

Sivera

Haines

Falzon

Ramiro

(2020). Assessment of competences in rheumatology training: Results of a systematic literature review to inform EULAR points to consider. RMD Open, 6(2), e001330. https://doi.org/10.1136/rmdopen-2020-001330

Andrades

Yousuf

Sattar

K. A.

Kausar

(2017). Predictive validity of family medicine resident’s performance at Induction with their future performance during residency. Journal of the College of Physicians and Surgeons-Pakistan: JCPSP, 27(6), 338–341. https://pubmed.ncbi.nlm.nih.gov/28689521/

Arrogante

López-Torre

E. M.

Carrión-García

Polo

Jiménez-Rodríguez

(2021). High-fidelity virtual objective structured clinical examinations with standardized patients in nursing students: An Innovative Proposal during the COVID-19 pandemic. Healthcare, 9(3), 355. https://doi.org/10.3390/healthcare9030355

Auewarakul

Downing

S. M.

Jaturatamrong

Praditsuwan

(2005). Sources of validity evidence for an internal medicine student evaluation system: An evaluative study of assessment methods. Medical Education, 39(3), 276–283. https://doi.org/10.1111/j.1365-2929.2005.02090.x

Australian Medical Council Limited . (2021, July). Clinical examination Specifications. https://www.amc.org.au/wp-content/uploads/2021/07/2021-07-06-Clinical-Exam-Spec.V0.5.pdf

Biolik

Heide

Lessig

Hachmann

Stoevesandt

Kellner

, et al (2018). Objective structured clinical examination “Death Certificate” station—Computer-based versus conventional exam format. Journal of Forensic and Legal Medicine, 55, 33–38. https://doi.org/10.1016/j.jflm.2018.02.010

Boardman

Wilhite

J. A.

Adams

Sartori

Greene

Hanley

, et al (2021). Telemedicine training in the COVID Era: Revamping a Routine OSCE to Prepare medicine residents for virtual care. Journal of Medical Education and Curricular Development, 8, 23821205211024076. https://doi.org/10.1177/23821205211024076

Bramer

W. M.

Rethlefsen

M. L.

Kleijnen

Franco

O. H.

(2017). Optimal database combinations for literature searches in systematic reviews: A prospective exploratory study. Systematic Reviews, 6(1), 245. https://doi.org/10.1186/s13643-017-0644-y

10.

Butler

K. L.

Hirsh

D. A.

Petrusa

E. R.

Yeh

D. D.

Stearns

Sloane

D. E.

, et al (2017). Surgery clerkship evaluations are insufficient for clinical skills Appraisal: The value of a medical student surgical objective structured clinical examination. Journal of Surgical Education, 74(2), 286–294. https://doi.org/10.1016/j.jsurg.2016.08.018

11.

Chibnall

J. T.

Blaskiewicz

R. J.

(2008). Do clinical evaluations in a psychiatry clerkship favor students with positive personality characteristics? Academic Psychiatry: The Journal of the American Association of Directors of Psychiatric Residency Training and the Association for Academic Psychiatry, 32(3), 199–205. https://doi.org/10.1176/appi.ap.32.3.199

12.

Cömert

Zill

J. M.

Christalle

Dirmaier

Härter

Scholl

(2016). Assessing communication skills of medical students in objective structured clinical examinations (OSCE)—a systematic review of rating Scales. PLoS ONE, 11(3), e0152717. https://doi.org/10.1371/journal.pone.0152717

13.

Couto

L. B.

Durand

M. T.

Wolff

A. C. D.

Restini

C. B. A.

Faria

Romão

G. S.

, et al (2019). Formative assessment scores in tutorial sessions correlates with OSCE and progress testing scores in a PBL medical curriculum. Medical Education Online, 24(1), 1560862. https://doi.org/10.1080/10872981.2018.1560862

14.

Darr

A. Y.

Kyner

Fletcher

Yoder

(2021). Comparison of pharmacy students’ performance in a Laboratory Course Delivered live versus by virtual Facilitation. American Journal of Pharmaceutical Education, 85(2), 8072. https://doi.org/10.5688/ajpe8072

15.

Dennehy

P. C.

Susarla

S. M.

Karimbux

N. Y.

(2008). Relationship between dental students’ performance on standardized multiple-choice examinations and OSCEs. Journal of Dental Education, 72(5), 585–592. https://doi.org/10.1002/j.0022-0337.2008.72.5.tb04523.x

16.

Dizon

Malcolm

J. C.

Rethans

J. J.

Pugh

(2021). Assessing the validity of an OSCE developed to assess rare, emergent or complex clinical conditions in endocrinology and metabolism. BMC Medical Education, 21(1), 288. https://doi.org/10.1186/s12909-021-02653-4

17.

Dong

Swygert

K. A.

Durning

S. J.

Saguil

Gilliland

W. R.

Cruess

, et al (2014). Validity evidence for medical school OSCEs: Associations with USMLE® step assessments. Teaching and Learning in Medicine, 26(4), 379–386. https://doi.org/10.1080/10401334.2014.960294

18.

Dong

Zahn

Saguil

Swygert

K. A.

Yoon

Servey

, et al (2017). The Associations between clerkship objective structured clinical examination (OSCE) Grades and subsequent performance. Teaching and Learning in Medicine, 29(3), 280–285. https://doi.org/10.1080/10401334.2017.1279057

19.

Eftekhar

Labaf

Anvari

Jamali

Sheybaee-Moghaddam

(2012). Association of the pre-internship objective structured clinical examination in final year medical students with comprehensive written examinations. Medical Education Online, 17(1), 15958. https://doi.org/10.3402/meo.v17i0.15958

20.

Farrell

S. E.

Junkin

A. R.

Hayden

E. M.

(2022). Assessing clinical skills via Telehealth objective standardized clinical examination: Feasibility, Acceptability, comparability, and educational value. Telemedicine Journal and E-Health: The Official Journal of the American Telemedicine Association, 28(2), 248–257. https://doi.org/10.1089/tmj.2021.0094

21.

General Medical Council (2017). Generic professional capabilities framework. https://www.gmc-uk.org/education/standards-guidance-and-curricula/standards-and-outcomes/generic-professional-capabilities-framework

22.

General Medical Council (2023). What is the PLAB 2 exam? https://www.gmc-uk.org/registration-and-licensing/join-the-register/plab/plab-2-guide/what-is-the-plab-2-exam

23.

Gillette

Stanton

R. B.

Anderson

H. G.

(2017). Student performance on a knowledge-based exam may predict student ability to communicate effectively with a standardized patient during an objective structured clinical examination. Currents in Pharmacy Teaching and Learning, 9(2), 201–207. https://doi.org/10.1016/j.cptl.2016.11.004

24.

Gilson

G. J.

George

K. E.

Qualls

C. M.

Sarto

G. E.

Obenshain

S. S.

Boulet

(1998). Assessing clinical competence of medical students in women’s health care: Use of the objective structured clinical examination. Obstetrics and Gynecology, 92(6), 1038–1043. https://doi.org/10.1016/s0029-7844(98)00329-9

25.

Grand’Maison

Lescop

Rainsberry

Brailovsky

C. A.

(1992). Large-scale use of an objective, structured clinical examination for licensing family physicians. Canadian Medical Association Journal, 146(10), 1735–1740. https://pubmed.ncbi.nlm.nih.gov/1596809/

26.

Guyatt

Rennie

Meade

M. O.

Cook

D. J.

(2015). Users’ Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice (3rd ed.). McGraw-Hill Education.

27.

Harden

R. M.

Stevenson

Downie

W. W.

Wilson

G. M.

(1975). Assessment of clinical competence using objective structured examination. British Medical Journal, 1(5955), 447–451. https://doi.org/10.1136/bmj.1.5955.447

28.

Holbrook

Liu

J. T.

Rieder

Gibson

Levine

Foster

, et al (2019). Prescribing competency assessment for Canadian medical students: A pilot evaluation. Canadian Medical Education Journal, 10(1), e103-e110. https://doi.org/10.36834/cmej.61646

29.

Hsia

S. L.

Zhou

Gruenberg

Trinh

T. D.

Assemi

(2021). Implementation and evaluation of a virtual objective structured clinical examination for pharmacy students. Journal of the American College of Clinical Pharmacy, 4(7), 837–848. https://doi.org/10.1002/jac5.1448

30.

Hull

A. L.

Hodder

Berger

Ginsberg

Lindheim

Quan

, et al (1995). Validity of three clinical performance assessments of internal medicine clerks. Academic Medicine: Journal of the Association of American Medical Colleges, 70(6), 517–522. https://doi.org/10.1097/00001888-199506000-00013

31.

Huwendiek

Reichert

Duncker

de Leng

B. A.

van der Vleuten

C. P. M.

Muijtjens

A. M. M.

, et al (2017). Electronic assessment of clinical reasoning in clerkships: A mixed-methods comparison of long-menu key-feature problems with context-rich single best answer questions. Medical Teacher, 39(5), 476–485. https://doi.org/10.1080/0142159X.2017.1297525

32.

Hytönen

Näpänkangas

Karaharju-Suvanto

Eväsoja

Kallio

Kokkari

, et al (2021). Modification of national OSCE due to COVID-19—Implementation and students’ feedback. European Journal of Dental Education: Official Journal of the Association for Dental Education in Europe, 25(4), 679–688. https://doi.org/10.1111/eje.12646

33.

Jameel

Noor

S. M.

Ayub

Ali

S. S.

Park

Y. S.

Tekian

(2015). Feasibility, relevance and effectiveness of teaching and assessment of ethical status and communication skills as attributes of professionalism. JPMA. The Journal of the Pakistan Medical Association, 65(7), 721–726. https://pubmed.ncbi.nlm.nih.gov/26160080/

34.

Jamil

A Wahab

Jamludin

N. A.

Ahmad Hisham

(2022). Development and Conduct of Tele-objective structured clinical examination (Tele-OSCE) to assess clinical pharmacy competencies during COVID-19 Emergency Remote teaching. Medical Science Educator, 32(2), 321–327. https://doi.org/10.1007/s40670-022-01527-8

35.

Jutant

E. M.

Sesé

Patout

Messika

Maître

Gille

, et al (2022). Objective structured clinical examinations (OSCEs) for students’ training and assessment in the French respiratory medicine departments in 2021: An overview. Respiratory Medicine and Research, 81, 100883. https://doi.org/10.1016/j.resmer.2022.100883

36.

Kelly

Leung

Lindstrom

Wunder

J. C.

(2022). Virtual objective structured clinical examination Experiences and performance in physical medicine and Rehabilitation residency. American Journal of Physical Medicine and Rehabilitation, 101(10), 947–953. https://doi.org/10.1097/PHM.0000000000001942

37.

Kelly

M. E.

Regan

Dunne

Henn

Newell

O’Flynn

(2013). To what extent does the health professions Admission test-Ireland predict performance in early undergraduate tests of communication and clinical skills? An observational cohort study. BMC Medical Education, 13, 68. https://doi.org/10.1186/1472-6920-13-68

38.

Kupfer

(2020. June 22). Med school final exam plagued with technical issues after moving online due to COVID-19. CBC https://www.cbc.ca/news/canada/ottawa/medical-mcc-exam-technical-issues-1.5619168

39.

Lara

Foster

C. W.

Hawks

Montgomery

(2020). Remote assessment of clinical skills during COVID-19: A virtual, high-Stakes, summative Pediatric objective structured clinical examination. Academic Pediatrics, 20(6), 760–761. https://doi.org/10.1016/j.acap.2020.05.029

40.

Liu

Wong

Foster

Holbrook

(2018). Prescribing competency of medical students: National Survey of medical education Leaders. Journal of Population Therapeutics and Clinical Pharmacology, 25(1), e18-e24. https://doi.org/10.22374/1710-6222.25.1.2

41.

Luke

Petitt

Tombrella

McGoff

(2021). Virtual evaluation of clinical competence in Nurse Practitioner students. Medical Science Educator, 31(4), 1267–1271. https://doi.org/10.1007/s40670-021-01312-z

42.

Martinez

Holley

Brown

Abid

(2020). Addressing the rapidly Increasing need for Telemedicine education for future physicians. PRiMER, 4, 16. https://doi.org/10.22454/PRiMER.2020.275245

43.

Mazor

K. M.

Ockene

J. K.

Rogers

H. J.

Carlin

M. M.

Quirk

M. E.

(2005). The relationship between checklist scores on a communication OSCE and analogue patients’ perceptions of communication. Advances in Health Sciences Education: Theory and Practice, 10(1), 37–51. https://doi.org/10.1007/s10459-004-1790-2

44.

Medical Board of Australia . (2021, April 20). Good medical practice: A code of conduct for doctors in Australia. https://www.medicalboard.gov.au/Codes-Guidelines-Policies/Code-of-conduct.aspx

45.

Medical Council of Canada (2023). NAC Overview. https://mcc.ca/examinations/nac-overview/

46.

Nackman

G. B.

Griggs

Galt

(2006). Implementation of a novel web-based objective structured clinical evaluation. Surgery, 140(2), 206–211. https://doi.org/10.1016/j.surg.2006.05.004

47.

Novack

D. H.

Cohen

Peitzman

S. J.

Beadenkopf

Gracely

Morris

(2002). A pilot test of WebOSCE: A system for assessing trainees’ clinical skills via teleconference. Medical Teacher, 24(5), 483–487. https://doi.org/10.1080/0142159021000012504

48.

Nuovo

Bertakis

K. D.

Azari

(2006). Assessing resident’s knowledge and communication skills using four different evaluation tools. Medical Education, 40(7), 630–636. https://doi.org/10.1111/j.1365-2929.2006.02506.x

49.

Nusanti

DearainiBani

A. P.

Bani

A. P.

Kartasasmita

A. S.

Muhammad Ichsan

Virgana

, et al (2021). Delivering a modified continuous objective structured clinical examination for ophthalmology residents through a hybrid online method. Korean Journal of Medical Education, 33(4), 419–430. https://doi.org/10.3946/kjme.2021.210

50.

Oliven

Nave

Gilad

Barch

(2011). Implementation of a web-based interactive virtual patient case simulation as a training and assessment tool for medical students. Studies in Health Technology and Informatics, 169, 233–237. https://doi.org/10.3233/978-1-60750-806-9-233

51.

Page

M. J.

McKenzie

J. E.

Bossuyt

P. M.

Boutron

Hoffmann

T. C.

Mulrow

C. D.

, et al (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ (Clinical Research ed.), 372, n71. https://doi.org/10.1136/bmj.n71

52.

Patrício

Julião

Fareleira

Young

Norman

Vaz Carneiro

(2009). A comprehensive checklist for reporting the use of OSCEs. Medical Teacher, 31(2), 112–124. https://doi.org/10.1080/01421590802578277

53.

Peters

M. D. J.

Godfrey

C. M.

Khalil

McInerney

Parker

Soares

C. B.

(2015). Guidance for conducting systematic scoping reviews. International Journal of Evidence-Based Healthcare, 13(3), 141–146. https://doi.org/10.1097/XEB.0000000000000050

54.

Pugh

Hamstra

S. J.

Wood

T. J.

Humphrey-Murto

Touchie

Yudkowsky

, et al (2015). A procedural skills OSCE: Assessing technical and non-technical skills of internal medicine residents. Advances in Health Sciences Education: Theory and Practice, 20(1), 85–100. https://doi.org/10.1007/s10459-014-9512-x

55.

Qayyum

Wright

Lee

Leung

Sada

Holbrook

(2012). Medical student opinions on their training in clinical pharmacology and therapeutics. McMaster University Medical Journal, 9(1), 4–8. https://www.researchgate.net/publication/260158131_Qayyum_F_Wright_M_Lee_M_Leung_C_Sada_A_Holbrook_A_Medical_student_opinions_on_their_training_in_clinical_pharmacology_and_therapeutics_McMaster_University_Medical_Journal_2012914-8

56.

Rice

D. B.

Kloda

L. A.

Levis

Kingsland

Thombs

B. D.

(2016). Are MEDLINE searches sufficient for systematic reviews and meta-analyses of the diagnostic accuracy of depression screening tools? A review of meta-analyses. Journal of Psychosomatic Research, 87, 7–13. https://doi.org/10.1016/j.jpsychores.2016.06.002

57.

Royal College of Physicians and Surgeons of Canada (2015). CanMEDS: Better standards, better physicians, better care. https://www.royalcollege.ca/rcsite/canmeds/canmeds-framework-e

58.

Rushforth

H. E.

(2007). Objective structured clinical examination (OSCE): Review of literature and implications for nursing education. Nurse Education Today, 27(5), 481–490. https://doi.org/10.1016/j.nedt.2006.08.009

59.

Sam

A. H.

Hameed

Harris

Meeran

(2016). Validity of very short answer versus single best answer questions for undergraduate assessment. BMC Medical Education, 16(1), 266. https://doi.org/10.1186/s12909-016-0793-z

60.

Schleicher

Leitner

Juenger

Moeltner

Ruesseler

Bender

, et al (2017). Does quantity ensure quality? Standardized OSCE-stations for outcome-oriented evaluation of practical skills at different medical faculties. Annals of Anatomy, 212, 55–60. https://doi.org/10.1016/j.aanat.2017.03.006

61.

Schoeman

Chandratilake

(2012). The anatomy competence score: A new marker for anatomical ability. Anatomical Sciences Education, 5(1), 33–40. https://doi.org/10.1002/ase.263

62.

Schwartz

R. W.

Donnelly

M. B.

Sloan

D. A.

Johnson

S. B.

Strodel

W. E.

(1995). The relationship between faculty ward evaluations, OSCE, and ABSITE as measures of surgical intern performance. American Journal of Surgery, 169(4), 414–417. https://doi.org/10.1016/s0002-9610(99)80187-1

63.

Scoular

Huntsberry

Patel

Wettergreen

Brunner

J. M.

(2021). Transitioning competency-based communication assessments to the online platform: Examples and student outcomes. Pharmacy, 9(1), 52. https://doi.org/10.3390/pharmacy9010052

64.

Sihombing

A. T.

Taher

Rodjani

Mochtar

C. A.

Hakim

Daryanto

, et al (2021). Assessing the online objective structured clinical examinations in urology qualifying exam for urology residents in Indonesia during COVID-19 time. MethodsX, 8, 101316. https://doi.org/10.1016/j.mex.2021.101316

65.

Simon

S. R.

Bui

Day

Berti

Volkan

(2007). The relationship between second-year medical students’ OSCE scores and USMLE Step 2 scores. Journal of Evaluation in Clinical Practice, 13(6), 901–905. https://doi.org/10.1111/j.1365-2753.2006.00768.x

66.

Simon

S. R.

Volkan

Hamann

Duffey

Fletcher

S. W.

(2002). The relationship between second-year medical students’ OSCE scores and USMLE Step 1 scores. Medical Teacher, 24(5), 535–539. https://doi.org/10.1080/0142159021000012586

67.

Sudan

Clark

Henry

(2015). Cost and logistics for implementing the American College of Surgeons objective structured clinical examination. American Journal of Surgery, 209(1), 140–144. https://doi.org/10.1016/j.amjsurg.2014.10.001

68.

Tamblyn

Abrahamowicz

Dauphinee

W. D.

Hanley

J. A.

Norcini

Girard

, et al (2002). Association between licensure examination scores and practice in primary care. Journal of the American Medical Association, 288(23), 3019–3026. https://doi.org/10.1001/jama.288.23.3019

69.

Tricco

A. C.

Lillie

Zarin

O’Brien

K. K.

Colquhoun

Levac

, et al (2018). PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and Explanation. Annals of Internal Medicine, 169(7), 467–473. https://doi.org/10.7326/M18-0850

70.

Turner

J. L.

Dankoski

M. E.

(2008). Objective structured clinical exams: A Critical review. Family Medicine, 40(8), 574–578. https://pubmed.ncbi.nlm.nih.gov/18988044/

71.

van der Vleuten

C. P. M.

Swanson

D. B.

(1990). Assessment of clinical skills with standardized patients: State of the art. Teaching and Learning in Medicine, 2(2), 58–76. https://doi.org/10.1080/10401339009539432

72.

van Enst

W. A.

Scholten

R. J. P. M.

Whiting

Zwinderman

A. H.

Hooft

(2014). Meta-epidemiologic analysis indicates that MEDLINE searches are sufficient for diagnostic test accuracy systematic reviews. Journal of Clinical Epidemiology, 67(11), 1192–1199. https://doi.org/10.1016/j.jclinepi.2014.05.008

73.

Walsh

Bailey

P. H.

Koren

(2009). Objective structured clinical evaluation of clinical competence: An integrative review. Journal of Advanced Nursing, 65(8), 1584–1595. https://doi.org/10.1111/j.1365-2648.2009.05054.x

74.

Wenghofer

Klass

Abrahamowicz

Dauphinee

Jacques

Smee

, et al (2009). Doctor scores on national qualifying examinations predict quality of care in future practice. Medical Education, 43(12), 1166–1173. https://doi.org/10.1111/j.1365-2923.2009.03534.x

75.

Chan

Maxwell

S. R.

Levine

M. A.

Perri

Sebalt

R. J.

, et al (2015). Development and validation of the McMaster prescribing competency assessment for medical trainees (MacPCA). Journal of Population Therapeutics and Clinical Pharmacology, 22(2), e173-e178. https://jptcp.com/index.php/jptcp/article/view/253

76.

Zayyan

(2011). Objective structured clinical examination: The assessment of choice. Oman Medical Journal, 26(4), 219–222. https://doi.org/10.5001/omj.2011.55

77.

Zhang

X. Y.

Holbrook

A. M.

Nguyen

Lee

Al Qahtani

Garcia

M. C.

, et al (2019). Evaluation of online clinical pharmacology curriculum resources for medical students. British Journal of Clinical Pharmacology, 85(11), 2599–2604. https://doi.org/10.1111/bcp.14085

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.16 MB

0.12 MB