A Systematic Review on the Effectiveness of Systems-Based Practice Curricula in Health Professions Education

Abstract

This systematic review aims to evaluate the effectiveness of systems-based practice (SBP) curricula from the perspective of health professions students and workers. A total of 8468 citations were sourced from six electronic databases and manual searches conducted independently by two researchers, of which 44 studies were eventually included. A meta-analysis using a random effects model and a meta-synthesis using the thematic synthesis approach were conducted. Most studies targeted medical students, residents, and resident physicians from various clinical specialties. Almost half of all studies focused on didactic or knowledge-based interventions to teach SBP. About a third of all studies measured non-self-evaluated knowledge change, clinical abilities, and clinical outcomes. Both meta-analysis and meta-synthesis results revealed positive outcomes of increased knowledge of SBP, increased recognition of SBP as a core competency in one’s profession, and increased application of SBP knowledge in one’s profession. Meta-synthesis results also revealed negative outcomes at the institutional and teacher/health professions level. This review highlights the importance of SBP education and supports the effectiveness of SBP curricula. There is a need to address the negative outcomes at the institutional and teacher/health professions level. Moreover, future studies could investigate the integration of self-assessment outcomes with comparison to some external standard.

Keywords

systems-based practice systems thinking health professions systematic review education evaluation

Introduction

Systems thinking is the ability to recognise, understand, and synthesise the interactions and interdependencies in a set of components designed for a specific purpose (Dolansky & Moore, 2013). It is an important competency for health professionals to acquire. For example, the ability to engage in systems thinking is critical to systems-based practice, one of six core competencies specified by the Accreditation Council of Graduate Medical Education (ACGME) for physicians in training (Colbert et al., 2011), as well as a core competence described in the Future of Nursing Report for nurses (Dolansky et al., 2020). Systems-based practice refers to a healthcare professional’s awareness of and responsiveness to the larger context and system of healthcare, as well as the ability to call effectively on other resources in the system to provide optimal healthcare. In 2001, the ACGME started requiring medical education programmes in the United States of America (USA) to incorporate the SBP competency into their curricula. Since then, other health professions such as nursing and public health have also followed suit (Colbert et al., 2011). The core domains of a SBP curricula for health professions typically includes health care structures and processes, healthcare policy, economics, and management, clinical informatics and health information technology, population and public health, value-based care, and health system improvement (Gonzalo et al., 2017a).

Despite the importance of SBP as a core competency in the health professions education, there are a few gaps in literature. First, to the best of our knowledge, there is no systematic review pertaining to the evaluation of the effectiveness of SBP curricula from the perspective of health professions students or workers. Although there are recent reviews on systems thinking in healthcare, these focused exclusively on the basic definition and the practice of SBP in healthcare institutions or public health workforce settings (Carey et al., 2015; Rusoja et al., 2018), but do not describe how SBP is taught or assessed as a competency in health professions education. Second, there is no consensus to-date on the best practices for teaching and learning SBP among healthcare professionals. Fundamental questions, such as how and when to best teach SBP, remain unanswered. For example, some have suggested that SBP requires classroom and experiential learning as well as incorporation of reflective practice and authentic experiences. In addition, advocates believe that instruction should begin at the premedical or prelicensure level and continue through residency, or even into practice through continuing professional education (Phillips & Stalter, 2016). Third, promoting SBP in the health professions education is challenging. To many, SBP is often perceived to be lacking, limited in scope, and laborious to practice (Gonzalo et al., 2018). For example, medical students were generally less likely to welcome the integration of topics like SBP, which were often believed as less relevant to licensing examinations or success in residency match, or less beneficial for their future career (Gonzalo et al., 2016). This poses tremendous challenges in teaching and learning SBP, which could consequentially limit the potential for interdisciplinary collaboration among healthcare professionals. The aim of this systematic review is thus to evaluate the effectiveness of SBP curricula from the perspective of health professions students and workers.

Methods

This systematic review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement (Page et al., 2021). The review protocol was prospectively registered with the International Prospective Register of Systematic Reviews, PROSPERO (registration ID CRDCRD42021295438).

Literature Search

A combination of computerised and manual searches was performed to identify all relevant data for the review. The first two authors searched the following six electronic databases independently using the date ranges January 1, 2001, to December 31, 2021: Education Resources Information Centre (ERIC), ProQuest Central, PubMed, Scopus, Web of Science, and EBSCOhost. In addition, the authors manually searched the reference lists of all studies included in this review to identify additional relevant studies independently. The electronic database searches were conducted using abstract and title terms. The following were entered for searches in the six electronic databases: (“teaching” OR “learning”) AND (“systems thinking” OR “systematic thinking” OR “whole systems approach” OR “systems-based practice”) AND (“public health” OR “health care” OR “healthcare” OR “allied health” OR “health care structures” OR “health care processes” OR “heath care policy” OR “health care economics” OR “health care management” OR “clinical informatics” OR “health information technology” OR “population health” OR “value-based care” OR “health system improvement”) AND (“higher education” OR “tertiary education”). No language or publication status restrictions were specified.

Inclusion and Exclusion Criteria

To be included in the review, a study had to meet the following criteria: (1) the learners were health professions students or workers from universities, professional schools (e.g., nursing, medical, pharmacy schools) and graduate education programmes (e.g., residency and fellowship programmes) across the continuum of health professions (e.g., medicine, dentistry, nursing, pharmacy, allied health, and public health); (2) the concept of SBP or systems thinking was taught; and (3) the outcomes of SBP or systems thinking curricula were evaluated from the perspective of health professions students or workers. There was no restriction on the types of pedagogical activities used, and the evaluation could be carried out using quantitative, qualitative, or mixed methods approaches. Studies without any evaluation results were excluded.

Study Selection and Data Extraction

The first two authors, who had prior experience in conducting reviews, carried out the study selection and data extraction independently. Citations from the databases and reference lists were merged and duplicates were removed. A two-stage screening process was adopted – the first screening involved reviewing the titles and abstracts using the pre-specified inclusion and exclusion criteria, and if either of the authors evaluated the title or abstract to be relevant to the review, the second screening involved reading the full texts of the articles. Both authors reviewed all the titles, abstracts, and full texts of these identified articles. There was a good interrater reliability of Cohen’s kappa coefficient of 0.84 for the inclusion/exclusion of studies.

The subsequent data extraction focused on study design, setting, type of student or worker, sampling/assignment technique and sample size, outcome measurements, method of data collection, evaluation results, and study limitations. There was a good inter-rater reliability of Cohen’s kappa coefficient of 0.79 for the extracted data from the included studies. Any discrepancies in eligibility assessment or data extraction were resolved through discussion between both authors.

Quality Assessment of Included Studies

Methodological quality of the included studies was independently evaluated by the first two authors using the Mixed Methods Appraisal Tool, MMAT (Hong et al., 2018) as the review contained qualitative, quantitative, and mixed methods studies. The authors applied two screening questions, which yielded a ‘yes’ for all studies. Next, for each study, the authors determined the type of design before using the corresponding criteria to appraise the study’s quality. For example, if the study was qualitative, the five criteria in the qualitative category was used for appraisal. For each criterion, there were the responses of ‘yes,’ ‘no’ or ‘can’t tell.’ The authors referred to the explanation and algorithm for each criterion to come to a response. The ‘can’t tell’ response meant that the study did not report appropriate information for a ‘yes’ or ‘no’ response, or reported unclear information related to the criterion. There was a good interrater reliability of Cohen’s kappa coefficient of 0.85. Any discrepancies were resolved through discussion to reach a consensus between both authors. An overall score for each study was not obtained since this was discouraged in the latest version of the MMAT (Hong et al., 2018).

Meta-Analysis Procedure

Most of the outcomes pertaining to the evaluation of SBP curricula in this review were too heterogenous to be combined. There were only three similar self-assessment outcomes that could be pooled for meta-analysis – understanding of SBP, application of SBP knowledge in one’s profession, and recognition of SBP as a core competency in one’s profession. A total of 15 studies were used for the meta-analysis.

For the first outcome on understanding of SBP, this was determined using the systems thinking score based on the self-assessed System Thinking Scale, STS (Dolansky et al., 2020). The systems thinking score was a composite score that ranged from 0 to 80, measured using the 20-item STS on a 0 to 4 Likert scale, with a higher score showing better understanding of SBP (Dolansky et al., 2020). Seven out of the 44 included studies used the STS (Aboumatar et al., 2012; Fura & Wisser, 2017; Gaupp et al., 2016; Gonzalo et al., 2017b; Könings et al., 2018; Sanko et al., 2020; Thornton Bacon et al., 2018). Out of these seven studies, four could be pooled for meta-analysis. Two studies that measured systems thinking using a one-item question instead of the 20-item STS (Gonzalo et al., 2017b; Könings et al., 2018) as well as one study that did not publish the descriptive summary of the STS score (Thornton Bacon et al., 2018) were excluded.

For the second outcome regarding the application of SBP, respondents were asked to self-assess the extent to which they agreed with the statement about applying their SBP knowledge in their profession post-intervention. This was measured on a 1 to 5 Likert scale with the responses of strongly disagree, disagree, neutral, agree, and strongly agree – the higher the self-rated score, the better the perceived application. Eleven studies could be pooled on this outcome.

For the third outcome regarding the recognition of SBP, respondents were asked to self-assess the extent to which they agreed with the statement about recognising SBP as a core competency in their profession post-intervention. This was measured on a 1 to 5 Likert scale with the responses of strongly disagree, disagree, neutral, agree, and strongly agree – the higher the self-rated score, the better the perceived recognition. Four studies could be pooled on this outcome.

The Inverse Variance method was used to pool the overall mean values of these outcomes across studies (Fleiss, 1993). Heterogeneity between the studies and sampling variance within the studies were assessed using the estimate of the absolute total observed variance T² and proportion of the total observed variance from heterogeneity I² (Higgins et al., 2003; Cheung, 2019). The low T² and moderate to high I² of the three outcomes showed that the total observed variance was low and there was more heterogeneity than sampling error from these studies. Hence, the random effects model with restricted maximum likelihood method was used for all the three outcomes. The Egger test and funnel plot were used to detect publication bias. All meta-analyses were conducted using STATA SE 16 and the results were presented using forest plots.

Meta-Synthesis Procedure

In addition, the first two authors conducted a meta-synthesis on the outcomes of SBP curricula. A thematic synthesis approach was used to gather information and identify all themes. The authors adapted the inductive analysis which involved three stages (Sandelowski & Barroso, 2006): (1) extraction and coding of findings for each article; (2) grouping of findings (codes) according to their topical similarity to determine whether findings confirm, extend, or refute each other; and (3) abstraction of findings (analysing the grouped codes to identify additional patterns, overlaps, comparisons, and redundancies to form a set of concise statements that capture the content of the findings). All data under the headings “Results”, “Discussion” and “Conclusions” were read several times, and line by line. Relevant quotes were extracted, and these were analysed and organised into codes and groupings. We used the process of constant comparative analysis. Emerging groupings from early codes were checked with ongoing coding and used to guide later coding. Final groupings were reviewed to ensure codes were similar in all groups and that no potential groupings were missed during the process (Strauss & Corbin, 1994). Cohen’s kappa coefficient yielded a good agreement of 0.82. Any disagreement was resolved by consensus between both authors.

Results

Study Selection Process Results

Our database and manual searches identified 8468 citations, of which 44 studies were included in our systematic review (Figure 1). Of these, 31 reported quantitative results, four reported qualitative results, and nine reported mixed methods results. Fifteen studies were used for meta-analysis, and 13 for meta-synthesis.

Figure 1.

Flow Chart of Identification and Selection of Studies for the Systematic Review.

Descriptive Findings

Supplementary Table 1 shows the characteristics of the included studies. Twenty-three studies (52.3%) targeted residents and resident physicians from various clinical specialities such as internal medicine, general surgery, paediatrics, anaesthesiology, radiology, orthopaedics, psychiatry, emergency medicine, ophthalmology, otolaryngology, obstetrics, and gynaecology. This was followed by 12 studies (27.3%) which focused on medical students. The other seven studies (15.9%) targeted students and workers from other health professions such as nursing, public health, nutrition, and dietetics. The remaining two studies (4.5%) focused on students and workers from various health professions. Of the 44 studies, close to half (19 studies, 43.2%) consisted of mainly didactic or knowledge-based interventions to teach SBP. The rest comprised experiential learning activities, role playing exercises, case-based or problem-based learning, clinical projects, simulation training, collaborative learning activities, service-learning activities, site visits and debates, and game-based learning activities.

Supplementary Table 2 shows the core domains of SBP covered in the curricula and the sub-competencies evaluated according to the classification by ACGME. Almost all studies covered multiple domains and evaluated multiple sub-competencies. The top three domains covered were health care structure and processes (n = 37, 84.1%), followed by health system improvement (n = 25, 56.8%) and then health care policy, economics, and management (n = 23, 52.3%). The top three sub-competencies evaluated were coordination of patient care within the health care system relevant to their clinical specialty (n = 35, 79.5%), followed by working effectively in various health care delivery settings and systems relevant to their clinical specialty (n = 28, 63.6%), and working in interprofessional teams to enhance patient safety and improve patient care quality (n = 26, 59.1%).

All included studies utilised self-evaluated measurements or self-reports to assess changes in knowledge, attitudes, beliefs, and perceptions pertaining to SBP. Supplementary Table 3 shows a summary of the included studies on measurements related to non-self-evaluated knowledge change, clinical abilities, and clinical outcomes. Only about a third measured non-self-evaluated knowledge change, clinical abilities, and clinical outcomes. Of the 44 included studies, 12 measured non-self-evaluated knowledge change pertaining to patient safety, healthcare finance, healthcare system, healthcare policy etc. All 12 studies reported a positive change in knowledge level. In addition, two other studies reported an improvement in non-self-evaluated clinical abilities pertaining to navigating the clinic’s electronic medical record system as well as designing an anaesthetic for all cases within the pre-set price limit. Both studies reported an improvement in clinical abilities assessed by the faculty team. Finally, only one study reported an improvement in clinical outcome pertaining to a reduction in the mean door-to-intravenous tissue plasminogen activator time by 9.64 minutes for acute ischaemic stroke treatment after implementation of a simulation-based training programme.

Quality Appraisal Results

The quality appraisal of the included studies is presented in Supplementary Table 4. Twenty-three of the 44 studies fulfilled at least three out of five criteria outlined by the MMAT for each study design.

Meta-Analysis Findings

The meta-analysis of the three outcomes post-intervention is shown in Figure 2. The higher the points reported, the more favourable was the outcome. The pooled mean of the score on understanding of SBP improved by 3.80 (95% CI 1.57–6.04) post-intervention. The other two outcomes had a pooled mean of >3.0 indicating that health professions students and workers generally perceived an improvement for these outcomes post-learning. The pooled mean of the score on application of SBP knowledge in one’s profession was 3.74 (95% CI 3.30–4.19) out of the highest score of 5 post-intervention. The pooled mean of the score on recognition of SBP as a core competency in one’s profession was 4.01 (95% CI 3.81–4.20). While no publication bias was detected for the outcomes on application of SBP knowledge in one’s profession and recognition of SBP as a core competency in one’s profession, we could not rule this out for the score on understanding of SBP.

Figure 2.

Forest Plots Illustrating the Pooled (A) Mean Difference in Understanding of Systems-Based Practice Pre- and Post-Intervention, (B) Mean Score on Application of Systems-Based Practice Knowledge in One’s Profession Post-Intervention, and (C) Mean Score on Recognition of Systems-Based Practice as a Core Competency in One’s Profession Post-Intervention.

Meta-Synthesis Findings

Table 1 shows the themes, subthemes, and sub-subthemes on the outcomes of SBP curricula. Qualitative results were split into two overarching themes: positive and negative outcomes, each with a myriad of subthemes and sub-subthemes. Supplementary Table 5 shows the corresponding illustrative quotes.

Table 1.

Themes, Subthemes, and Sub Subthemes on the Outcomes of Systems-Based Practice Curricula in Health Professions Education.

			Ackerman et al., 2016	Castro et al., 2021	Gheihman et al., 2020	Gonzalo et al., 2017b	Gonzalo et al., 2018	Gonzalo et al., 2019	Iyer et al., 2021	Martinez et al., 2013	Mills et al., 2017	Nagler et al., 2010	Perez et al., 2009	Sheu et al., 2013	Zenni et al., 2006
Themes	Subtheme	Sub Subtheme	Ackerman et al., 2016	Castro et al., 2021	Gheihman et al., 2020	Gonzalo et al., 2017b	Gonzalo et al., 2018	Gonzalo et al., 2019	Iyer et al., 2021	Martinez et al., 2013	Mills et al., 2017	Nagler et al., 2010	Perez et al., 2009	Sheu et al., 2013	Zenni et al., 2006
Positive outcomes	Improved systems-based practice competency	Increased recognition of systems-based practice as a core competency in one’s profession (8)			✓				✓	✓	✓	✓	✓	✓	✓
		Better understanding of the problems patients faced in navigating the healthcare system (5)		✓		✓	✓							✓	✓
		Better appreciation of the importance of systems-based practice in healthcare (4)						✓	✓		✓	✓
		Increased awareness of healthcare as a complex system (4)	✓	✓					✓					✓
		Increased knowledge of systems-based practice (3)					✓		✓				✓
		Increased appreciation of the social determinants of health in influencing health outcomes (3)		✓			✓								✓
		Increased application of systems-based practice knowledge in one’s profession (3)					✓		✓				✓
	Improved communication with patients	Increased empathy for patients (4)		✓		✓	✓								✓
	Improved communication with patients	Improved patient counselling skills (3)	✓	✓			✓
	Improved interprofessional collaboration and teamwork	Better understanding of the need for an interdisciplinary approach in healthcare (4)		✓			✓					✓		✓
		Increased appreciation of the contributions of non-physician staff and the difficulties they faced (3)	✓	✓			✓
		Increased opportunities for interprofessional collaborative practice (2)			✓							✓
	Improved professionalism	Increased professional identity (3)	✓					✓					✓
Negative outcomes	Institutional level	Perception that systems-based practice had ‘eaten’ into the time for more important curriculum or in one’s practice (3)			✓			✓			✓
		Perception that systems-based practice was disconnected from the rest of the curriculum or in one’s practice (3)			✓			✓			✓
		Perception that systems-based practice was not introduced early enough in the curriculum or in one’s practice (1)			✓
	Teacher/health professions level	Perception that not all teachers or health professions were interested in systems-based practice (1)						✓
	Teacher/health professions level	Perception that not all teachers or health professions were experienced to teach systems-based practice (1)						✓

Note. Number inside the bracket indicates the number of studies with the subtheme.

Positive Outcomes

Thirteen studies described positive outcomes. There were four subthemes: (1) improved SBP competency; (2) improved communication with patients; (3) improved interprofessional collaboration and teamwork, and (4) improved professionalism.

Under the subtheme of improved SBP competency, there were three sub-subthemes which were similar to the meta-analysis findings: (1) increased recognition of SBP as a core competency in one’s profession, “I was embarrassed I didn’t know this information [health policy] prior to hearing these lectures. This should be a standard part of our training and is an important core competency to acquire.” (Nagler et al., 2010); (2) increased application of SBP knowledge in one’s profession, “It was a useful experience. It gave us a good application as to how a private practice works and the different aspects to be considered while setting up and running a practice. The examples used were useful for my application.” (Perez et al., 2009); and (3) increased knowledge of SBP, “I didn’t know a whole lot about what went into (disaster resilience and response) - we had 1 day in med school - I didn’t really know who all was in charge of responding to disasters - how they plan ahead of time - I definitely think I know a lot more now than I did before.” (Iyer et al., 2021).

Negative Outcomes

Three studies described negative outcomes. There were two subthemes, namely institutional level, and teacher/health professions level. Under the subtheme of institutional level, there were three sub-subthemes: (1) perception that SBP had ‘eaten’ into the time for more important curriculum or in one’s practice; (2) perception that SBP was disconnected from the rest of the curriculum or in one’s practice; and (3) perception that SBP was not introduced early enough in the curriculum or in one’s practice. Under the subtheme of teacher/health professions level, there were two sub-subthemes: (1) perception that not all teachers or health professions were interested in SBP; and (2) perception that not all teachers or health professions were experienced to teach SBP.

Discussion

This was the first review to evaluate the effectiveness of SBP curricula in health professions education. Most studies targeted medical students, residents, and resident physicians from various clinical specialties. Close to half of the studies consisted of mainly didactic or knowledge-based interventions to teach SBP. The top SBP core domain covered was health care structure and processes, and the top sub-competency evaluated was coordination of patient care within the health care system relevant to their clinical specialty. All studies utilised self-evaluated measurements or self-reports while only about a third measured non-self-evaluated knowledge change, clinical abilities, and clinical outcomes. Both meta-analysis and meta-synthesis results revealed positive outcomes of increased knowledge of SBP, increased recognition of SBP as a core competency in one’s profession, and increased application of SBP knowledge in one’s profession. Meta-synthesis results also revealed negative outcomes at the institutional and teacher/health professions level.

The meta-analysis and meta-synthesis conducted in this review provide evidence that SBP curricula have a positive impact on the understanding and recognition of SBP as a core competency in healthcare professions. Previous research has also reported that early implementation of education and training in SBP, such as during medical school, can improve knowledge and understanding of the importance of SBP in healthcare delivery (Stalter et al., 2017; Borkan et al., 2021). Our finding of increased knowledge of SBP is consistent with previous studies that have reported enhanced understanding of the importance of SBP in healthcare delivery with the incorporation of SBP curricula in health professions education (Stalter et al., 2017; Borkan et al., 2021). Moreover, the recognition of SBP as a core competency in one’s profession is crucial in ensuring that healthcare professionals recognise its importance in healthcare delivery. This outcome also acknowledges the efforts to increase emphasis on SBP education in healthcare training programmes.

The meta-synthesis results indicate that SBP curricula have several positive outcomes beyond the improvement of SBP competency. Improved communication with patients is one such positive outcome. Effective communication is essential for building trust and establishing a good rapport with patients. It also ensures that patients are well-informed and have a thorough understanding of their health status and available treatment options, which can lead to better treatment adherence (Kwame & Petrucka, 2021). Another positive outcome is improved interprofessional collaboration and teamwork. This is crucial for healthcare professionals to address complex patient needs to provide coordinated and quality care (Rosen et al., 2018). Furthermore, improved professionalism is another positive outcome. This is characterised by ethical and responsible behaviour, compassion, and respect for patients. It is associated with improved patient outcomes and satisfaction (Bekalu & Wudu, 2023).

Almost half of the studies focused on didactic or knowledge-based interventions to teach SBP, of which only about a third measured non-self-evaluated knowledge change, clinical abilities, and clinical outcomes. Despite the meta-analysis and meta-synthesis illustrating positive outcomes, these were largely based on self-evaluation and self-reporting. Evidence from the medical literature suggests that the self-assessment of residents generally do not correlate well with their competency performance, often due to factors relating to learners’ personal backgrounds, cultures, the specific contexts of the learning environment and cognitive bias (Fleming et al., 2021; Gabbard & Romanelli, 2021). Such cognitive bias, termed as the Dunning-Kruger Effect, occurs when lower skilled individuals overestimate their abilities, while experts underestimate theirs (Athy et al., 2021). That said, self-assessment is critical for healthcare professionals to improve and adapt to the advances in their profession. Going forward, educators and researchers should avoid relying on self-assessment data alone, and future studies could investigate the integration of self-assessment data with some external standard, such as an expert rater, or the use of a validated instrument on SBP. The process of comparing self-assessments with external standards facilitates self-improvement, as the physician is made aware of any discordance between his/her self-assessment and an assessment based on credible data and established standards (Abadel & Hattab, 2013). Moreover, when designing future studies aimed at assessing the impact of educational interventions, educators and researchers should also include non-self-evaluated knowledge-centric data as one of the learning outcomes (Gabbard & Romanelli, 2021).

Furthermore, health professions educators, researchers in healthcare pedagogy or medical education curriculum, and programme developers should take note of the negative outcomes identified in this review, such as the perception that SBP had ‘eaten’ into the time for more important curriculum, SBP was disconnected from the rest of the curriculum or practice, and not all teachers/health professions were interested in SBP or were experienced to teach SBP. As such, more remains to be done to promote the importance of SBP in the health professions education. At the institutional level, SBP should be promoted as a norm and an integral part of health professions education. However, this is difficult to execute given that traditional healthcare training is often conducted in a reductionist approach – students are expected to understand the determinants of health and health behaviours as opposed to understanding system behaviours driven by various relationships within the healthcare system. Yet, the healthcare system is one that works best when an interconnectedness and collaborative effort of healthcare professionals is practiced within that system (Rosen et al., 2018). Therefore, it is imperative for health sciences universities and schools to integrate SBP into the health professions curriculum as well as allocate protected time for students to learn about SBP.

At the teacher/health professions level, teachers and healthcare professionals are the best agents to push for this change given this review’s findings. There are three aspects in which they could do so, namely by enhancing their pedagogical approach towards teaching SBP, enhancing their personal experience on SBP, as well as acting as advocates of SBP. Many educators in the health professions are expert clinicians, and some might have learnt SBP informally or on the job. Therefore, they would not be trained in the metacognitive processes or equipped with the analytical tools underlying SBP, making it challenging for them to break down SBP into the requisite knowledge and skills needed for pedagogical instruction (Plack et al., 2019). For example, many practicing physicians have confusion regarding how SBP could apply to their individual practices. In such cases, where the teachers themselves lacked real-life and ground experience in SBP, the quality of their teaching could be compromised (Batalden & Leach, 2009). Considering this, educational institutes should provide training and development activities for junior or inexperienced faculty. This would assist faculty in their teaching of SBP, and in their ability to provide guidance to students in reflecting on their own interactions within the overall healthcare system. Apart from this, institutions should also design and offer professional development activities to formally train faculty and licensed healthcare providers in the metacognitive processes and analytical tools underlying SBP, as well as provide strategies for teaching and assessing outcomes of SBP (Khanna et al., 2021).

There were several strengths and limitations in this current review. One strength was the moderate to high quality of the included studies, with over 50% of these studies assessed to be at low risk of bias for at least half of the inclusion criteria. Another strength was the use of both meta-analysis and meta-synthesis in a single review to achieve methodological triangulation of the results. Nevertheless, a few limitations must be highlighted. First, publication bias could not be excluded given the likelihood that interventions without significant or positive evaluation results were not published. Second, as this review focused largely on electronic databases, the evaluation of SBP curricula might not be reported in journals or be published. Finally, despite the meta-analysis and meta-synthesis results illustrating positive outcomes, these were largely based on self-evaluation and self-reported responses, which may not always correlate well with actual competency performance.

Conclusion

To conclude, this review highlights the importance of SBP education and supports the effectiveness of SBP curricula. There is a need to address the negative outcomes at the institutional and teacher/health professions level. Moreover, future studies could investigate the integration of self-assessment data with some external standard.

Supplemental Material

Supplemental Material - A Systematic Review on the Effectiveness of Systems-Based Practice Curricula in Health Professions Education

Supplemental Material for A Systematic Review on the Effectiveness of Systems-Based Practice Curricula in Health Professions Education by Raymond Boon Tar Lim, Kenneth Wee Beng Hoe, Claire Gek Ling Tan and Huili Zheng in Evaluation & the Health Professions

Supplemental Material

Supplemental Material - A Systematic Review on the Effectiveness of Systems-Based Practice Curricula in Health Professions Education

Supplemental Material

Supplemental Material - A Systematic Review on the Effectiveness of Systems-Based Practice Curricula in Health Professions Education

Supplemental Material

Supplemental Material - A Systematic Review on the Effectiveness of Systems-Based Practice Curricula in Health Professions Education

Supplemental Material

Supplemental Material - A Systematic Review on the Effectiveness of Systems-Based Practice Curricula in Health Professions Education

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Raymond Boon Tar Lim

Data Availability Statement

The data given this article are All data generated or analysed during this review are included in this published article and its supplementary files.

Supplemental Material

Supplemental material for this article is available online.

References

References marked with * are included in the systematic review.

Abadel

F. T.

Hattab

A. S.

(2013). How does the medical graduates’ self-assessment of their clinical competency differ from experts’ assessment? BMC Medical Education, 13(24), 1–9. https://doi.org/10.1186/1472-6920-13-24, Retrieved from https://link-springer-com-443.web.bisu.edu.cn/article/10.1186/1472-6920-13-24

* Aboumatar

H. J.

Thompson

Dawson

Colbert

Marsteller

Kent

Lubomski

L. H.

Paine

Pronovost

(2012). Development and evaluation of a 3-day patient safety curriculum to advance knowledge, self-efficacy and system thinking among medical students. BMJ Quality and Safety, 21(5), 416–422. https://dx-doi-org.web.bisu.edu.cn/10.1136/bmjqs-2011-000463

* Ackerman

S. L.

Boscardin

Karliner

Handley

M. A.

Cheng

Gaither

T. W.

Hagey

Hennein

Malik

Shaw

Trinidad

Zahner

Gonzales

(2016). The action research program: Experiential learning in systems-based practice for first-year medical students. Teaching and Learning in Medicine, 28(2), 183–191. https://doi.org/10.1080/10401334.2016.1146606

Athy

Talmon

Samson

Martin

Nelson

(2021). Faculty versus resident self-assessment using pathology milestones: How aligned are we? Academic Pathology, 8(1), 1–7. https://doi.org/10.1177/23742895211060526

Batalden

P. B.

Leach

D. C.

(2009). Sharpening the focus on systems-based practice. Journal of Graduate Medical Education, 1(1), 1–3. https://doi.org/10.4300/01.01.0001

Bekalu

Y. E.

Wudu

M. A.

(2023). Level of professionalism and associated factors among nurses working in South Wollo zone public hospitals, Northeastern Ethiopia, 2022, (9, pp. 1–8). Sage Open Nursing. https://doi.org/10.1177/23779608231158976

*Boltri

J. M.

Vogel

R. L.

Tindall

W. N.

Tedders

S. H.

(2002). Improving medical students’ knowledge of systems-based practice. Managed Care Quarterly, 10(1), 29–32. https://pubmed.ncbi.nlm.nih.gov/15991398/

Borkan

J. M.

Hammoud

M. M.

Nelson

Oyler

Lawson

Starr

S. R.

Gonzalo

J. D.

(2021). Health systems science education: The new post-Flexner professionalism for the 21st century. Medical Teacher, 43(sup2), S25–S31. https://doi.org/10.1080/0142159X.2021.1924366

10.

* Brandon

C. J.

Mullan

P. B.

(2013). Teaching medical management and operations engineering for systems-based practice to radiology residents. Academic Radiology, 20(3), 345–350. https://doi.org/10.1016/j.acra.2012.09.025

11.

Carey

Malbon

Carey

Joyce

Crammond

Carey

(2015). Systems science and systems thinking for public health: A systematic review of the field. BMJ Open, 5(12), 1–9. https://dx-doi-org.web.bisu.edu.cn/10.1136/bmjopen-2015-009002

12.

* Castro

M. R. H.

Calthorpe

L. M.

Fogh

S. E.

McAllister

Johnson

C. L.

Isaacs

E. D.

Ishizaki

Kozas

Rennke

Davis

Chang

(2021). Lessons from learners: Adapting medical student education during and post COVID-19. Academic Medicine, 96(12), 1671–1679. https://doi.org/10.1097/ACM.0000000000004148

13.

* Chan

E. Y.

Deziel

D. J.

Orkin

B. A.

Wool

N. L.

(2015). Systems-based practice: Learning the concepts using a teamwork competition model. American Journal of Surgery, 209(1), 40–44. https://doi.org/10.1016/j.amjsurg.2014.08.034

14.

Cheung

M. W. L.

(2019). A guide to conducting a meta-analysis with non-independent effect sizes. Neuropsychology Review, 29(4), 387–396. https://doi.org/10.1007/s11065-019-09415-6, Retrieved from https://link-springer-com-443.web.bisu.edu.cn/article/10.1007/s11065-019-09415-6

15.

Colbert

C. Y.

Ogden

P. E.

Ownby

A. R.

Bowe

(2011). Systems-based practice in graduate medical education: Systems thinking as the missing foundational construct. Teaching and Learning in Medicine, 23(2), 179–185. https://doi.org/10.1080/10401334.2011.561758

16.

*David

R. A.

Reich

L. M.

(2005). The creation and evaluation of a systems-based practice/managed care curriculum in a primary care internal medicine residency program. The Mount Sinai. Journal of Medicine, New York, 72(5), 296–299. https://pubmed.ncbi.nlm.nih.gov/16184291/

17.

Dolansky

M. A.

Moore

S. M.

(2013). Quality and Safety Education for Nurses (QSEN): The key is systems thinking. Online Journal of Issues in Nursing, 18(3), 1–12. https://doi.org/10.3912/OJIN.Vol18No03Man01

18.

Dolansky

M. A.

Moore

S. M.

Palmieri

P. A.

Singh

M. K.

(2020). Development and validation of the systems thinking scale. Journal of General Internal Medicine, 35(8), 2314–2320. https://doi.org/10.1007/s11606-020-05830-1, Retrieved from https://link-springer-com-443.web.bisu.edu.cn/article/10.1007/s11606-020-05830-1

19.

* Eiser

A. R.

Connaughton-Storey

(2008). Experiential learning of systems-based practice: A hands-on experience for first-year medical residents. Academic Medicine, 83(10), 916–923. https://doi.org/10.1097/ACM.0b013e31818509ca

20.

Fleiss

J. L.

(1993). Review papers: The statistical basis of meta-analysis. Statistical Methods in Medical Research, 2(2), 121–145. https://doi.org/10.1177/096228029300200202

21.

Fleming

Vautour

McMullen

Cofie

Dalgarno

Phelan

Mizubuti

G. B.

(2021). Examining the accuracy of residents' self-assessments and faculty assessment behaviours in anesthesiology. Canadian Medical Education Journal, 12(4), 17–26. https://doi.org/10.36834/cmej.70697

22.

* Fura

L. A.

Wisser

K. Z.

(2017). Development and evaluation of a systems thinking education strategy for baccalaureate nursing curriculum: A pilot study. Nursing Education Perspectives, 38(5), 270–271. https://doi.org/10.1097/01.NEP.0000000000000165

23.

Gabbard

Romanelli

(2021). The accuracy of health professions students’ self-assessments compared to objective measures of competence. American Journal of Pharmaceutical Education, 85(4), 275–280. https://doi.org/10.5688/ajpe8405

24.

* Gaupp

Körner

Fabry

(2016). Effects of a case-based interactive e-learning course on knowledge and attitudes about patient safety: A quasi-experimental study with third-year medical students. BMC Medical Education, 16(1), 1–8. https://doi.org/10.1186/s12909-016-0691-4

25.

* Gheihman

Forester

B. P.

Sharma

So-Armah

Wittels

K. A.

Milligan

T. A.

(2020). Introduction to quality improvement and systems-based practice: A two-part module for clinical clerkship students. Journal of General Internal Medicine, 35(12), 3742–3744. https://doi.org/10.1007/s11606-020-05788-0

26.

* Gillen

J. R.

Ramirez

A. G.

Farineau

D. W.

Hoke

T. R.

Schirmer

B. D.

Williams

M. D.

Lau

C. L.

(2016). Using interdisciplinary workgroups to educate surgery residents in systems-based practice. Journal of Surgical Education, 73(6), 1052–1059. https://doi.org/10.1016/j.jsurg.2016.05.017

27.

* Gonzalo

J. D.

Davis

Thompson

B. M.

Haidet

(2019). Unpacking medical students’ mixed engagement in health systems science education. Teaching and Learning in Medicine, 32(3), 250–258. https://doi.org/10.1080/10401334.2019.1704765

28.

Gonzalo

J. D.

Dekhtyar

Starr

S. R.

Borkan

Brunett

Fancher

Green

Grethlein

S. J.

Lai

Lawson

Monrad

O’Sullivan

Schwartz

M. D.

Skochelak

(2017a). Health systems science curricula in undergraduate medical education: Identifying and defining a potential curricular framework. Academic Medicine, 92(1), 123–131. https://doi.org/10.1097/ACM.0000000000001177

29.

*Gonzalo

J. D.

Graaf

Kass

L. E.

Promes

S. B.

Wolpaw

D. R.

George

D. R.

(2017b). Medical students as systems ethnographers: Exploring patient experiences and systems vulnerabilities in the emergency department. AEM Education and Training, 1(3), 225–233. https://doi.org/10.1002/aet2.10038.

30.

Gonzalo

J. D.

Haidet

Blatt

Wolpaw

D. R.

(2016). Exploring challenges in implementing a health systems science curriculum: A qualitative analysis of student perceptions. Medical Education, 50(5), 523–531. https://doi.org/10.1111/medu.12957

31.

*Gonzalo

J. D.

Wolpaw

Graaf

Thompson

B. M.

(2018). Educating patient-centered, systems-aware physicians: A qualitative analysis of medical student perceptions of value-added clinical systems learning roles. BMC Medical Education, 18(1), 248. https://doi.org/10.1186/s12909-018-1345-5

32.

* Hacken

Jarvis-Selinger

S. S.

Armstrong

A. D.

Black

Pratt

Black

K. P.

(2019). A systems-based practice curriculum in orthopaedics: Successes and challenges moving forward. The Journal of Bone and Joint Surgery, 101(1), 1–7. https://doi.org/10.2106/JBJS.17.00595

33.

Higgins

J. P. T.

Thompson

S. G.

Deeks

J. J.

Altman

D. G.

(2003). Measuring inconsistency in meta-analyses. British Medical Journal, 327(7414), 557–560. https://doi.org/10.1136/bmj.327.7414.557

34.

Hong

Q. N.

Fàbregues

Bartlett

Boardman

Cargo

Dagenais

Gagnon

M. P.

Griffiths

Nicolau

O'Cathain

Rousseau

M. C.

Vedel

Pluye

(2018). The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers. Education for Information, 34(4), 285–291. https://doi.org/10.3233/EFI-180221

35.

* Iyer

M. S.

C. B.

Scherzer

D. J.

MacDowell

Gupta

McManus

Stewart

Linakis

S. W.

Stanley

(2021). The COVID-19 elective for pediatric residents: Learning about systems-based practice during a pandemic. Cureus, 13(2), Article e13085. https://doi.org/10.7759/cureus.13085

36.

* Kerfoot

B. P.

Conlin

P. R.

Travison

McMahon

G. T.

(2007). Web-based education in systems-based practice: A randomized trial. Archives of Internal Medicine, 167(4), 361–366. https://doi.org/10.1001/archinte.167.4.361

37.

Khanna

Roberts

Lane

(2021). Designing health professional education curricula using systems thinking perspectives. BMC Medical Education, 21(1), 1–8. https://doi.org/10.1186/s12909-020-02442-5

38.

*Kim

Y. M.

Yoon

Y. S.

Hong

H. C.

Min

(2019). Effects of a patient safety course using a flipped classroom approach among undergraduate nursing students: A quasi-experimental study. Nurse Education Today, 79(1), 180–187. https://doi.org/10.1016/j.nedt.2019.05.033

39.

*Könings

K. D.

de Jong

Lohrmann

Sumskas

Smith

O’Connor

S. J.

Spanjers

I. A. E.

Van Merriënboer

J. J. G.

Czabanowska

(2018). Is blended learning and problem-based learning course design suited to develop future public health leaders? An explorative European study. Public Health Reviews, 39(1), 1–12. https://doi.org/10.1186/s40985-018-0090-y

40.

Kwame

Petrucka

P. M.

(2021). A literature-based study of patient-centered care and communication in nurse-patient interactions: Barriers, facilitators, and the way forward. BMC Nursing, 20(1), 1–10. https://doi.org/10.1186/s12912-021-00684-2

41.

* Lapp

Caldwell

K. A.

(2012). Using food ethnographies to promote systems thinking and intergenerational engagement among college undergraduates. Food, Culture and Society, 15(3), 491–509. https://doi.org/10.2752/175174412X13276629246082

42.

* Li

Gupta

Wasser

(2019). A novel initiative contextualizing quality improvement and systems based practice in psychiatric clinical teaching settings. The Journal of Mental Health Training, Education and Practice, 14(3), 156–163. https://doi.org/10.1108/JMHTEP-10-2018-0060

43.

* Looman

W. S.

(2020). Teaching systems thinking for advanced family nursing practice: A theory-based tool. Journal of Family Nursing, 26(1), 5–14. https://doi.org/10.1177/1074840719891439

44.

*Martinez

Phillips

Fein

(2013). Perspectives on the changing healthcare system: Teaching systems-based practice to medical residents. Medical Education Online, 18(1), 1–5. https://doi.org/10.3402/meo.v18i0.20746

45.

* Mehta

Strauss

Beland

Fortunato

Staff

Lee

(2018). Stroke simulation improves acute stroke management: A systems-based practice experience. Journal of Graduate Medical Education, 10(1), 57–62. https://doi.org/10.4300/JGME-D-17-00167.1

46.

*Mills

L. M.

Hoffman

A. B.

Khan

Lai

C. J.

(2017). Integrating health systems science in early undergraduate medical education: Barriers to implementation and lessons learned. MedEdPublish, 6(4), 1–9. https://doi.org/10.15694/mep.2017.000204

47.

* Mitchell

J. D.

Parhar

Narayana

(2010). Teaching and assessing systems-based practice: A pilot course in health care policy, finance, and law for radiation oncology residents. Journal of Graduate Medical Education, 2(3), 384–388. https://doi.org/10.4300/JGME-D-09-00092.1

48.

* Nagler

Andolsek

Dossary

Schlueter

Schulman

(2010). Addressing the systems-based practice requirement with health policy content and educational technology. Medical Teacher, 32(12), Article e559-e565. https://doi.org/10.3109/0142159X.2010.528809

49.

*O'Connell

M. T.

Rivo

M. L.

Mechaber

A. J.

Weiss

B. A.

(2004). A curriculum in systems-based care: Experiential learning changes in student knowledge and attitudes. Family Medicine, 36(1), 98–104. Retrieved from https://pubmed.ncbi.nlm.nih.gov/14961411/

50.

Page

M. J.

McKenzie

J. E.

Bossuyt

P. M.

Boutron

Hoffmann

T. C.

Mulrow

C. D.

Shamseer

Tetzlaff

J. M.

Akl

E. A.

Brennan

S. E.

Chou

Glanville

Grimshaw

J. M.

Hróbjartsson

Lalu

M. M.

Loder

E. W.

Mayo-Wilson

McDonald

Moher

(2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. International Journal of Surgery, 88(1), 1–9. https://doi.org/10.1016/j.ijsu.2021.105906

51.

* Perez

J. A.

Jr Faust

Kenyon

(2009). The virtual practice: Using the residents’ continuity clinic to teach practice management and systems-based practice. Journal of Graduate Medical Education, 1(1), 104–108. https://doi.org/10.4300/01.01.0017

52.

* Peters

A. S.

Kimura

Ladden

M. D.

March

Moore

G. T.

(2008). A self-instructional model to teach systems-based practice and practice-based learning and improvement. Journal of General Internal Medicine, 23(7), 931–936. https://doi.org/10.1007/s11606-008-0517-0

53.

Phillips

J. M.

Stalter

A. M.

(2016). Integrating systems thinking into nursing education. Journal of Continuing Education in Nursing, 47(9), 395–397. https://doi.org/10.3928/00220124-20160817-05

54.

Plack

M. M.

Goldman

E. F.

Scott

A. R.

Brundage

S. B.

(2019). Systems thinking in the healthcare professions: A guide for educators and clinicians. The George Washington University.

55.

* Pulcrano

Chahine

A. A.

Saratsis

Divine-Cadavid

Narra

Evans

S. R. T.

(2015). Putting residents in the office: An effective method to teach the systems-based practice competency. Journal of Surgical Education, 72(2), 286–290. https://doi.org/10.1016/j.jsurg.2014.09.001

56.

Rosen

M. A.

DiazGranados

Dietz

A. S.

Benishek

L. E.

Thompson

Pronovost

P. J.

Weaver

S. J.

(2018). Teamwork in healthcare: Key discoveries enabling safer, high-quality care. The American Psychologist, 73(4), 433–450. https://doi.org/10.1037/amp0000298

57.

Rusoja

Haynie

Sievers

Mustafee

Nelson

Reynolds

Sarriot

Swanson

R. C.

Williams

(2018). Thinking about complexity in health: A systematic review of the key systems thinking and complexity ideas in health. Journal of Evaluation in Clinical Practice, 24(3), 600–606. https://doi.org/10.1111/jep.12856

58.

Sandelowski

Barroso

(2006). Handbook for synthesizing qualitative research. Springer Publishing Company.

59.

* Sanko

J. S.

Gattamorta

Young

Durham

C. F.

Sherwood

Dolansky

(2020). A multisite study demonstrates positive impacts to systems thinking using a table-top simulation experience. Nurse Educator, 46(1), 29–33. https://doi.org/10.1097/NNE.0000000000000817

60.

* Santen

S. A.

Hamstra

S. J.

Yamazaki

Gonzalo

Lomis

Allen

Lawson

Holmboe

E. S.

Triola

George

Gorman

P. N.

Skochelak

(2021). Assessing the transition of training in health systems science from undergraduate to graduate medical education. Journal of Graduate Medical Education, 13(3), 404–410. https://doi.org/10.4300/JGME-D-20-01268.1

61.

* Sheu

O’Brien

O’Sullivan

P. S.

Kwong

Lai

C. J.

(2013). Systems-based practice learning opportunities in student-run clinics: A qualitative analysis of student experiences. Academic Medicine, 88(6), 831–836. https://doi.org/10.1097/ACM.0b013e31828ff92c

62.

* Siri

Reed

A. I.

Flynn

T. C.

Silver

Behrns

K. E.

(2007). A multidisciplinary systems-based practice learning experience and its impact on surgical residency education. Journal of Surgical Education, 64(6), 328–332. https://doi.org/10.1016/j.jsurg.2007.05.002

63.

* Soto

R. G.

Cormican

D. S.

Gallagher

C. J.

Seidman

P. A.

(2010). Teaching systems-based competency in anesthesiology residency: Development of an education and assessment tool. Journal of Graduate Medical Education, 2(2), 250–259. https://doi.org/10.4300/JGME-D-09-00078.1

64.

*Spiker

Hege

Giddens

Cummings

Steinmetz

Tagtow

Bergquist

Burns

Campbell

Stadler

Combs

Prange

Schwartz

Brown

Sauer

(2021). Leveraging online learning to promote systems thinking for sustainable food systems training in dietetics education. Frontiers in Nutrition, 8(1), 1–9. https://doi.org/10.3389/fnut.2021.623336

65.

Stalter

A. M.

Phillips

J. M.

Dolansky

M. A.

(2017). QSEN institute RN-BSN task force: White paper on recommendation for systems-based practice competency. Journal of Nursing Care Quality, 32(4), 354–358. https://doi.org/10.1097/NCQ.0000000000000262

66.

Strauss

Corbin

(1994). Grounded theory methodology: An overview. In Denzin

Lincoln

(Eds), Handbook of qualitative research (pp. 273–285). Sage Publications, Inc.

67.

* Thornton Bacon

Trent

McCoy

T. P.

(2018). Enhancing systems thinking for undergraduate nursing students using friday night at the ER. The Journal of Nursing Education, 57(11), 687–689. https://doi.org/10.3928/01484834-20181022-11

68.

* Turley

C. B.

Roach

Marx

(2007). Systems survivor: A program for house staff in systems-based practice. Teaching and Learning in Medicine, 19(2), 128–138. https://doi.org/10.1080/10401330701332201

69.

* Voss

J. D.

Nadkarni

M. M.

Schectman

J. M.

(2005). The clinical health economics system simulation (CHESS): A teaching tool for systems- and practice-based learning. Academic Medicine, 80(2), 129–134. https://doi.org/10.1097/00001888-200502000-00004, Retrieved from https://journals.lww.com/academicmedicine/fulltext/2005/02000/the_clinical_health_economics_system_simulation.4.aspx

70.

*Wilper

A. P.

Smith

C. S.

Weppner

(2013). Instituting systems-based practice and practice-based learning and improvement: A curriculum of inquiry. Medical Education Online, 18(1), 1–6. https://doi.org/10.3402/meo.v18i0.21612

71.

* Zenni

E. A.

Ravago

Ewart

Livingood

Wood

Goldhagen

(2006). A walk in the patients’ shoes: A step toward competency development in systems-based practice. Ambulatory Pediatrics, 6(1), 54–57. https://doi.org/10.1016/j.ambp.2005.08.003

72.

* Ziegelstein

R. C.

Fiebach

N. H.

(2004). “The mirror” and “the village”: A new method for teaching practice-based learning and improvement and systems-based practice. Academic Medicine, 79(1), 83–88. https://doi.org/10.1097/00001888-200401000-00018, Retrieved from https://journals.lww.com/academicmedicine/fulltext/2004/01000/_the_mirror__and__the_village___a_new_method_for.18.aspx

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.53 MB

0.34 MB

0.44 MB

0.38 MB

0.69 MB