Curriculum Viability Indicators: A Delphi Study to Determine Standards and Inhibitors of a Curriculum

Abstract

Curriculum evaluation is typically done by using quality standards defined by accrediting bodies. This does not include inhibitors that hinder the achievement of standards. Hence, to address both standards and inhibitors, we have coined the new concept of “curriculum viability.” This study establishes consensus among experts on curriculum viability indicators, i.e. standards and inhibitors, and aims to provide a framework for evaluating the curriculum viability. The study was done in two phases. In the first phase, a consensus was established on the curriculum viability indicators using the Modified Delphi Technique using two rounds. In the first round of the Delphi process, 25 experts participated, which were reduced to 19 in the second round. After two rounds, experts developed a consensus on 40 out of 44 items. These included 27 standards and 13 inhibitors. In the second phase, 18 experts rank-ordered the indicators according to their relative importance in the areas of educational content and strategies, faculty, leadership, assessment, students, educational/working environment, communication, and technology. The list of indicators provides a framework for evaluating the curriculum viability, and their ordering enables curriculum managers to prioritize them during curriculum evaluation.

Keywords

curriculum viability quality standards curriculum evaluation Delphi study inhibitors

Introduction

Curriculum evaluation is routinely used to determine the quality of a curriculum by comparing it against certain quality standards. Curriculum evaluation may show that a curriculum is either meeting or not meeting the expected standards (Mcleod & Steinert, 2015). Usually, the curriculum evaluation does not consider inhibitors which indicate problems that may negatively affect curriculum quality and offer justifications on why the standards were not met.

In this paper, we introduce a new term: “curriculum viability.” To understand curriculum viability, the terms curriculum, quality, standards, indicators, and inhibitors are explained first.

Thomas et al. defined the curriculum as a planned educational experience (Thomas et al., 2016), whereas Abrahamson characterized it as a dynamic living entity (Abrahamson, 1978). Some educators take a narrow view, with the curriculum comprising only a collection of courses and syllabi. In our viewpoint, curriculum is more than a set of syllabi and courses; rather, it involves all the materials and activities that aim to facilitate students’ learning (Harden, 2001). Moreover, the definition of curriculum has evolved. Bosco described the basic structure of curriculum through his curriculum theory, which included aims, contents, methods of teaching, and evaluation (Bosco, 1971). As research in education has continued, the definition of curriculum has expanded, influenced by curriculum development and instructional design models (Edgar, 2012; Harden, 2001). Accreditation standards that measure the quality of medical education further expand the concept of curriculum beyond the core areas of aim, content, pedagogy, and assessment to include extended/supportive areas, such as the role of students, faculty, governance, and curriculum renewal (Gjerde & Sheehan, 1980; Karle, 2006).

When we employ the metaphor of a human being for the curriculum (Abrahamson, 1978) and consider it a dynamic living entity, we must acknowledge the probability that it may become sick. This could be due to problems affecting different components of the curriculum, as stated earlier. These problems are referred to as “inhibitors” that impede the curriculum from meeting quality standards. Therefore, to ensure the viability or “well-being” of a curriculum, the inhibitors of the curriculum should also be identified.

Quality in medical education can be defined in relative terms as the “state of reaching required standards as prescribed by the external agencies, and it meets those standards time and time again” (Joshi, 2012, p. 285). Here, we refer to standards as basic quality requirements that serve as a benchmark against which the quality of a program is evaluated and indicate where a program is falling short of achieving quality. They are a separate entity from facilitators that promote the curriculum quality (Bendermacher et al., 2017). The term indicator is used to denote specific, measurable characteristics of the program on which evidence can be collected. Our term curriculum viability includes both standards and inhibitors as indicators.

Quality standards are typically laid by higher education councils and accreditation bodies and agencies. For instance, the World Federation for Medical Education (WFME) in collaboration with the World Health Organization (WHO) provides “Basic medical education quality standards,” which have wide global acceptance (Karle, 2008). The standards in each of the areas define the criteria of achieving quality but do not consider the inhibitors of the curriculum. In the literature, inhibitors affecting the curriculum quality have been reported, but they have not been considered part of the accreditation standards (Rezaeian et al., 2013; Tackett et al., 2015). For instance, in one of the sub-areas of “educational program,” the WFME document describes one standard to be achieved as, “The medical school must define the curriculum models and instructional methods employed,” and another standard as “The curriculum and instructional methods should ensure that the students have responsibility for their learning process” (World Federation for Medical Education, 2015, p. 10). Still, the possible inhibitors that may impede the achievement of these standards are not stated.

Although inhibitors are not a part of quality standards, reviewers during program evaluation may explore them, based on the queries raised by the institution. Some of these queries can be: “What are the weaknesses of a curriculum?” or “What are the reasons for identified gaps between the developed and implemented curriculum?” (Posner, 2004). While evaluating a curriculum, some reviewers may use CIPP, Logic, or any other model relevant to curriculum evaluation (Frye & Hemmer, 2012; Ruhe & Boudreau, 2013). If a reviewer uses the CIPP model, which involves the evaluation of context, input, process, and product of a program’s value, curriculum viability can provide a road map for evaluating different areas of the curriculum with an additional value of already defined inhibitors. Some of the inhibitors already identified in the literature are: extreme ownership of the subject and faculty fighting for the available hours for teaching the discipline, limited opportunities for faculty members to meet and interact, abrupt and unplanned response to adjust or modify changes in the curriculum to meet societal demands and expectations, lack of student engagement with faculty, presence of strong disciplinary cultures, a research culture that undervalues education, lack of communication channels (Bendermacher et al., 2017), lack of sufficient study time, teacher resistance to student demands, and low-quality quizzes (Olson et al., 2013). These inhibitors can inform the reviewer about the issues or challenges that may be hindering the achievement of quality standards. One such use of curriculum viability can be that inhibitors such as “lack of sufficient study time” and “teacher resistance to student demands” can inform the reviewers about the issues faced by students, as part of the “input” component of the CIPP model. In the Logic model, which consists of resources, activities, outputs, and outcomes, the resources and activities are related to the inputs dedicated to the program and the actions taken by it to achieve the desired outcomes, respectively (Frye & Hemmer, 2012). Using curriculum viability indicators, the resources and activities can be evaluated considering both the standards and inhibitors in a similar way as stated for the CIPP model.

To close the gap in the curriculum evaluation literature wherein inhibitors have been largely ignored so far, we use the concept of curriculum viability to include both standards and inhibitors. A curriculum evaluated through the lens of curriculum viability would furnish a more realistic picture of its current status. We could not find studies that address the consensus and perceived importance of both standards and inhibitors. Hence, we performed a scoping review to explore standards and inhibitors to characterize curriculum viability. These standards and inhibitors reported in the literature were in the domains of educational strategy and content, faculty, leadership, assessment, students, educational environment, communication, and technology (Khan et al., 2019).

This study explores two questions: (1) Which standards and inhibitors addressing curriculum viability in undergraduate medical education, do the experts agree upon? (2) How do experts rank curriculum viability indicators by their importance?

Method

This study was done in two phases. In the first phase, a pilot and two rounds of modified Delphi were conducted to establish consensus on curriculum viability indicators. In the second phase, the indicators upon which consensus was developed were rank-ordered according to their relative importance. This process is depicted in Figure 1. The duration of the study was 11 months, including its conception, data collection, and reporting. The data were collected in 7 months. The gap between the first and second rounds of the Delphi study was 4 months; the gap between the second round of the Delphi study and the second phase of the study was 3 months. Ethical approval was obtained from the Ethical Review Committee of Riphah International University (Reference # Riphah/ERC/17/0246).

Figure 1.

Phases of the study.

Participants

Based on their formal qualifications and experience in Education, 34 experts were sent a request to participate in the Delphi study. Some of these experts were in the authors’ network, while others were sent email requests because they had published on curriculum quality and/or curriculum. Among the 25 experts who agreed to participate, 12 held PhDs in educational sciences, 10 Masters in health professions education, and one each in education and psychology. One participant was a PhD in Internal Medicine but was involved in medical education for 35 years. Their educational experience had a range of 14–48 years with a mean of 19 years and a median of 14 years. All experts had experience in curriculum design; 10 had experience in program evaluation, and 8 also had experience in accreditation. The experts included 15 males and 10 females from seven medical universities and two organizations, from both developing and developed countries. This was done to maximize the diversity of participants having exposure to different curricula in different regional and social contexts and also because standards and inhibitors may differ in these regions. The countries where they were working included Australia, Egypt, Malaysia, The Netherlands, Pakistan, Sri Lanka, and United States of America. Universities included Drexel University (USA), Maastricht University, Utrecht University and University of Groningen (Netherlands), The University of Western Australia (Australia), University of Kelaniya (Sri Lanka), Universiti Sains Malaysia (Malaysia), Jinnah Sindh Medical University, Khyber Medical University, Nur International University, and Riphah International University (Pakistan). One participant each was from FAIMER (Foundation of Advancement in Medical Education & Research) and WHO (World Health Organization).

Materials

To answer our first research question, a questionnaire containing 44 items was constructed based on a scoping review (Khan et al., 2019). The main headings constituted broad areas of the medical curriculum, whereas the subheadings comprised standards and inhibitors. This questionnaire was modified for a second round based on the consensus developed and feedback provided by the experts in the first round.

For the second phase of the study, the questionnaire was based on the 40 items on which experts agreed following the Delphi method. It had an option for the experts to order the indicators according to their importance in affecting the curriculum viability in descending order, 1 being the highest rank.

Procedure

Phase 1—Pilot study and Delphi rounds

This study was based on a modified Delphi method (Esmaily et al., 2008; Skulmoski et al., 2007). We developed the content of the questionnaire for the first round through extensive literature search done for the scoping review. This is different from a traditional Delphi study in which the first round explores the content for the questionnaire through opinions of experts obtained through face-to-face discussion or distant communication modes such as email (Al-Eraky et al., 2014).

Pilot study

A pilot study was done before Delphi Round 1 involving five participants who had done a Master’s program or equivalent course in health professions education or who had more than five years of experience in education. The questionnaire was sent to them via a link through email. Participants were asked to provide feedback on the questionnaire based on language, structure, understanding of the questions, accessibility of the questionnaire on the website (www.qualtrics.com), ease of browsing, and time required to fill it out. They reported satisfaction regarding the questionnaire through face-to-face meetings and via phone with the primary researcher and suggested no changes to it.

Delphi round 1

After the pilot study, the link to the questionnaire was sent to the selected experts through email for the first round. Anonymity among the expert participants was ensured to minimize bias. They were requested to score each item according to its importance to an undergraduate medical program, based on a 5-point Likert scale (consisting of 1 = extremely important, 2 = very important, 3 = moderately important, 4 = slightly important, and 5 = not at all important). They were also asked to provide a justification if they selected the options “extremely important” or “not at all important.” This was done to gain an understanding of the reason behind choosing an extreme value on the “Likert scale” so that quantitative data obtained through selecting an option were further strengthened by the qualitative data, mentioned in the results section as “representative quotes.”

Delphi round 2

For the second round, expert participants were again sent an email containing their individual and group results in an Excel sheet and a link to the second questionnaire, which included questions based on the items for which no consensus was reached. Anonymity was again ensured. Statements in the questionnaire that required more explanation or were not clear to the expert participants were modified by the primary author in consultation with co-authors, based on the responses from Round 1. Expert participants were asked to provide a reason if they changed their response from the previous round. This was done to understand their considerations so that we could better interpret the data. Between each round, those who did not respond were sent two to three reminders after a gap of three weeks. This helped increase the participation of the experts.

After two rounds, a consensus was developed for 40 out of 44 items on predetermined criteria, as explained in the data analysis section. Hence a third round was not conducted (Campbell et al., 1999; Fernández-Llamazares et al., 2013).

Phase 2: Ranking the curriculum viability indicators

To answer our second research question, a 40-item questionnaire comprising standards and inhibitors on which consensus was reached in the first two rounds was sent to all the 25 expert participants. They were asked to rate the items in the eight areas specified above so that the relative importance of these items could be determined.

Data Analysis

The consensus agreement was predetermined. For the Delphi Study, the first and second rounds were studied by analyzing the percentages of combinations of adjacent Likert scores. Although literature reports that agreement of more than 50% has also been used for consensus development (Powell, 2003; Skulmoski et al., 2007), we selected a higher percentage for agreement to make the process of selection of items more rigorous. A percentage of 80 or more on two adjacent scores was considered as agreement on that particular item. Hence, a combined percentage of 80 or more of “extremely important” and “very important,” “very important” and “moderately important,” “moderately important” and “slightly important,” and “slightly important” and “not important at all” were used to measure consensus on a particular item.

The expert feedback (i.e., quotes) was gathered by the primary researcher (RAK) for synthesis through the Qualtrics website and shared with co-authors. Quotes were selected by three authors (RAK, UM & MAL) independently, and then consensus was reached on representative quotes, which was further validated by two co-authors (AS and JVM). Quotes that were illustrative for one of the indicators (standards or inhibitors) brought up by the experts were considered as the representative quotes. Criteria for selecting them were based on clarity and alignment with the indicators, and this helped to address the discrepancies between the quotes by the experts.

In Phase 2 of the study, the mean values of indicators were calculated to order them under each area addressing curriculum viability. The mean was calculated by dividing the sum of the total score of a particular standard or inhibitor, as marked by participants divided by the total number of participants responding. The mean values were then arranged in descending order, with the lowest value indicating the highest priority. This was done because “1” was given the highest rank order number.

Results

Twenty-five experts participated in the first round of Phase 1. This number was reduced to 19 (74% participation, 26% dropout) in the second round. After two rounds, experts developed a consensus on 40 out of 44 items (91%). These included 27 standards and 13 inhibitors. In the second phase, the final questionnaire was sent to all the experts again to rank the indicators. As shown in Table 1, 18 out 25 experts (72% participation, 28% dropout) ordered the curriculum viability indicators according to their relative importance in areas of educational content and strategies, faculty, leadership, assessment, students, educational/working environment, communication, and technology.

Table 1.

Items (Standards/Inhibitors) in the Questionnaire Affecting Curriculum Viability and Their Ranking.

Area	No.	Indicators (Standards and Inhibitors)	Phase 1Developing Consensus		Phase 2RankingIndicators		Representative quotes
Area	No.	Indicators (Standards and Inhibitors)	R1	R2	Mean	SD	Representative quotes
Educational Strategies and Content (6 items)	01	Mission and Objectives	✓	NS	1.39	1.01	“…without mission and objectives, it is difficult to ascertain the direction where the curriculum is leading to.” (Mission and Objectives-R1) “…bad curriculum design can lead to the demise of a curriculum in the very beginning.” (Curriculum Design-R1). “Without explicit guidelines, official curricula never become operational curricula.” (Guidelines for implementing Curriculum-R1). “Information explosion demands regular review of instructional material” (Reviewing instructional material-R1). “Integration is not the only form of making a viable curriculum; many of the top medical institutions of the world do not use integration as a curricular design yet produce great graduates.” (Low quality of Integration-R1). “A viable curriculum has to be implementable…” (Guidelines for implementing Curriculum-R2).
	02	Curriculum design	✓	NS	2.32	0.98
	03	Guidelines for implementing curriculum	✗	✓	2.83	0.50
	04	Reviewing instructional material	✗	✓	4.00	1.00
	05	Irrelevant curriculum content	✗	✓	4.44	0.76
	06	Low quality of integration	✗	✗	-	-
Faculty (5 items)	07	Competence of instructors	✓	NS	1.89	1.41	“Staff needs faculty development…otherwise, teaching becomes a ritual dance.” (Faculty Development-R1)
	08	Staff involvement in organizational decision making	✗	✓	2.79	1.20
	09	Faculty Development	✓	NS	3.00	1.45
	10	Ability to perform multiple roles	✗	✓	3.37	1.18
	11	Lack of staff involvement in organizational decision-making	✓	NS	4.68	1.03
Leadership (6 items)	12	Allocate resources for optimal institutional functioning	✓	NS	2.28	0.73	“Without policy, the curriculum may become chaotic and ‘stuurloos’ (a Dutch word means out of control. (Lack of policies and procedures-R1). “Methods are more important than money and they don’t all require lots of resources. (Lack of resources in an institute-R1).
	13	Achieving internal/external goals of the institute	✓	NS	2.78	1.99
	14	Lack of policies and procedures	✗	✓	3.44	1.64
	15	Communication gatekeepers	✗	✓	3.72	1.59
	16	Lack of resources in an institute	✗	✓	4.28	1.59
	17	Focus on inspection and control	✗	✓	4.50	1.26
Assessment (3 items)	18	Measurement of Student’s learning outcomes	✓	NS	1.79	0.77	“Without feedback, learning becomes a blind game.” (Prompt Feedback to Students-R2).
	19	Prompt feedback	✓	NS	1.89	0.79
	20	Low-quality online quizzes	✓	NS	2.32	0.80
Students (12 items)	21	Student engagement with faculty, staff and administration	✓	NS	3.32	2.25	“Medical schools should help students develop their identity…Nothing is more demotivating than an aimless life” (Student’s Social self-perception -R1). “If students resist a curriculum, it simply cannot maintain its viability.” (Student’s resistance to new curriculum -R1).
	22	Student’s perception of teaching	✓	NS	3.58	2.35
	23	Student support services	✗	✓	4.63	2.37
	24	Student’s Perception of teachers	✓	NS	4.74	2.51
	25	Student’s Perception of atmosphere	✓	NS	4.79	2.21
	26	Active learning techniques	✓	NS	4.79	2.59
	27	Student’s Social self-perception	✓	NS	5.26	2.57
	28	Degree to which student complaints are addressed	✓	NS	6.89	1.97
	29	Lack of time for sufficient studying	✗	✓	8.05	2.24
	30	Neglecting student demands	✗	✓	8.95	1.32
	31	Student’s academic self-perception	✗	✗	–	–
	32	Student’s resistance to new curriculum	✗	✗	-	-
Educational/working Environment (6 items)	33	Learner-centered environment	✓	NS	1.74	0.91	“(Its) not a problem if the rigidity is focused on insisting on well-written objectives, aligned education methods, aligned assessment.” (Rigid control-oriented cultures-R1). “Students feel it (the learner-centered environment) and respond accordingly.” (Educational/Working Environment-R2).
	34	Climate of trust and shared understanding	✓	NS	1.84	0.81
	35	Flexible people-oriented culture	✓	NS	3.05	0.83
	36	Research culture undervaluing education	✗	✓	4.05	0.94
	37	Presence of strong disciplinary cultures	✗	✓	4.32	1.08
	38	Rigid control-oriented cultures	✗	NS	-	-
Communication (3 items)	39	Communicating policies and strategies	✓	NS	1.11	0.46	“Lack of suitable and efficient communication channels hampers any curricular design.” (Communicating policies and strategies -R1).
	40	Lack of sharing best practices across the organization	✓	NS	2.44	0.50
	41	Lack of social interaction	✓	NS	2.44	0.60
Technology (3 items)	42	Reliability of technology	✓	NS	1.61	0.59
	43	Appropriate tools and media	✓	NS	1.78	0.79
	44	Documented technology plan	✓	NS	2.61	0.68

Note. Bold items = inhibitors; ✓ = agreement on the indicator; ✗ = no agreement; NS = not submitted; R1 = round 1; R2 = round 2; SD = standard deviation.

Standards and Inhibitors Addressing Curriculum Viability Indicators (First Research Question)

In Round 1, experts agreed on 26 out of 44 items (59%), of which 22 were standards, and 4 were inhibitors. The 18 items (41%) on which consensus was not reached included 6 standards and 12 inhibitors (see Table 1).

In Round 2, experts agreed on all but three items (91%), which included low quality of integration (item 6), students’ academic self-perception (item 31), and student’s resistance to new curriculum (item 32). Among these items, two were inhibitors, and one was a standard.

In Rounds 1 and 2, experts also provided reasons for selecting “extremely important” and “not at all important” scores. These reasons are presented as representative quotes in Table 1.

Importance of Curriculum Viability Indicators (Second Research Question)

In phase 2, expert participants ranked the 40 indicators in 07 areas according to their perceived importance, as shown in Table 1.

Discussion

This study builds and documents consensus on curriculum viability indicators and ranks them according to their relative importance. In the first phase, we have established a consensus on curriculum viability indicators in eight areas of the curriculum. Interestingly, experts have made consensus on standards more than on inhibitors. This could be because standards define the aims that a curriculum should achieve and are routinely used for curriculum evaluation (Shahabudin, 2005; van Zanten et al., 2012), whereas inhibitors represent the problems that impede the achievement of these standards (Bendermacher et al., 2017; Olson et al., 2013), which is a relatively unfamiliar concept. Routine curriculum evaluations do not explore inhibitors except when explicitly asked for by the institutions or the accreditation bodies. Hence, there is a possibility that experts in our study did not regard inhibitors as of equal importance with standards.

Unlike standards, inhibitors may not be specific to one area and may impede the achievement of standards in other areas of the curriculum as well. For example, an inhibitor such as “irrelevant curriculum content” (Item 5 in Table 1) in the area “Educational content and strategy” may affect “Assessment” because irrelevant curriculum would result in irrelevant assessment as well. Similarly, among the inhibitors, “lack of policies and procedures” and “focus on inspection and control” (items 14 and 17 in Table 1) in the area of “Leadership” may hinder “faculty development” (item 9 in Table 1) as well.

We further concur with the experts that standards are important to portray the “perfect” curriculum. Yet, how often is a perfect curriculum encountered in reality? Curricula are like humans, seldom free of errors/diseases. If we use the analogy of a fever, treating it requires not only diagnosis but also identifying its underlying cause. Hence, for curriculum viability, the inhibitors are as important as standards, because identifying inhibitors effectively identifies the problems that the curriculum is fraught with, acting as a diagnostic tool for the prevention and treatment of curricular diseases. More research is recommended to study curriculum viability inhibitors and to explain the pathophysiology of how they affect the standards of a viable curriculum.

Inhibitors are not part of quality standards provided by the WFME, but some accrediting bodies such as the LCME in its self-study guide provide questions to explore challenges faced by the institutions. One such example is, “Is there sufficient time within and outside of formal class hours for students to acquire self-directed learning skills?” (Standards, Publications, & Notification Forms LCME, 2019).

The second objective of the current study was to rank indicators according to their relative importance. Here again, the experts uniformly ranked inhibitors lower than standards in all areas. The reasons for this could be similar to those mentioned above for reaching less consensus on inhibitors than on standards. The ranking of standards and inhibitors can be used to sort and document standards in accreditation documents according to their importance. Curriculum assessors can give marks for the standards and inhibitors according to their importance while assessing curriculum viability. This ranking can also be used to develop a tool to measure curriculum viability.

Currently, many tools are available to evaluate specific areas of a curriculum. For instance, HELES (Rusticus et al., 2019), DREEM (Rotthoff et al., 2012) measures the learning and educational environment respectively, PHEEM (Bari et al., 2018) assesses the Postgraduate hospital educational environment, and AIM (Sajjad et al., 2018) measures the implementation of assessment in medical schools. These tools are not only specific for a curriculum area but also lack options to identify possible inhibitors. The results of our study were rich enough to propose a preliminary curriculum viability framework that would address both developed (paper on curriculum) and implemented (taught) curriculum. Thus, based on the consensus and ranking of indicators, we have developed a framework in seven areas (Table 2). We excluded the area of “technology” as no consensus was obtained regarding inhibitors in this area. The curriculum viability framework can be used by medical educationalists, educators, and administrators to assess curriculum viability holistically and get a broad picture of the well-being of a curriculum. The curriculum viability framework can be applied in institutions to evaluate viability in different areas of the curriculum (Table 2). For example, “lack of staff involvement in curricular decisions” can be checked in policy documents of the curriculum on paper and also in the curriculum committee meeting minutes in the implemented curriculum. Another inhibitor, “low-quality quiz,” can be identified for the area of assessment in the implemented curriculum. This will enable the evaluator to understand the issues responsible for non-achievement of standards in specific areas of the curriculum.

Table 2.

Framework of Curriculum Viability.

Areas	Curriculum Viability Indicators	Measures of Curriculum Viability
Areas	Curriculum Viability Indicators	Curriculum on Paper	Implemented Curriculum
Educational Strategies and Content	Mission and Objectives (+) Curriculum design (+) Guidelines for implementing curriculum (+) Reviewing instructional material (+)	(1-4) Curricular Document	(1-3) Not Applicable (NA) (4) Minutes of Curriculum Review meetings
Educational Strategies and Content	Irrelevant curriculum content (-)	Content in Curriculum	Taught content
Faculty	Competence of instructors (+) Staff involvement in organizational decision-making (+) Faculty Development (+) Ability to perform multiple roles (+)	(7-8) Curricular Document (6,9) NA	(6,9) Faculty Evaluation report, Students feedback (7,8) NA
Faculty	Lack of staff involvement in organizational decision-making (-)	Curricular Document	Curriculum committee minutes
Leadership	Allocate resources for optimal institutional functioning (+) Achieving internal/external goals of the institute (+)	(11) Curricular Document (12) NA	(11) NA (12) Annual Academic Council Meetings, Interviews from Deans/Heads of Institute
Leadership	Lack of policies and procedures (-) Communication gatekeepers (-) Lack of resources in an institute (-) Focus on inspection and control (-)	(13,15,16) Curricular Document (14) NA	(13-16) Onsite Inspection, Interviews from the faculty
Assessment	Measurement of Student’s learning outcomes (+) Prompt feedback (+)	(17-18) NA	(17) Annual Academic Council Meetings (18) Faculty and student Interviews
Assessment	Low-quality quizzes (-)	Not Applicable	Post item analysis reports
Students	Student engagement with faculty, staff and administration (+) Perception of teaching (+) Student support services (+) Perception of teachers (+) Perception of atmosphere (+) Active learning techniques (+) Social self-perception (+) Degree to which student complaints are addressed (+)	(20) Curricular Document (21-27) NA	(20-27) Student Feedback, DREEM
Students	Lack of time for sufficient studying (-) Neglecting student demands (-)	(28) Timetables (29) NA	(28-29) Student Interviews
Educational/ Working Environment	Learner-centered environment (+) Climate of trust and shared understanding (+) Flexible people-oriented culture (+)	(30-32) NA	(30-32) Faculty and student Interviews
Educational/ Working Environment	Research culture undervaluing education (-) Presence of strong disciplinary cultures (-)	(33-34) NA	(33-34) Faculty and student Interviews
Communication	Communicating policies and strategies	Curricular Document	Faculty interviews
Communication	Lack of sharing best practices across the organization Lack of social interaction	(36-37) NA	(36-37) Faculty and student Interviews

Note. NA = not applicable.

Limitations

Our study has certain limitations. The questionnaire developed for data collection only had standards and inhibitors, based on what could be extracted from the literature.

Although experts in our study represented 7 countries and 13 institutions, it was observed that the non-participants in the second Delphi round were from countries other than the home countries of the research team. Due to the distant nature of the applied Delphi technique, it was difficult to convince these experts to participate in the subsequent rounds of the study.

Due to the varied background of experts, there was a possibility of a different perspective on the utility of standards. This may have affected their decision to relate the importance of an indicator to the curriculum viability differently. However, this was also considered a strength of the study because a variety of backgrounds and experiences would provide more insights.

Future Recommendations

The curriculum viability framework provides curriculum reviewers and experts an opportunity to review the curriculum with broader insight. However, to identify strengths and weaknesses in specific areas, further research would be required, particularly on curriculum viability inhibitors, where less work has been reported in the literature. The development of validated tools to identify curriculum inhibitors can inform curriculum experts about the possible factors that can undermine the curriculum. The study further provides directions on exploring inhibitors and developing an evaluation instrument that considers both standards and inhibitors of the curriculum. Future research identifying facilitators and exploring their effect on curriculum viability would also be interesting, since this is a relatively under-explored area.

Accreditation bodies can also consider re-writing standards to include curriculum inhibitors. In this respect, the data collection instrument and the institutional self-study guide of LCME provide helpful resources. However, we recommend that inhibitors be described alongside the corresponding quality standards. This will help curriculum developers to consider the inhibitors relevant to the particular standards and to design the curriculum to avoid curricular issues in the implementation phase. For the program evaluator, it would be easy to identify curricular issues from a set of already-identified inhibitors that may affect a curriculum.

Conclusion

This study establishes consensus on standards and inhibitors of curriculum viability reported in the literature. The curriculum viability framework we developed, provides a way of evaluating the health of a curriculum by not only considering the standards to be achieved but also by identifying the inhibitors that make it challenging to reach those standards.

Footnotes

Abbreviations

Acknowledgments

The authors are extremely thankful to Ms. Pamela Walter from Scott Memorial library, Jefferson University, Philadelphia, USA for her comments on the academic writing that helped in improving the manuscript.

Author Contributions

RAK, AS and JVM conceived and designed the study. RAK conducted the Delphi and rank ordering of the viability indicators. AS, JVM, UM and MAL helped in preparing the manuscript by providing repetitive feedbacks.

Availability of Data and Material

The data generated and analyzed during the study are not publicly available due to confidentiality and anonymity assured to the participants. However, it is available from the corresponding author on reasonable request.

Ethical Approval and Consent to Participate

This study does not report or involve any human or animal data or tissue. Ethical approval was taken from the ERC (Ethical Review Committee) of Islamic International Medical College, Riphah International University. The informed consent form was provided to all the participants through electronic link in the questionnaire. Participants were assured of anonymity and confidentiality of all the information and sharing of the results of the study on their requests. Only the Primary researcher was aware about their responses. The participants were not promised of any rewards or incentives in view of their participation in the study.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Rehan Ahmed Khan

References

Abrahamson

(1978). Diseases of the curriculum. Journal of Medical Education, 53(9), 951–957.

Al-Eraky

M. M.

Donkers

Wajid

van Merrienboer

J. J. G.

(2014). A Delphi study of medical professionalism in Arabian countries: The Four-Gates model. Medical Teacher, 36 (Suppl 1), S8–S16.

Bari

Khan

R. A.

Rathore

A. W.

(2018). Postgraduate residents’ perception of the clinical learning environment; use of postgraduate hospital educational environment measure (PHEEM) in Pakistani context. Journal of the Pakistan Medical Association, 68(3), 417–422.

Bendermacher

G. W. G.

oude Egbrink

M. G. A.

Wolfhagen

I. H. A. P.

Dolmans

D. H. J. M.

(2017). Unravelling quality culture in higher education: A realist review. Higher Education, 73(1), 39–60.

Bosco

(1971). Curriculum Theory [Review of the book Curriculum Theory (2nd ed.) by George A. Beauchamp]. The Educational Forum, 35(2), 259–260.

Campbell

S. M.

Hann

Roland

M. O.

Quayle

J. A.

Shekelle

P. G.

(1999). The effect of panel membership and feedback on ratings in a two-round Delphi survey: Results of a randomized controlled trial. Medical Care, 37(9), 964–968.

Edgar

D. W.

(2012). Learning theories and historical events affecting instructional design in education: Recitation literacy toward extraction literacy practices. SAGE Open, 2(4), 1–9.

Esmaily

H. M.

Savage

Vahidi

Amini

Zarrintan

M. H.

Wahlstrom

(2008). Identifying outcome-based indicators and developing a curriculum for a continuing medical education programme on rational prescribing using a modified Delphi process. BMC Medical Education, 8(1), 33.

Fernández-Llamazares

C. M.

Hernández-Gago

Pozas

Cabañas

M. J.

Feal

Villaronga

Valverde

(2013). Two-round Delphi technique for the consensual design of a paediatric pharmaceutical care model. Pharmacological Research, 68(1), 31–37.

10.

Frye

A. W.

Hemmer

P. a.

(2012). Program evaluation models and related theories: AMEE Guide No. 67. Medical Teacher, 34(5), e288–e299.

11.

Gjerde

Sheehan

T. J.

(1980). The development of performance standards for medical students. Evaluation & the Health Professions, 3(2), 237–248.

12.

Harden

R. M.

(2001). The learning environment and the curriculum. Medical Teacher, 23(4), 335–336.

13.

Joshi

(2012). Quality assurance in medical education. Indian Journal of Pharmacology, 44(4), 285–287.

14.

Karle

(2006). Global standards and accreditation in medical education: A view from the WFME. Academic Medicine, 81(12), s43–s48.

15.

Karle

. (2008). World federation for medical education policy on international recognition of medical schools’ programme. Annals of the Academy of Medicine Singapore, 37(12), 1041–1043.

16.

Khan

R. A.

Spruijt

Mahboob

van Merrienboer

J. J. G.

(2019). Determining ‘curriculum viability’ through standards and inhibitors of curriculum quality: A scoping review. BMC Medical Education, 19(1), 336.

17.

LCME. (2019, April). Standards, publications, & notification forms | LCME, 1–17.

18.

Mcleod

Steinert

(2015). Twelve tips for curriculum renewal. Medical Teacher, 37(3), 232–238.

19.

Olson

B. G.

Mata

Koszalka

T. A.

(2013). Implementing an online curriculum for medical education: Examining the critical factors for success. International Journal on E-Learning, 12(2), 197–208.

20.

Posner

G. J.

(2004). Analyzing the curriculum (3rd ed.). McGraw Hill.

21.

Powell

(2003). The Delphi technique: Myths and realities. Journal of Advanced Nursing, 41(4), 376–382.

22.

Rezaeian

Jalili

Nakhaee

Jahroomi Shirazi

Jafari

A. R.

(2013). Necessity of accreditation standards for quality assurance of medical basic sciences. Iranian Journal of Public Health, 42(Suppl 1), 147–154.

23.

Rotthoff

Ostapczuk

M. S.

de Bruin

Kröncke

K.-D.

Decking

Schneider

Ritz-Timme

(2012). Development and evaluation of a questionnaire to measure the perceived implementation of the mission statement of a competency-based curriculum. BMC Medical Education, 12(1), 109.

24.

Ruhe

Boudreau

J. D.

(2013). The 2011 Program Evaluation Standards: A framework for quality in medical education programme evaluations. Journal of Evaluation in Clinical Practice, 19(5), 925–932.

25.

Rusticus

S. A.

Wilson

Casiro

Lovato

(2019). Evaluating the quality of health professions learning environments: Development and validation of the health education learning environment survey (HELES). Evaluation & the Health Professions, 1, 16.

26.

Sajjad

Khan

R. A.

Yasmeen

(2018). Measuring assessment standards in undergraduate medical programs: Development and validation of AIM tool. Pakistan Journal of Medical Sciences, 34(1), 164.

27.

Shahabudin

S. H.

(2005). Medical education today: Globalising with quality. Medical Journal of Malaysia, 60, 4–10.

28.

Skulmoski

Hartman

Krahn

(2007). The Delphi method for graduate research. Journal of Information Technology, 6(1), 1–21.

29.

Tackett

Grant

Mmari

(2015). Designing an evaluation framework for WFME basic standards for medical education. Medical Teacher, 38(3), 291–296.

30.

Thomas

Kern

Hughes

Chen

(2016). Curriculum development for medical education: A six-step approach. Johns Hopkins University Press.

31.

van Zanten

Boulet

J. R.

Greaves

(2012). The importance of medical education accreditation standards. Medical Teacher, 34(2), 136–145.

32.

World Federation for Medical Education. (2015). Basic medical education WFME global standards for quality improvement. https://wfme.org/download/wfme-global-standards-for-quality-improvement-bme/