Assessing Curriculum Efficiency Through Monte Carlo Simulation

Abstract

A Monte Carlo algorithm is designed to predict the average time to graduate by enrolling virtual students in a degree plan. The algorithm can be used to improve graduation rates by identifying bottlenecks in a degree plan (e.g., low pass rate courses and prerequisites). Random numbers are used to determine whether students pass or fail classes by comparing them to institutional pass rates. Courses cannot be taken unless prerequisites and corequisites are satisfied. The output of the algorithm generates a relative frequency distribution which plots the number of students who graduate by semester. Pass rates of courses can be changed to determine the courses that have the greatest impact on the time to graduate. Prerequisites can also be removed to determine whether certain prerequisites significantly affect the time to graduate.

Keywords

curriculum efficiency graduation rate Monte Carlo simulation

Student debt continues to rise. In 2014–2015, 61% of students receiving bachelor’s degrees graduated with an average debt of $28,100 (Baum, Ma, Pender, & Welch, 2016). Currently in 2017, student loan debt has reached $1.4 trillion which translates into an average debt of $34,144 per student (Dickler, 2017). Another startling statistic is that only one in every three students graduate in 4 years (Neuman, 2017), and just over 60% of students complete their college degree in 6 years (Deming & Figlio, 2016; Neuman, 2017).

In New Mexico and in other states, graduation rates are tied to a funding formula. Thus, reduced graduation rates can lead to increased scrutiny of an institution’s practices and effectiveness (Bailey, Calcagno, Jenkins, Leinback, & Kienzl, 2006). Institutions of higher education are continually examining ways to improve student retention and reduce attrition (Heileman, Babbitt, Abdallah, & Dougher, 2014) and identify the variables which contribute to timely graduation and academic success. The Higher Education Research Institute at University of California, Los Angeles uses a logistic regression model to predict completion rates using student and institutional variables (DeAngelo, Franke, Hurtado, Pryor, & Tran, 2011).

Academic success factors fall into two categories: preinstitutional and institutional (Wegdahl, Heileman, Slim, & Abdallah, 2014). Preinstitutional factors may include grade point average, Scholastic Aptitude Test (SAT) and American College Testing (ACT) scores, and socioeconomic status.Institutions may not have a large degree of control over preinstitutional factors due to their geographical location and the preparation of incoming and transferring students. However, different institutions continue to show different success rates with similar preinstitutional cohorts of students. The difference can be attributed to institutional factors that include the level of student advisement, student support services, scheduling, quality of teaching, and student engagement (Wegdahl et al., 2014). While each of these factors plays a role, a degree program is fundamentally defined by its curriculum. An efficient degree plan may allow a student to graduate in 4 years in spite of an occasional course failure, while a complex degree plan may add semesters or years (and financial burden) to complete a degree. Thus, a degree plan’s complexity can ultimately determine whether an at-risk student completes a degree. Thus, it behooves an institution to design effective institutional degree plans that remain rigorous while minimizing the impact of disruptions in its prescribed course sequence recommendations.

Measures of Curriculum Efficiency

Previous work on improving graduation rates has focused on advisement (Hatch & Garcia, 2017), course restructuring through corequisite models (Complete College America, 2016; Upasana & Santhosh, 2017), freshman year support (Veenstra, 2009), first year experiences (Connolly, Flynn, Jemmott, & Oestreicher, 2016), and early study skills courses (Windham, Rehfuss, Williams, Pugh, & Tincher-Ladner, 2014). In an effort to improve graduation rates, the University of Texas at San Antonio (2011, p. iii) recognized that

the university’s degree plans have become increasingly complex because of the adoption of more course prerequisites, a reduction in elective courses, and/or multiple choices of concentration areas within majors. In many cases, greater curricular complexity translates into delayed college completion.

Wegdahl et al. (2014) first defined metrics that quantify the curricular complexity or efficiency of a degree plan. A degree plan is represented by a directed graph where individual courses form the vertices of the graph and prerequisites form the edges. Measures used to quantify the curriculum efficiency include the total number of hours required to complete the degree, the number of edges that impinge on a vertex, the number of edges that emanate from a vertex, and the longest path of edges required to reach a course. Curriculum rigidity is defined to be the total number of edges divided by the total number of vertices. Bottleneck courses are courses that have more than three impinging edges, more than three emanating edges, or more than five combined impinging or emanating edges.

The research of Wegdahl et al. (2014) motivated us to design a statistical simulation tool to quantify curriculum efficiency based on the average time to graduate. A statistical simulation is appropriate since the number of paths to a degree can be shown to be greater than two raised to the number of courses in the degree since a student can pass or fail each individual course in the degree. For example, a degree plan with 40 courses will have at least a trillion paths. A Monte Carlo simulation avoids the task of enumerating all paths and their probabilities and uses straightforward programming logic to assess the average time to graduate and quantify a curriculum’s efficiency.

Monte Carlo Algorithm

Monte Carlo methods (Hammersley & Handscomb, 1965; Metropolis & Ulam, 1949) have historically been used to model physical and biological phenomena, solve optimization problems, and calculate risk in business. The large number of degrees of freedom and uncertainty in the application are addressed through the use of repeated random sampling. We apply the Monte Carlo method to assess curriculum efficiency by enrolling a large number of virtual students in a degree plan, statistically determining whether each student passes each course using random numbers, reevaluating a student’s degree trajectory based on course failures and successes, and recording the amount of time it takes each student to complete a degree plan.

Our algorithm requires the following inputs:

Courses in a degree plan and their recommended order.

Prerequisites and corequisites of courses.

Pass rates of courses (based on institutional data).

The algorithm is divided into the following phases.

Determine whether a student passes the prescribed courses in a semester. A course is passed if a randomly chosen number r, between 0 and 1, is less than the pass rate. The pass rate is a number between 0 and 1, which represents the average number of students which pass a class. For example, if the pass rate is .7, 70% of students pass the class on average.

Assemble a list of courses for the following semester based on the courses that are passed or failed in the previous semester.

Increase the pass rate of a course if a course is taken multiple times.

A student cannot take a course unless the prerequisites and corequisites for the course have been met. Two options are available when assembling a list of courses for the following semester if courses in a previous semester were not passed. The student can fill his or her schedule with courses failed in the previous semester based on their original order, or alternatively, give priority to courses which serve as prerequisites for the most number of courses when scheduling.

When adjusting the pass rates of courses for courses that are taken more than once, we use the following equations:

f_{r} = 1 - p_{r}, f_{r a} = \frac{f_{r}}{2^{n}}, p_{r a} = 1 - f_{r a} .

(1)

where p_r is the original pass rate of a course, f_r represents the original failure rate of a course, f_ra is the adjusted failure rate if a student has taken the course n times, and p_ra is the adjusted pass rate if a student has taken a course multiple times. Thus, the failure rate is reduced by two each time a course is retaken.

In our Monte Carlo simulations, hundreds of thousands of virtual students make their way through a degree plan. The number of semesters required to complete the degree plan is recorded for each student. A relative frequency distribution can then be constructed which shows the percentage of students who graduate by semester. The average time to graduate $\bar{x}$ can also be computed, and a confidence interval (Pagano, 2012) can be constructed using the computed standard deviation s, the number of students N, and a critical t value $t_{c r i t}$ based on the confidence level (e.g., 99%):

[\bar{x} - \frac{s}{\sqrt{N}} t_{c r i t}, \bar{x} + \frac{s}{\sqrt{N}} t_{c r i t}] .

(2)

The distribution of averages $\bar{x}$ will be normally shaped for thousands of students according to the central limit theorem.

The pass rates of individual courses can be changed to determine which courses most significantly affect the average time to graduate. A department may assign experienced instructors to these courses in order to improve graduation rates. The prerequisites of individual courses may also be removed to determine whether they significantly affect the average time to graduate. If a prerequisite delays graduation, a program may choose to evaluate whether the prerequisite is essential.

Degree Plans Assessed

We evaluate the curriculum efficiency of the BS mathematics and the Bachelor of Engineering Information Engineering Technology (BEng IT) degree at Northern New Mexico College (NNMC).

Figure 1 lists the recommended courses by semester along with their pass rates (included below each course in light blue or orange) for the BS mathematics degree plan at NNMC. The pass rates were provided by Institutional Research at NNMC for 2016–2017 and represent the percentage of A, B, and C’s received in all sections of the course. There are a total of 36 courses in the degree plan. Prerequisites are also listed in the last rows of the spreadsheet and are highlighted in purple. Similarly, Figure 2 shows the degree plan for the BEng IT degree at NNMC which contains a total of 41 courses. Both degrees require a total of 120 hours. Courses are either 3 or 4 credit hours.

Figure 1.

Input for an eight semester BS mathematics degree plan. Each odd row (highlighted in yellow) lists the recommended courses for each of the eight semesters. For example, MATH 130L, ENG 111, FYE 101, HUM1, and ELEC1 are the recommended courses for the first semester. MATH 395, MATH 401, MATH 466, and MATH 441 are the recommended courses for the eighth semester. The rows highlighted in light blue or orange list the pass rates below each course. The orange pass rates are estimated. For example, {.82, .7, .89, .9, .9} are the pass rates for MATH 130L, ENG 111, FYE 101, HUM1, and ELEC1, respectively. The last rows (highlighted in purple) assign prerequisites. The first column stores the course, and the subsequent columns store its prerequisites. For example, MATH 130L and ENG 111 serve as prerequisites for CHEM 121/L. Similarly, MATH 130L serves as a prerequisite for MATH 150.

Figure 2.

Input for an eight semester BEng IT degree plan. A total of 41 courses are required for the degree.

Results

Figure 3 shows the relative frequency distribution of time to graduate using a simulation with 100,000 virtual students for the BS mathematics degree plan described in Figure 1. Figure 4 shows the result of the same simulation using the courses and pass rates in the BEng IT degree. We reduced the pass rates in Figures 1 and 2 by an additional 5% to account for withdrawals and scheduling conflicts. Also if a reduced set of courses is taken in the last semester (e.g., two courses vs. a normal four courses), only that fraction of the last semester (2/4 = .5 semester) is used to calculate the average time to graduate. For the BS mathematics degree, the average student graduates in 8.721 semesters when failed courses are retaken in their original order. We calculate the 99% confidence interval to be [8.714, 8.728] which means that if the simulation was conducted multiple times, 99% of the simulations would compute an average time to graduate that falls within this interval. For the BEng IT degree, the average student graduates in 9.642 semesters when failed courses are retaken in their original order. The 99% confidence interval for the BEng IT degree is [9.636, 9.648]. Giving courses priority that serve as prerequisites for the most subsequent courses when rescheduling failed courses decreased the time to graduate for the BEng IT degree to 9.524. Prioritizing courses for the mathematics degree when rescheduling failed courses actually increased the time to graduate by a small amount to 8.726.

Figure 3.

Percentage of students who graduate as a function of semester for the BS mathematics degree. NNMC = Northern New Mexico College.

Figure 4.

Percentage of students who graduate as a function of semester for the BEng IT degree. NNMC = Northern New Mexico College.

Figure 5 shows the effect of decreasing the pass rates of individual courses by 20% for courses in the BS mathematics degree. The percentage increase (PI) in time to graduate was calculated using Equation 3

P I = 100 (\frac{T_{k}}{T} - 1)

(3)

where T refers to the average time to graduate with unaltered pass rates, and T_k is the average time to graduate when the pass rate of course k is reduced by 20%. The only courses with percentage increases (PIs) above 2% in the average time to graduate are Course 1—MATH 130L (Intermediate Algebra), Course 6—MATH 150 (College Algebra), Course 11—MATH 155 (Trigonometry and Precalculus), Course 15—MATH 162 (Calculus I), Course 20—MATH 163 (Calculus II), and Course 24—MATH 264 (Calculus III). When the pass rates are individually reduced by 20% in the BEng IT degree, the PIs are minimal and lie between .32% and 1.4%. The simulation in Figure 5 took 41 seconds on a Lenovo desktop with a 3.10 GHz AMD processor with 8 GB of RAM. The code was written in Fortran.

Figure 5.

Effect of decreasing pass rates by 20% in BS mathematics degree.

Figure 6 shows the effect of removing individual prerequisites of courses using simulations with 100,000 virtual students for the BS mathematics degree. The percentage decrease in time to graduate is defined as (−PI) where PI is defined by Equation 3. The only courses which experienced a noticeable decrease in the time to graduate were MATH 150, MATH 155, MATH 162, MATH 163, MATH 264, MATH 313 (Complex Variables), MATH 312 (Partial Differential Equations), MATH 395 (Practicum), and MATH 466 (Mathematical Methods). We do not plot the values for all courses with prerequisites since the decreases were relatively small (.14%) Removing prerequisites for MATH 163 (Calculus II) decreased the average time to graduate by 4.8%. Unfortunately, the prerequisite (Calculus I) for MATH 163 is needed. When the prerequisites are individually set to zero in the BEng IT degree, the highest percentage decrease is 1.32% for the last course, IT 491. The percentage decreases for the remaining courses lie between −.028% and .054%. Occasionally, removing a prerequisite can actually increase the time to graduate (e.g., −.028%) because of the stochastic nature of the Monte Carlo simulation if the prerequisite has little effect on time to graduate. However, the magnitudes of PIs in time to graduate were all less than or equal to .05% in both the BS mathematics and BEng IT degrees if the time to graduate increased when removing a prerequisite.

Figure 6.

Effect of removing prerequisites from individual courses in BS mathematics degree.

Figure 7 shows the effect of changing the overall pass rates on the average time to graduate for both degrees keeping in mind that the maximum pass rate of a course cannot exceed 1.0. The change in the PI in time to graduate seems to be mostly linear. The slope is approximately −1.3 for the BS mathematics degree and −1.0 for the BEng IT degree.

Figure 7.

Effect of varying overall pass rates on average time to graduate.

Assumptions and Implications

Our model makes the following assumptions:

Students do not drop out or take reduced loads while attending college.

While students can take courses outside their major, we assume students follow their prescribed degree plan and take the recommended course loads. This may depend on the quality of advisement at the institution. Students also do not change majors.

We assume the pass rates of courses taken multiple times improves according to Equation 1.

Pass rates in courses are accurate.

Courses are offered to students when needed. For example, a student would not need to wait a semester or year for a course to be offered.

If one is willing to accept these assumptions, our model does predict the average time to graduate as well as the variability in time to graduate through a relative frequency distribution. While we acknowledge that these assumptions may not apply to every student, an institution often does not have control over a student’s decision to switch majors or circumstances which require a student to temporarily withdraw from college. An institution, however, does have control over the prescribed curriculum within each degree plan which is what we believe our model can assess.

Discussion and Conclusion

We have developed a program to quantify the efficiency of a curriculum using a Monte Carlo algorithm which predicts the average time to graduate. The program creates a relative frequency distribution of time to graduate and identifies course pass rates and prerequisites that significantly affect the time to graduate.

We use the program to assess the curriculum efficiency of both the BS mathematics and BEng IT at NNMC. The course sequence of College Algebra, Trigonometry and Precalculus, Calculus I, Calculus II, and Calculus III in the mathematics degree seems to be an important sequence since decreasing pass rates in these courses have the most impact on time to graduate. Each previous course serves as a prerequisite for the next course. Calculus II and III also serve as prerequisites for many of the upper division courses. In contrast, the time to graduate is not very sensitive to pass rates or prerequisites for individual courses in the BEng IT degree.

We believe the program can be useful in assessing the curriculum efficiency of other degree plans and suggesting changes to those degree plans that could reduce the average time to graduate.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Author Biographies

David Torres is the chair of Mathematics and Physical Science at Northern New Mexico College.

Jorge Crichigno is an associate professor in the College of Engineering and Computing at the University of South Carolina.

Carmella Sanchez is the director of Institutional Research at Northern New Mexico College.

References

Bailey

Calcagno

J. C.

Jenkins

Leinback

Kienzl

(2006). Is student-right-to-know all you should know? An analysis of community college graduation rates. Research in Higher Education, 47(5), 491–519.

Baum

Pender

Welch

(2016). Trends in student aid 2016. New York, NY: The College Board.

Complete College America. (2016). Corequisite remediation: Spanning the completion divide: Breakthrough results fulfilling the promise of college access for underprepared students: Executive Summary. Indianapolis, IN: Author.

Connolly

Flynn

E. E.

Jemmott

Oestreicher

(2016). First year experience for at-risk college students. College Student Journal, 51(1), 1–6.

DeAngelo

Franke

Hurtado

Pryor

J. H.

Tran

(2011). Completing college: Assessing graduation rates at four-year institutions. Los Angeles: Higher Education Research Institute, University of California, Los Angeles.

Deming

D. J.

Figlio

(2016). Accountability in US education: Applying lessons from K-12 experience to higher education. Journal of Economic Perspectives, 30(3), 33–56.

Dickler

(2017). Student loan balances jump nearly 150 percent in a decade. Englewood Cliffs, NJ: CNBC LLC.

Hammersley

J. M.

Handscomb

D. C.

(1965). Monte Carlo methods. London, England: Chapman & Hall.

Hatch

D. K.

Garcia

C. E.

(2017). Academic advising and the persistence intentions of community college students in their first weeks in college. The Review of Higher Education, 40(3), 353–390.

10.

Heileman

G. L.

Babbitt

Abdallah

C. T.

Dougher

M. J.

(2014, April). Efficient curricula: The complexity of degree plans and their relation to degree completion. Paper presented at the Higher Learning Commission Conference. Retrieved from http://cop.hlcommission.org/Learning-Environments/heileman/Print.html

11.

Metropolis

Ulam

(1949). The Monte Carlo method. Journal of the American Statistical Association, 44, 335–341.

12.

Neuman

(2017). How to graduate from college on time: Part I. Westford, MA: Carnegie Communications.

13.

Pagano

(2012). Understanding statistics in the behavioral sciences (10th ed.). Boston, MA: Cengage Learning.

14.

The University of Texas at San Antonio (2011, December). Four-year graduation rate improvement plan. Retrieved from provost.utsa.edu/home/docs/UTSA_Graduation_Rate_Plan_2011-FINAL.PDF

15.

Upasana

Santhosh

(2017). Corequisite model: An effective strategy for remediation in freshman level quantitative reasoning course. Journal of STEM Education: Innovation & Research, 18(2), 23–29.

16.

Veenstra

(2009). A strategy for improving freshman college retention. The Journal for Quality and Participation, 31, 19–23.

17.

Wegdahl

Heileman

G. L.

Slim

Abdallah

C. T.

(2014, June). Curricular efficiency: what role does it play in student success? Paper presented at the 121st ASEE Annual Conference & Exposition, Indianapolis, IN. Retrieved from https://peer.asee.org/curricular-efficiency-what-role-does-it-play-in-student-success

18.

Windham

M. H.

Rehfuss

M. C.

Williams

C. R.

Pugh

J. V.

Tincher-Ladner

(2014). Retention of first-year community college students. Community College Journal of Research and Practice, 38(5), 466–477.