Abstract
Studies regarding the effectiveness of online learning compared with that of face to face (F2F) learning are conflicting. Some studies show students studying online have better outcomes, some show they have worse outcomes, and others show there is no difference. This retrospective cohort study compares competence in epidemiological concepts at the end of a graduate unit between Masters of Public Health students who studied F2F and those who studied online. In this unit, F2F students attended a 1-hour lecture (which was recorded) and a 2-hour tutorial each week. Online students listened to the recorded lecture and covered the same tutorial material through a facilitated asynchronous discussion board or a weekly synchronous 2-hour webinar. Students completed the same optional in-semester assignment and end of semester open-book exam. The results from 442 students (55% F2F) who completed the unit between 2015 and 2018 inclusive were included. The analysis compared final unit marks, controlling for prior academic performance. Results indicate that competence was reasonable in both formats of the unit but higher in F2F students, who after adjustment for prior degree academic performance achieved an average of 4.6 (95% confidence interval [2.2, 7.1]) more marks than online students. The better performance for F2F students was particularly true for students with poorer prior academic performance. These results suggest that F2F mode was more effective than online mode, particularly for students with a lower prior academic performance. Course instructors could usefully focus on enhancing student–instructor interaction and targeting students with lower academic ability when delivering online units of study.
Introduction
Online learning is now well established within the university sector. There are limited data on the number and extent of online units of study within universities in Australia; however, it is estimated that over half of the universities in the country offer degrees fully online (Darlo Higher Education, 2018). With 42% of prospective university students expressing a preference for the majority of their degree to be delivered online, the demand for online learning will continue (Austrade, 2018).
Students play an important role in the quality of online learning. Their engagement, time management, motivation, and attitude toward online learning are critical factors to their success, as well as the success of the course (Yukselturk & Bulut, 2007). There are many benefits for students from studying online. Geographic boundaries are nonexistent, allowing many students to undertake higher education when, due to a variety of reasons, they may otherwise be unable to access it. In an asynchronous environment, students are able to access learning materials at a time that is suitable to them, and in a synchronous environment, students are able to interact with the instructors and their peers in real time (Anderson, 2008).
Despite these benefits, a recent market research survey (conducted by Ernst & Young, an independent business advisory service) of Australian adults found that only 37% of current students felt that online learning was just as effective as traditional learning methods (Ernst & Young Australia, 2018). While this survey did not offer insight into the reasons behind this result, recent qualitative research in the United States suggests that students thought that receiving poorer grades would stem from the perceived lack of interaction with their peers and with their professors and that successful online study required increased self-motivation (Tichavsky, Hunt, Driscoll, & Jicha, 2015). Fittingly, recent research has shown that frequent and quality interaction between students and course instructors were important predictors of student success and satisfaction in online learning (Jaggars & Xu, 2016; Kauffman, 2015).
Even though the delivery of online learning at universities is widespread and likely to continue, the evidence, including systematic reviews, on the impact online learning has on the learning outcomes achieved in comparison with face-to-face (F2F) learning is conflicting. Some work shows that students studying primarily online have slightly better outcomes than those who study primarily F2F (Means, Toyama, Murphy, Bakia, & Jones, 2010), some show that online students have worse outcomes (Xu & Jaggars, 2011), and others show that there is no difference (Russell, 1999). Part of the complexity in these results is probably attributable to the different content being studied and to the blurred boundaries between different modes of study in numerous instances; for example, many F2F courses incorporate elements of online learning such as lecture recordings.
The aim of this study was to compare basic competence in epidemiological concepts, at the end of a graduate unit on introductory epidemiology, between Masters of Public Health (MPH) students who studied in F2F mode and those who studied in online mode. The results should be relevant to instructors of public health courses as they provide insight into the effectiveness of commonly used modes of study in this field and whether this differs between students of different academic abilities. This information could be used to help ensure that students are offered equal opportunity to achieve learning outcomes, regardless of their mode of study.
Method
An introductory unit in epidemiology methods and uses was offered to students as a core unit in the MPH program at the University of Sydney in Australia. At Australian universities, unit marks are recorded as an integer number between 0 to 100, with a pass mark achieved at 50 or above. Credit grades are given at a unit mark of 65 to 74, Distinction at 75 to 84, and High Distinction at 85 to 100. At the University of Sydney, marks are not standardized to a curve.
In the introductory epidemiology unit that is the focus of this study, F2F students attended a 1-hour lecture (which was recorded) and a 2-hour tutorial each week. Tutorials utilize a flipped classroom approach, which incorporates problem-based learning. This consists of students preparing answers to a range of questions in advance of the tutorial, and discussing the concepts covered and the answers to the questions with other students, facilitated by the tutor. Online students listened to the recorded lecture and completed the same tutorial material as the F2F students, either through a facilitated asynchronous tutorial on a discussion board (80% of online students) or through weekly synchronous online 2-hour webinars. In addition, all students had access to F2F and online support, an online discussion board, and sporadic additional tutorials held F2F and via webinar. All students each year completed the same optional in-semester assignment (25% or 30%) and the same open-book exam at the end of semester (70%). In some years, optional quizzes were also completed, accounting for 5% of the final mark. However, for the purposes of this analysis, all students’ final marks were calculated based on the assignment (30%) and exam (70%), or 100% exam result, whichever was higher. The exam was paper-based and invigilated for all students, regardless of their mode of study. Marking for both assignment and exam were essentially blind to study mode, although for some online students who undertook the exam at a remote center, it would have been possible to know that they were online students because the exam booklets in which the answers were recorded were different from the usual booklets or had been photocopied.
Data Sources
All data used for this retrospective cohort study were obtained from routinely collected information used for assessment and administrative purposes. Student enrolment mode (F2F or online), assessment marks, and year of enrolment in the unit were all obtained from unit class lists. Prior degree performance (undergraduate or postgraduate degree) was assessed for most students who applied to enter the MPH course (this assessment was conducted for a different, administrative, reason and did not occur in some semesters). This occurred at the time students applied to be accepted into the course. Based on prior degree performance, students were grouped into one of three categories based on weighted average grade (Pass; Credit; Distinction or High Distinction). Students whose previous degree was a PhD were awarded a prior degree performance score equivalent to a Distinction/High Distinction. Prior degree performance data were not available for students who began the MPH in a semester when prior academic performance was not assessed or who submitted relevant documents after the cut-off date for assessment in the year they applied.
Study Population
To be eligible for the study, students had to be domestic students enrolled in the MPH, who completed the introductory epidemiology methods unit between 2015 and 2018 inclusive and who had commenced the MPH in a semester when their prior academic performance was assessed. Over the years of the study, 1,391 students enrolled in the unit and attempted the end of semester unit exam. Of these, 715 were not eligible because they were enrolled in a different degree (n = 533) or were not a domestic student (n = 182). Another 234 were potentially eligible but were excluded as they did not have prior performance data because they submitted documents after the assessment deadline, applied in a semester when prior performance was not assessed, or did not supply supporting documents. Therefore, there were 442 out 676 (65%) potential students for whom data were available on their prior degree performance and all other variables of interest.
Statistical Analysis
The association between enrolment mode and final assessment mark was calculated using simple linear regression for unadjusted estimates. The association was also calculated using multiple linear regression with prior degree performance and year of enrolment in the unit as categorical covariates. Both covariates were significant at the p = .05 level in the final model. There was no evidence of collinearity and interactions. All analyses were conducted using SAS 9.4.
Results
Of the 442 students, 72% were female, 35% of students had a medical or postgraduate qualification, and 37% obtained at least a distinction average in their prior degree (Table 1). Half of the students opted to study online. There was a tendency for those with lower prior degree performance scores to opt to study face to face (χ22 = 10.13, p = .006), with 64% of participating students with a pass average studying face to face, compared with only 44% of those with a distinction or high distinction average. Students with postgraduate or medical degrees were more likely to study online (χ22 = 61.05, p < .0001), with 75% of participating students with a postgraduate or medical degree studying online compared with 37% of participating students with an undergraduate-only degree. The effect of this was to produce two groups that were different in terms of prior academic performance (pass average students made up 29% of F2F students and 16% of online students) and level of prior degree (undergraduate degree students made up 83% of F2F students and 47% of online students).
Characteristics of Study Participants, by Study Mode and Overall.
Univariate analysis showed that study mode (t = 2.94, p = .003), prior degree results (t = 3.84, p < .0001), and year of study (t = 4.13, p < .0001) were predictors of success in the unit results. Level of previous degree and gender were not associated.
After adjusting for prior degree results and year of study, F2F students achieved an average mark that was higher than that of those who studied online. Those studying F2F received an average mark of 69.6, compared with those who studied online who received an average mark of 64.9 (difference = 4.7; 95% confidence interval [CI] [2.3, 7.2]; Table 2).
Comparison of Unit Marks for Both Study Groups—Unadjusted and Adjusted.
Note. CI = confidence interval.
Adjusted for prior degree results and year of study.
Upon stratifying by prior degree performance, after adjusting for year of study, students performed better F2F than online at all levels of prior degree performance. Importantly, the better performance for F2F students was particularly true for students with poorer prior academic performance, with pass average students who studied F2F receiving an average mark that was 7.9 (95% CI [2.7, 13.1]) points higher than that of students with a prior degree result of pass average who studied online (Table 3).
Comparison of Adjusted Unit Marks for Both Study Groups, Stratified by Prior Academic Performance.
Note. CI = confidence interval.
Adjusted for year of study.
Discussion
This study found that students who studied introductory epidemiology in F2F mode performed better overall compared with students who studied online, taking into account prior degree performance and year of study. The difference was greatest in students who had completed their previous degree at pass level.
If it is accepted that students who study F2F in this context perform better in terms of general epidemiological principles than those who study online, why might this be so? The unit was designed to give all students a similar educational experience, to the extent this was possible given the different modes. The same content was used, with online students being able to listen to the same lectures the F2F students attended. Indeed, many of the F2F students did not attend the live lectures, instead choosing to listen to the recorded lectures at a later time. In addition, many students both attended lectures and listened to the recordings. So, in fact, the F2F mode did have an online component for many students. The biggest difference between the two modes was in the tutorials. F2F students attended weekly live tutorials where tutors and students discuss the concepts covered in the preset tutorial questions in real-time and tutors are able to use classroom facilities, such as whiteboards and projectors, to aid explanation. In contrast, most online students received input into the same weekly tutorials exercises only through contributing to an asynchronous discussion board. This board was monitored by tutors who would post comments that invited discussion on the tutorial questions, respond to questions posted by other students, and provide written explanations of key epidemiological concepts. Many online students had minimal contact with their asynchronous tutor or their fellow students throughout the semester. A minority of online students attended synchronous tutorials conducted via a webinar. If the apparent better performance in F2F students reflects reality, it seems likely that the main influence on the difference is the interaction with tutors and fellow students in the live tutorial environment. Since F2F students could listen to lectures online in place of attending the live lecture, it is possible that the combined effect of this flexibility and the F2F interactions in tutorials lead to better learning outcomes.
Other potential advantages for F2F students include that struggling F2F students are more easily identified by tutors, making it more likely that support mechanisms will be put in place; and F2F students have easier access to the academics running and supporting the unit. This would explain why students with the lowest prior degree performance have the greatest difference in final marks between modes of study compared with the other performance levels (Table 3).
Some insights are provided by previously published work. Different students prefer one mode or the other depending on a range of factors, including the curriculum (Poechter & Maier, 2010) and their personality type (Bolliger & Erichsen, 2013). A study of university students in Austria found that students preferred F2F teaching when the curriculum focused on understanding concepts or learning to apply skills, and preferred online approaches when the focus was engaging in “self-regulated learning” (being able to engage with material when and where the student wanted). The conclusion of the authors was that a blended approach to learning was likely to be most appropriate for many courses (Poechter & Maier, 2010), which is in line with findings from a systematic review (Means et al., 2010) that showed that blended learning was more effective than online or F2F learning alone. The results provide some support to the findings in the current study, as much of the content of the epidemiology unit involved understanding concepts or learning to apply skills. Another study, of computer programming students, examined several variables (gender, age, educational level, locus of control, learning style, intrinsic goal orientation, extrinsic goal orientation, task value, control beliefs, self-efficacy, test anxiety, cognitive strategy use, and self-regulation) that might influence a student’s success in the online learning environment. The study found that self-regulated learning was positively associated with better online learning (Yukselturk & Bulut, 2007). A study of graduate psychology students in Pakistan found that they had similar preferences for online or F2F modes of study (Mahmood, Mahmood, & Malik, 2012). Two common themes in published studies are the need to improve the quality of online courses and the educational value of encouraging online students to interact with teaching staff and other students in the online environment (Yukselturk & Bulut, 2007).
Limitations and Strengths
This study had several limitations. The students self-selected into study mode rather than being randomized, randomization not being possible for practical reasons (students usually had commitments that influenced which study mode they chose and/or had a strong preference to study in one mode or the other). It was suspected from experience in previous years that students who chose to study online would tend to be better academically, and the study results supported this suspicion. Adjusting for prior academic performance attempted to account for any such differences, but it is likely that this adjustment was incomplete. To the extent that it was incomplete, it would be expected to lead to an underestimation of any advantage associated with F2F study mode. Alternatively, if prior academic performance was not a good indicator of academic ability (which seems unlikely), the adjustment using this variable could have resulted in bias in either direction. Prior academic performance is just one of a number of potentially important factors that might affect a student’s performance in epidemiology. Other factors, such as employment commitments, demands from family and friends, prior knowledge and fluency in English, might all be expected to influence the results. Potential differences in fluency in English were partly controlled for by excluding international students. Unfortunately, there was no information available that could be used to directly control for any influence the other variables might have had. Finally, students who commenced the unit but did not sit for the final exam were not included in the study. These students were essentially subjects lost to follow-up. Typically, students who continue the unit beyond the census date but do not complete it do so for a range of reasons, most not connected to their academic performance in the subject. It is possible that some students did not sit for the exam because they were concerned about their academic performance and that this concern related to their mode of study. However, the number of such students was small (3.2% of students over the 4 years of the study), and this is not expected to be an important cause of bias in the study.
Strengths of the study include the involvement of most eligible students (at least 65%), the included students being representative of all students (comparable proportions of female [73% vs. 72%] and online students [49% vs. 50%]), measuring exposure (study mode) and confounders prior to the outcome occurring and measuring the outcome (the mark achieved) blind to study mode and confounder status, including variables attempting to control for what was perceived to be the most important potential confounder (academic ability/performance), the analysis attempting to take account of relevant variables, and including enough students to provide good power.
Conclusion and Implications
F2F students performed better than online students in an introductory epidemiology unit. This difference was greatest in those with a lower academic ability (based on prior degree results). The study provides support for the suggestion that current online learning is not as effective as F2F learning for some subjects, including epidemiology, particularly for students with lower academic ability. The ramifications of these results should be considered by instructors of online units and courses. Broadly, teaching methods should be carefully assessed for effectiveness in the online environment rather than simply attempting to replicate F2F approaches. It is likely that efforts to improve student–tutor interaction in tutorials will be most valuable. In fact, as a result of this study, we have started to put more resources into running live tutorials for online students via webinars. The results also suggest that there is a place for a targeted approach where online students with known lower academic ability are actively encouraged to engage with course instructors and any educational support that can be offered.
Footnotes
Ethical Approval
This study has been approved by the University of Sydney Human Ethics Committee.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: A.K. is supported by a National Health and Medical Research Council scholarship.
