Can online discussions benefit students’ learning in online courses? Evidence From teaching introduction to microeconomics

Abstract

In this research, we investigated whether online discussions can significantly benefit students’ learning in online courses. We designed an experiment by dividing 129 students who enrolled in four fully online “Introduction to Microeconomics” courses taught by the same instructor into four groups (one control group and three experimental groups). We adopted econometric analysis to investigate the issue. Our empirical evidence demonstrated that online discussions can significantly benefit students’ exam performance in online courses only when instructors are effectively engaged in discussions. Instructors who do not effectively engage in online discussions had a less than significant effect on students’ exam performance even when peer-responses were required. On the other hand, the effect was positive and significant when instructors effectively engaged but the level of significance was weak (at the 10% level of significance) and began with a later exam (Exam 4, out of a total of five exams) rather than the first exam or earlier. Detailed discussions about these phenomena are offered as are several possible reasons for these findings. We recommend significantly increasing weights for online discussions in the final course grade and including exam questions that directly reflect online discussion questions (including serious participation in discussions and requiring that students provide answers to instructors’ follow-up questions). These additions would significantly benefit students’ learning in online courses.

Keywords

Online discussions Online learning performance Method of follow-up questions Engagement of the instructor Introduction to Microeconomics

Introduction

“Introduction to Microeconomics” is one of the fundamental core courses for undergraduate business and economics majors. This course mainly focuses on two essential topics: consumer theory and firm theory. Consumer theory helps us to understand and predict how consumers will allocate their incomes when selecting different goods and services; firm theory helps us to understand how firms apportion resources in different types of market environments (competition, monopoly, oligopoly, etc.) while making decisions about production, employment, investments, etc.

In the past, instructors who have taught the “Introduction to Microeconomics” course might use classroom game-play experiments to improve students’ interest and engagement in a classroom in-person setting. Abundant empirical evidence has demonstrated that classroom game-play experiments are a fun, lively, and creative way to teach and learn economics (e.g., Dickie, 2006; Lin and Dunphy 2013; Lin, 2018). However, in a fully online setting, classroom game-play experiments may face some challenges due to the use of an in-person classroom platform, although some experiments may be able to be implemented in the online classroom. Morgan, Sharp and Grogan (2020) attempted to run three classroom experiments (public goods, prisoners dilemma, and pit market) in the fully online classroom, but they pointed out the challenges of running classroom experiments in the fully online setting. Therefore, in addition to classroom game-play experiments, instructors who teach a fully online course may need to adopt an online discussion forum to enhance students’ interest and engagement in an online course. An online discussion forum would be a convenient platform in a fully online course to offer participants (students and the instructor) an opportunity for interactions between students and their peers as well as students and their instructor, including asynchronous communication by sharing ideas, thoughts, and information. As Troussas et al. (2020) explained, the younger generation is more familiar with new technologies and intrinsically motivated to use Internet technology. For that reason, educators need to effectively utilize the Internet to help strengthen students’ engagement in online discussions in order to improve their online learning.

Although online discussions provide instructors teaching online courses with a new way to communicate with students, they may be the most time-consuming element in efforts to effectively engage students in discussions and respond to every student’s discussion post (Beaudoin, 2002; Rovai, 2007; Ebner and Holzinger, 2005; Davidson-Shivers, 2009; etc.). The question is: Can online discussions facilitate students’ learning in online courses? If instructors do not effectively design online class discussions or/and do not effectively engage in students’ online discussions, then these activities may not benefit students’ learning.

In the past decade, scholars have documented the positive benefits of online discussion forums for students’ learning in online courses (e.g., Cheng et al., 2011; Kwon et al., 2018; Aderibigbe, 2020 and 2021, etc.). Yet those offering these findings are education scholars who have used education courses as cases in their studies. Economics courses are quite different from education courses. Hence, we are not quite sure whether online discussions utilized in education courses would also have the same effect in economics courses. Therefore, the main purpose of this research was to use an economics course, “Introduction to Microeconomics”, to investigate whether online discussions can benefit students’ learning in online economics courses. This study was guided by three research questions:

Q1: Can online discussions significantly benefit students’ online learning if peer-responses are not required, and the instructor does not engage in students’ online discussions?

Q2: Can online discussions significantly benefit students’ online learning if peer-responses are required but the instructor does not engage in students’ online discussions?

Q3: Can online discussions significantly benefit students’ online learning if peer-responses are required, and the instructor also effectively engages in students’ online discussions?

This paper is organized as follows. First, we briefly review selected articles related to this topic and build research hypotheses. Second, we offer details on the development of the methodology, including experimental design, subjects and data, and research method – development of an econometric model for testing hypotheses. Third, we show our findings and explain whether our hypotheses are supported. Fourth, we discuss the results and explain possible reasons for our findings. Finally, we offer summaries and conclusions to wrap up the paper.

A brief literature review

The issue of online discussions and student learning has been studied over the past two decades, and the results have been mixed: either insignificant or significant associations have been found. Researchers, such as Davies and Graff (2005), Palmer, Holt and Bray (2008), and Cho and Tobias (2016), etc., did not find a significant relationship between online discussions and students’ learning achievement in online courses, implying that online discussions might not necessarily benefit students’ learning in online courses.

Davies and Graff (2005) did not find that high interactivity in online discussions positively affected student performance in online learning environments but showed that low interactivity in online discussions could lead to lower performance. Palmer, Holt and Bray (2008) had a similar finding: interactivity in online discussions might not be able to predict students’ achievement in online learning environments, but they emphasized one point — simply replying to and reading peers’ discussion messages without deep thinking would not significantly affect online learning outcomes. According to Cho and Tobias (2016), online discussions did not significantly influence the community of inquiry, time students spent on an online course, students’ satisfaction with the online course, and students’ performance.

However, many studies have demonstrated that online discussion forums can benefit students’ learning in online courses (e.g., Cheng et al., 2011; Comer and Lenaghan, 2012; Kwon et al., 2018; Aderbigbe, 2020; 2021). Students who engage in online discussions have more opportunities to read and respond to their peers’ posts, ensuring their engagement and learning by increasing learning opportunities through access to peers’ comments and responses.

Cheng et al. (2011) studied the effectiveness of a voluntary discussion forum in a fully online setting course and found that students who participated (posing and responding to messages) in online discussion forums performed slightly better on exams than students who did not participate in online discussion forums. In 2012, Comer and Lenaghan conducted a system of “original examples” and “value-added comments” in online classes and found that online asynchronous discussions improved students’ learning. In addition, Kwon et al. (2018) designed an experiment of graphic organizers in online discussions and examined the effects of receiving or generating graphic organizers on students’ engagement in online discussion forum. Their findings suggested that both generating and receiving graphic organizers could enhance students’ engagement in online discussions and further improve students’ interest and benefit their online learning. Moreover, Aderibigbe (2020)’s study suggested that online discussion forum assisted students’ engagement and learning in online courses. But Aderibigbe (2021) further emphasized that if instructors could provide students with clear guidelines and reasonable time to engage with their peers, online discussions could even foster students’ deep learning.

These previous studies offered an important point: students can learn more about topics from their peers’ comments and responses. That is, students can learn from each other. For that reason, lacking peer-responses in an online discussion forum (that is, students only post their discussion messages but do not comment on their peers’ discussion posts) will render an online discussion forum significantly less beneficial to students’ learning. For this reason, our first research hypothesis is as follows:

Hypothesis 1

Even though online discussions are required for students, without peer-responses and instructors’ effective engagement in students’ online discussions, online discussions still cannot significantly benefit students’ online learning.

Moreover, as mentioned above, Palmer, Holt and Bray (2008) pointed out that without opportunities for deeper thought, students’ peer-responses will not help their online learning. In addition to students’ peer-responses, instructors need to participate in students’ discussions through guidance and assistance; otherwise, online discussions will not benefit students’ online learning. Furthermore, scholars like Goshtasbpour, Swinnerton and Pickering (2022) offer a similar suggestion. They provided twelve tips for engaging learners in online discussions. One of their twelve tips emphasizes the importance of instructor presence in discussions. That is, instructors need to effectively engage in students’ online discussions to increase their benefits to students’ online learning.

In addition, it should be noted that many of these previous studies were performed by education or educational psychology scholars who used education or psychology courses as cases in their studies. Economics courses are quite different from education/psychology courses — economics includes many mathematical and abstract concepts. For this reason, economics courses may be more difficult for students, and require instructors’ assistance and guidance to figure out answers. Scholars (e.g., Goshtasbpour, Swinnerton and Pickering, 2022) have shown that a balance between student and instructor comments increases the likelihood of knowledge enhancement. That is, in online discussions instructors’ role is to offer information and knowledge from their teaching and research experiences while students offer more personal perspectives. Hence, without instructors’ effective engagement in students’ online discussions, we do not expect that online discussion would significantly benefit students’ online learning. Therefore, our second research hypothesis is:

Hypothesis 2

Even though online discussions and peer-responses are required for students, without instructors’ effective engagement in students’ online discussions, online discussions still cannot significantly benefit students’ online learning.

As remarked above, economics is a difficult subject that includes a significant number of mathematical and abstract concepts. For that reason, online discussions are most likely to significantly benefit students’ online learning of economics only when instructors effectively engage in online discussions to assist with student understanding of economics topics. Consequently, we develop our third research hypothesis:

Hypothesis 3

Online discussions can significantly benefit students’ online learning only when instructors effectively engage in students’ online discussions.

Methodology

Experimental design

In our experiment we divided students who enrolled in four fully online “Introduction to Microeconomics” courses taught by the same instructor into four groups (Group 1, Group 2, Group 3, and Group 4) in Fall 2020, Spring 2021, Fall 2021, and Spring 2022, respectively. These four groups are described below:

Group 1: The instructor did not require students to engage in online discussions.

Group 2: The instructor required students to engage in online discussions but did not require peer-responses, and the instructor did not respond to students’ discussion posts (i.e., no feedback to students).

Group 3: The instructor required students to engage in both online discussions and peer-responses, but the instructor did not respond to students’ discussion posts (i.e., no feedback to students).

Group 4: The instructor required students to engage in both online discussions and peer-responses, and responded to all discussion posts (i.e., provided feedback to every student).

Group 1 was used as the control group, while the other groups (Group 2, Group 3, and Group 4) were used as the experimental groups. In the experiment, we compared Group 2, Group 3, and Group 4, respectively, with Group 1 (i.e., Group 2 vs. Group 1, Group 3 vs. Group 1, and Group 4 vs. Group 1).

The instructor used the same textbook (covering the same chapters), discussion issues (ten discussion issues), problem sets (i.e., study guides), power point lecture notes, YouTube lectures, review zoom lecture videos for exams (created by the instructor) with all four groups (Group 1 did not have online discussions but had the same other materials as the other three groups).

In addition, the grading policy for each group was the same except for Group 1 — because online discussions were not required in this group, they were not part of their final course grade. Therefore, graded items in Group 1 only included five exams; each exam weights 20% of the final course grade; while graded items in the other three groups (Group 2, Group 3, and Group 4) included both 10 online discussions and five exams. Each online discussion weights 2% of the final course grade and each exam weights 16% of the final course grade in Groups 2–4. After completing two online discussions, there was an exam. That is, each exam covered two discussion issues. All exams were held online and were closed book, and exam materials were not cumulative from previous exam materials. Furthermore, after students completed each online discussion, the instructor posted grades and comments as well as correct answers. The experimental design is summarized in Table 1.

Table 1.

Summary of experimental design.

	Group 1	Group 2	Group 3	Group 4
Online discussions	No	Yes	Yes	Yes
Peer-responses	No	No	Yes	Yes
Feedback from instructor	No	No	No	Yes
Type of experiment	Control	Experimental	Experimental	Experimental
Semester	Fall 2020	Spring 2021	Fall 2021	Spring 2022
Number of participants	33	31	35	30
Number of discussions	0	10	10	10
Number of exams	5	5	5	5
Weight of each discussion	0	2%	2%	2%
Weight of each exam	20%	16%	16%	16%

Each semester was divided into five periods (i.e., five exam periods). The length of each period was 3 weeks. In each period, students engaged in two online discussions in the first 2 weeks and took an exam in the third week. Thus, the “Overall” shown in Tables 2, 3, and 4 reflects the integration of these five periods Moreover, the prerequisite for this course (Introduction to Microeconomics) is “Elementary Algebra” or “Fundamentals of Algebra”. Students enrolled in the online course used Canvas as its online web-based learning management system. The instructor had been certified by Quality Matters (QM).¹

Table 2.

Mean and standard deviation for variables.

Variables	Group 1 (N = 33)	Group 2 (N = 31)	Group 3 (N = 35)	Group 4 (N = 30)
Exam 1	72.61 (19.13)	74.45 (20.04)	72.00 (12.31)	79.60 (14.64)
Exam 2	80.97 (13.71)	80.00 (13.58)	81.14 (10.56)	82.13 (10.06)
Exam 3	75.76 (14.49)	74.06 (13.18)	73.49 (10.57)	78.40 (9.18)
Exam 4	76.36 (15.95)	78.97 (12.94)	79.54 (9.79)	82.93 (11.31)
Exam 5	74.48 (18.96)	73.10 (12.73)	76.17 (10.26)	80.27 (8.65)
Employment hours per week	29.24 (6.51)	30.97 (5.97)	30.86 (6.00)	30.50 (5.92)
Credit hours taken over the whole semester	13.91 (1.96)	13.16 (1.85)	13.03 (1.44)	12.90 (1.39)
Grade point average (GPA)	2.70 (0.64)	2.67 (0.63)	2.77 (0.44)	2.74 (0.45
Dummy variable – Male	0.36 (0.49)	0.32 (0.48)	0.63 (0.49)	0.50 (0.51)
Dummy variable – Kids	0.18 (0.39)	0.23 (0.43)	0.22 (0.42)	0.20 (0.41)
Dummy variable – Algebra	0.45 (0.51)	0.45 (0.51)	0.34 (0.48)	0.47 (0.50)
Dummy variable – Calculus	0.18 (0.39)	0.16 (0.37)	0.17 (0.38)	0.23 (0.43)

Note: Number in parentheses is the standard deviation.

Table 3.

Results of Hypothesis 1: Group 2 vs. Group 1.

Explanatory variables	Explained variable: lnEXA
Explanatory variables	Exam 1 (1)	Exam 2 (2)	Exam 3 (3)	Exam 4 (4)	Exam 5 (5)	Average (6)	Overall (7)
Constant	6.274*** (4.53)	5.361*** (6.73)	5.438*** (5.97)	4.123*** (4.17)	5.775*** (5.10)	5.383*** (7.61)	5.394*** (11.11)
ln GPA	0.628*** (4.06)	0.359*** (4.04)	0.361*** (3.55)	0.528*** (4.79)	0.435*** (3.45)	0.449*** (5.69)	0.462*** (8.53)
ln WHR	−0.409* (−1.93)	−0.284** (−2.32)	−0.219 (−1.57)	−0.059 (−0.39)	−0.298* (−1.72)	−0.249** (−2.30)	−0.25*** (−3.41)
ln CRD	−0.477* (−1.72)	−0.155 (−0.97)	−0.303 (−1.66)	−0.054 (−0.27	−0.357 (−1.57)	−0.262* (−1.85)	−0.27*** (−2.77)
MAL	0.003 (0.05)	0.054 (1.65)	0.046 (1.23)	−0.005 (−0.13)	0.075 (1.61)	0.034 (1.19)	0.034* (1.73)
KID	−0.090 (−1.27)	−0.025 (−0.62)	−0.066 (−1.42)	−0.016 (−0.21)	−0.076 (−1.31)	−0.052 (−1.45)	−0.054** (−2.16)
ALG	−0.029 (−0.45)	0.031 (0.80)	0.079* (1.81)	0.019 (0.39)	−0.046 (−0.85)	0.013 (0.38)	0.010 (0.45)
CAL	0.109 (1.21)	−0.002 (−0.03)	0.056 (0.94)	0.071 (1.09)	0.107 (1.44)	0.063 (1.36)	0.068** (2.15)
OD2	0.012 (0.22)	−0.006 (−0.19)	−0.025 (−0.68)	0.035 (0.88)	0.0001 (0.001)	0.005 (0.17)	0.003 (0.16)
R ²	0.580	0.587	0.552	0.544	0.535	0.715	0.498
${\bar{R}}^{2}$	0.519	0.527	0.486	0.478	0.468	0.673	0.485
F-statistic	9.51	9.77	8.46	8.21	7.92	17.21	38.59
Observations	64	64	64	64	64	64	320

Note: Number in parentheses is t-value; GPA = grade point average; WHR = employment hours per week; CRD = credit hours taken over the whole semester; MAL = dummy variable (male student = 1); KID = dummy variable (living with kids = 1); ALG = dummy variable (Algebra = 1); CAL = dummy variable (calculus = 1); and OD2 = dummy variable (Group 2 = 1). ***p < 0.01; **p < 0.05; *p < 0.10.

Table 4.

Results of Hypothesis 2: Group 3 vs. Group 1.

Explanatory variables	Explained variable: lnEXA
Explanatory variables	Exam 1 (1)	Exam 2 (2)	Exam 3 (3)	Exam 4 (4)	Exam 5 (5)	Average (6)	Overall (7)
Constant	4.320*** (4.07)	5.121*** (9.15)	6.173*** (9.09)	5.708*** (7.80)	4.970*** (5.79)	5.286*** (10.31)	5.258*** (14.10)
ln GPA	0.579*** (4.18)	0.445*** (6.09)	0.338*** (3.81)	0.423*** (4.42)	0.602*** (5.37)	0.469*** (7.00)	0.477*** (9.80)
ln WHR	−0.150 (−0.98)	−0.22*** (−2.77)	−0.34*** (−3.42)	−0.31*** (−2.90)	−0.183 (−1.47)	−0.24*** (−3.22)	−0.24*** (−4.45)
ln CRD	−0.059 (−0.25)	−0.168 (−1.33)	−0.42*** (−2.72)	−0.295* (−1.78)	−0.253 (−1.30)	−0.245** (−2.11)	−0.24*** (−2.83)
MAL	0.086* (1.71)	0.062** (2.34)	0.001 (0.02)	−0.057 (−1.64)	0.066 (1.60)	0.027 (1.12)	0.032* (1.77)
KID	−0.008 (−0.13)	−0.004 (−0.12)	−0.048 (−1.22)	−0.055 (−1.31)	−0.011 (−0.22)	−0.028 (−0.96)	−0.025 (−1.17)
ALG	−0.043 (−0.66)	0.002 (0.04)	0.066 (1.58)	0.035 (0.77)	−0.100* (−1.89)	−0.002 (−0.05)	−0.008 (−0.36)
CAL	0.086 (1.02)	−0.003 (−0.06)	0.070 (1.31)	0.081 (1.40)	0.118* (1.74)	0.065 (1.60)	0.071** (2.39)
OD3	−0.030 (−0.58)	−0.023 (−0.83)	−0.032 (−0.95)	0.063* (1.75)	−0.008 (−0.19)	−0.005 (−0.19)	−0.006 (−0.33)
R ²	0.447	0.644	0.560	0.561	0.581	0.711	0.464
${\bar{R}}^{2}$	0.372	0.596	0.501	0.502	0.524	0.672	0.451
F-statistic	5.95	13.36	9.39	9.44	10.22	18.15	35.80
Observations	68	68	68	68	68	68	340

Note: Number in parentheses is t-value; GPA = grade point average; WHR = employment hours per week; CRD = credit hours taken over the whole semester; MAL = dummy variable (male student = 1); KID = dummy variable (living with kids = 1); ALG = dummy variable (Algebra = 1); CAL = dummy variable (calculus = 1); and OD3 = dummy variable (Group 3 = 1). ***p < 0.01; **p < 0.05; *p < 0.10.

Subjects and data

The total number of subjects included 129 business undergraduate students who enrolled in fully online “Introduction to Microeconomics” classes (33 students in Fall 2020, 31 students in Spring 2021, 35 students in Fall 2021, and 30 students in Spring 2022). Students who enrolled in Fall 2020 belonged to Group 1 and therefore participated in the control group; students enrolled in Spring 2021, Fall 2021, and Spring 2022 belonged to Group 2, Group 3, and Group 4, respectively, and thus participated in the three different experimental groups. It should be noted that our sampling method was non-probability sampling that involved non-random selection based upon students’ self-selection for the section and the semester they enrolled. We did not randomly select students from these 129 students to assign them to each one of these four groups.

The data used in this study consisted of both instructor-reported data and student-self-reported data. In addition, this research was approved by the Institutional Review Board (IRB) and was granted exempt due to no greater than minimal risk to subjects and fitting into exempt Category 1: education practice. Data sources and measurements are described below. The instructor-reported data included the following:

1. Exam Scores (EXA). Five exam scores for each group were recorded and provided by the instructor. This variable was used as a proxy to reflect a student’s online learning achievement.

2. Grade Point Average (GPA). Each student’s GPA was offered by the Registrar’s Office. This variable was used as a proxy to reflect a student’s scholarly quality and motivation to learn the course.

3. Credit hours taken over the whole semester (CRD). This information was provided from the Registrar’s Office. This variable was used a proxy to reflect a student’s opportunity cost in taking the course. Another variable that also can identify a student’s opportunity cost in taking the course is employment hours per week. The information of employment hours per week needs to be collected via student-self-report on a questionnaire. We will describe the questionnaire later.

4. Gender (GEN). This was a dummy variable. We set male (MAL) as 1 and female as 0. The instructor provided this information. This variable was included in the research because some empirical evidence (e.g., Woodfield et al., 2006; Cortright et al., 2011; Paisey and Paisey, 2004) showed that gender could influence a student’s learning performance.

5. Intermediate Algebra (ALG) and Calculus (CAL). These two variables were dummy variables. We set “Completed” as 1, and “Never completed” as 0. The information of these two variables were obtained from the Office of Registrar. Although “Elementary Algebra” or “Fundamentals of Algebra” was required as a prerequisite for “Introduction to Microeconomics”, a stronger math background was bonus for students’ learning economics. For that reason, these two variables were used proxies to reflect a student’s stronger math background.

6. Online Discussions vs No Online Discussions (ODI). This was a dummy variable. In this research, we compared online discussions with no online discussions. Thus, we set 1 as “students in the online discussion groups” (OD2 = Online Discussion Group 2, OD3 = Online Discussion Group 3, and OD4 = Online Discussion Group 4), and 0 as “students in the no online discussion group”.

Moreover, the student-self-reported data included two variables: total employment hours per week (WHR) and living with young kids (KID). The variable of “total employment hours per week” was another variable used as a proxy to identify a student’s opportunity cost in taking the course. The variable of “living with young kids” was a dummy variable. We set “Yes, living with young kids whose age is below 10 years old” as 1, and “No, do not live with young kids whose age is below 10 years old” as 0. We included this variable because several students were non-traditional (older students with a family that included young children aged less than 10 years old). Young children may require more parental attention and care, which may limit a student’s time for studying and hence influence the student’s academic performance.

To gather information on “total employment hours per week” and “living with young kids”, a questionnaire was provided to students. The survey was sent to students via email and completed at the beginning of the semester. A brief statement of asking students for consent was included in the email. Students were notified that they could choose whether or not to fill out the survey – nonparticipation would not affect their final course grades. The short questionnaire was as follows:

1. Are you working for pay this semester? Yes: ___; No: ___. If “yes”, how many hours approximately per week are you working for pay? ____

2. Do you live with young children whose ages are below 10 years old? Yes: ___; No: ___.

We were not able to identify the reliability of students’ responses via survey. While there could be a very small probability that some students might not accurately respond, there was no way to control for this situation. Luckily, these two variables were not primary factors in the research and thus would not significantly influence the main results.

Table 2 reports descriptive statistics (mean and standard deviation) for variables used in the research. Additionally, Cronbach’s alphas were 0.9296, 0.8793, 0.7458, and 0.7191 for Groups 1– 4, respectively, indicating either strong or high consistency among these exams.

Research Method

When designing our research method, we adopted econometric analysis which will verify whether our three research hypotheses could be supported in the research. Below, we show the basic framework for a student’s learning performance:

E X A = f (G P A, W H R, C R D, M A L, K I D, A L G, C A L, O D I) .

The determinants are not an assemblage of suppositions. They are fundamentally based upon empirical findings from previous studies (e.g., Frank, 1997; Cohn and Johnson, 2006; Dickie, 2006; Lin (2013, 2016, 2018)).

A student’s exam performance was modeled as a transcendental function. Therefore, using the transcendental model, Equation (1) can be written as below:

E X A = a_{0} {(G P A)}^{α_{1}} {(W H R)}^{α_{2}} {(C R D)}^{α_{3}} e^{α_{4} (M A L) + α_{5} (K I D) + α_{6} (A L G) + α_{7} (C A L) + α_{8} (O D I)} .

(2)

We constructed the model to test Hypotheses 1–3 by taking natural logarithms of both sides of Equation (2). The exam performance function becomes linear, and thus the regression model can be displayed as follows:

\ln E X A_{t} = α_{0} + α_{1} \ln G P A + α_{2} \ln W H R + α_{3} \ln C R D + α_{4} M A L + α_{5} K I D

+ α_{6} A L G + α_{7} C A L + α_{8} {O D I}_{i} + ε_{1},

(3)

where α₀=lna₀, t = 1, 2, 3, 4, 5, whole semester; EXA_t = exam scores on Period t; GPA = grade point average; WHR = total employment hours for pay per week; CRD = total credit hours taken in the semester; MAL = male students (dummy variable); KID = living with young kids whose age is below 10 years old (dummy variable); ALG = completed intermediate algebra (dummy variable); CAL = completed calculus (dummy variable); ODI_i = students in the online discussion group i (i = 2, 3, or 4); and ε₁ = stochastic disturbance with a mean 0 and a variance σ².

In this formulation, the null hypothesis is that parameter α₈ = 0, while the alternative hypothesis is that parameter α₈ ≠ 0. In other words, if online discussions can significantly benefit students’ online learning, then α₈>0 and the effect should be significant.

Results

Hypothesis 1

The results for Online Discussion Group 2 vs. No Online Discussion Group 1 for equation (3) are presented in Table 3. As reported in that table, the null hypothesis that online discussions in Group 2 was not related to students’ exam performance was not rejected because online discussions in Group 2 (OD2, dummy variable) did not exert a statistically significant effect on student’s exam performance at any level (i.e., 10%, 5%, or 1% level) for each exam, average of five exams, and overall. (Note: “Average” shown in Tables 2, 3, and 4 means the average of all five exams.)

As a result, Hypothesis 1 was supported. Even though online discussions are required for students, without peer-responses and instructors’ effective engagement in students’ online discussions, online discussions still cannot significantly benefit students’ online learning.

Hypothesis 2

The results for Online Discussion Group 3 vs. No Online Discussion Group 1 for Equation (3) are reported in Table 4. As shown in that table, online discussions in Group 3 (OD3, dummy variable) did not have a statistically significant effect on students’ exam performance at any level (i.e., 10%, 5%, or 1% level) for Exam 1, Exam 2, Exam 3, Exam 5, Average, and Overall. However, online discussions in Group 3 (OD3, dummy variable) had a statistically significant effect on students’ exam performance at the 10% level for Exam 4.

While the 10% level is significant, it is weakly, and not strongly, significant. In addition, only Exam 4 had a significant effect at the 10% level, but all others (Exam 1, Exam 2, Exam 3, Exam 5, Average, and Overall) did not show any level of significance. The significant effect at the 10% level for Exam 4 might or might not necessarily result from the effect of online discussions. However, based upon this weak evidence, we cannot confidently claim that the null hypothesis that online discussions in Group 3 were not related to student learning performance was rejected. Therefore, we will claim that the null hypothesis was not rejected.

Consequently, Hypothesis 2 was supported. Even though online discussions and peer-responses are required for students, without instructors’ effective engagement in students’ online discussions, online discussions still cannot significantly benefit students’ online learning.

Hypothesis 3

The results for Online Discussion Group 4 vs. No Online Discussion Group 1 for Equation (3) are shown in Table 5. As displayed in that table, online discussions in Group 4 (OD4, dummy variable) did not provide a statistically significant effect on students’ exam performance at any level (i.e., 10%, 5%, or 1% level) for Exam 1, Exam 2, and Exam 3, but it exerted a statistically significant effect at the 10% level for Exam 4, Exam 5, and Average, and at the 1% level for Overall.

Table 5.

Results of Hypothesis 3: Group 4 vs. Group 1.

Explanatory variables	Explained variable: lnEXA
Explanatory variables	Exam 1 (1)	Exam 2 (2)	Exam 3 (3)	Exam 4 (4)	Exam 5 (5)	Average (6)	Overall (7)
Constant	5.658*** (4.86)	5.778*** (7.92)	0.577*** (7.07)	5.173*** (4.99)	5.034*** (4.88)	5.501*** (8.15)	5.443*** (12.45)
ln GPA	0.605*** (4.44)	0.350*** (4.09)	0.339*** (3.67)	0.368*** (3.03)	0.474*** (3.91)	0.419*** (5.29)	0.427*** (8.33)
ln WHR	−0.310* (−1.81)	−0.31*** (−2.93)	−0.188 (−1.62)	−0.225 (−1.47)	−0.203 (−1.33)	−0.252** (−2.53)	−0.25*** (−3.85)
ln CRD	−0.347 (−1.40)	−0.250 (−1.60)	−0.377** (−2.23)	−0.184 (−0.83)	−0.204 (−0.92)	−0.284* (−1.97)	−0.27*** (−2.91)
MAL	0.014 (0.30)	−0.015 (−0.52)	0.039 (1.24)	0.040 (0.94)	0.044 (1.05)	0.023 (0.82)	0.024 (1.37)
KID	−0.091 (−1.44)	−0.047 (−1.18)	−0.054 (−1.26)	−0.127** (−2.25)	−0.098* (−1.75)	−0.081** (−2.22)	−0.08*** (−3.51)
ALG	−0.103* (−1.79)	−0.019 (−0.54)	0.047 (1.20)	0.070 (1.36)	−0.027 (−0.52)	−0.001 (−0.04)	−0.006 (−0.30)
CAL	0.057 (0.75)	−0.032 (−0.66)	−0.015 (−0.30)	−0.020 (−0.30)	0.058 (0.86)	0.003 (0.08)	0.009 (0.33)
OD4	0.074 (1.51)	0.009 (0.29)	0.014 (0.41)	0.075* (1.73)	0.076* (1.74)	0.048* (1.67)	0.050*** (2.69)
R ²	0.572	0.552	0.456	0.445	0.521	0.652	0.451
${\bar{R}}^{2}$	0.508	0.485	0.375	0.362	0.450	0.600	0.436
F-statistic	9.01	8.31	5.66	5.40	7.34	12.64	31.40
Observations	63	63	63	63	63	63	315

Note: Number in parentheses is t-value; GPA = grade point average; WHR = employment hours per week; CRD = credit hours taken over the whole semester; MAL = dummy variable (male student = 1); KID = dummy variable (living with kids = 1); ALG = dummy variable (Algebra = 1); CAL = dummy variable (calculus = 1); and OD4 = dummy variable (Group 4 = 1). ***p < 0.01; **p < 0.05; *p < 0.10.

Although Exam 1, Exam 2, and Exam 3 did not show a significant effect at any level of significance, others (Exam 4, Exam 5, and Average) exerted a significant effect at the 10% level. More importantly, Overall had a strongly significant effect at the 1% level. Comparing these results with the previous results in Hypothesis 2, the evidence from Hypothesis 3 was more significant than the evidence from Hypothesis 2. For that reason, we are more confident in claiming that the null hypothesis that online discussions in Group 4 were not related to student learning performance was rejected. In the next section, we will discuss possible reasons for why the significant effect could not start at Exam 1 or earlier.

Therefore, we may conclude that Hypothesis 3 was supported. Online discussions can significantly benefit students’ online learning only when instructors effectively engage in students’ online discussions.

Discussion

According to our empirical evidence, online discussions do not seem to significantly benefit students’ online learning. Even though instructors effectively engaged in students’ online discussions, the benefit to students was not as great as initially expected. Therefore, we need to examine the reasons that these benefits are not as significant as expected. We raise two issues for discussion:

1. Without the instructor’s effective engagement in students’ online discussions, why couldn’t online discussions significantly benefit students’ online learning even though peer responses were required?

2. With the instructor’s effective engagement in students’ online discussions, why couldn’t the significant effect start at Exam 1 or earlier (it started later, with Exam 4), and the significant effect was only at the 10% level (weak significance) for Exams 4 and 5?

Below, we discuss these two issues.

Issue 1

The instructor reported effectively engaging in students’ online discussions. The instructor created discussion issues based on the textbook chapters focusing on critical thinking that cover three skills: conceptual, definition, and analytical. The instructor gave students constructive feedback, clear guidelines, and reflective questions. The instructor’s feedback included constructive comments and follow-up reflective questions for students based on their initial discussion posts. Students had to answer the instructor’s follow-up questions to receive full credit. This requirement gave students an opportunity to re-think discussion issues and re-read the textbook chapter.

Of course, students’ answers and reasons might lack perspective and knowledge. Therefore, the instructor not only offered students brief comments but also used the method of follow-up questions to guide students’ understanding of the issue and methods to follow in providing reasonable and correct explanations. Since each student might offer different answers and explanations, the instructor provided different comments and follow-up questions to each student. Certainly, responding to each student’s discussion post did take the instructor a significant amount of time — approximately 6–7 h for the whole class on one discussion issue, according to the instructor’s report. While both students and the instructor needed to spend more time to complete one discussion, doing so would benefit students’ learning on the topic, ensuring that they gain more knowledge. The instructor also received better evaluations from students, based on the instructor’s report. This result is consistent with the study done by Lin (2022) on the economic reciprocal relationship between student learning and professor teaching. A greater number of assignments for students increases the instructor’s workload, such that both will devote more time and effort to completing their jobs. Further, both will receive higher payoffs (i.e., students gain more knowledge, and the instructor receives better evaluations) eventually.

However, on those occasions where the instructor did not provide feedback (comments and follow-up questions), students did not have the benefit of the instructor’s guidance on understanding an issue and opportunities to provide reasonable and correct reasons. Students also did not have opportunities to re-think discussion topics and re-read the textbook chapter. Even when peer-responses were required, their availability did not help students’ learning. There were several reasons for this, including, most importantly, that since the students themselves also did not understand a topic due to a lack of knowledge, they were unable to provide valuable and correct comments and constructive feedback. Some students’ peer-responses were very short — like “hi Steve, I like your post. I had the same experience as you did…”, “I agree with you…”, “Hello John, … great post”, “Hey Helen, … you did a great job”, etc. These types of peer-responses do not benefit peer learning at all. In addition, in the group lacking instructor feedback and encouragement to engage in original thinking, the instructor normally gave students full credit whether their answers were correct or incorrect. When the instructor posted students’ grades, the instructor also posted comments and correct answers. Did students carefully read the instructor’s comments and correct answers? We are skeptical that they did so. They were equally unlikely to do so because exam questions were not the same as discussion issues.

Our explanation regarding our findings implies that the interaction between the instructor and the students, as well as the encouragement and affirmation in the feedback increased the students’ motivation to learn and thus improved their performance. Abundant empirical evidence (e.g., Brophy, 1981; Kern and Clemens, 2007; Hawkins and Heflin, 2011; Mamoon-Al-Bashir, Kabir and Rahman, 2016; Gan et al. 2021; Markovic, 2023, etc.) has shown that teacher’s positive feedback/praise, instant feedback, and encouragement are a powerful tool to motivate students to learn and hence enhance their academic performance. Moreover, numerous empirical evidence (e.g., Nugent, 2009; Akhtar et al., 2019; Rahman et al., 2020, etc.) has demonstrated that the impact of teacher-student interaction on student motivation and achievement is positive and significant, meaning that the more positive interactions between the teacher and the students, the higher the motivation and achievement the students will have. Those past studies mentioned above potently support our explanation regarding our findings.

Above all, we realize that the instructor’s role rather than the online discussion format indeed is key to student learning achievement in online courses, because the instructor spent extra 6–7 h on the class design for each discussion issue, including reading every student’s discussion post, writing instant and constructive feedback (comments and follow-up questions) to every student, and interacting with every student. Therefore, based upon our explanation above, without the instructor’s effective engagement in students’ online discussions, online discussions could not significantly benefit students’ online learning even though peer responses were required.

Issue 2

According to the evidence from Table 4, the positive and significant effect began with Exam 4 and the significant effect was only at the 10% level (weakly significant). Evidently, then, it took time for “the instructor’s engagement” to have a positive and significant effect on students’ exam performance, and it was not even strongly significant, such as at the 5% or 1% level. Below, we offer possible reasons.

First, although the instructor required students to answer follow-up questions, many students did not carefully answer them, and some even completely ignored these questions. Student who did not answer follow-up questions only lost 10 points. That is, those students who completely ignored the follow-up questions still could receive 90 points for the discussion. Each discussion only weights 2% of the final course grade, which means that the difference between 100 points and 90 points was just the difference between 2% and 1.8% — quite small and of no real significance to the final course grade. Students who discovered this would not spend time answering follow-up questions. This is an interesting choice because as long as students answered the follow-up questions, whether correctly or incorrectly, they still could receive full points unless the responses were poor quality, in which case the instructor just took off a few points (1–5 points). This still would not significantly affect students’ final course grades and meant that many students chose not to spend time re-thinking a course-related issue and re-reading the textbook chapter to find correct answers. In other words, the benefit was not as great as initially expected.

However, for students who continue to miss discussions and/or ignore follow-up questions, the loss of points would become large enough to affect their final course grades. For example, if students missed the first five discussions, they would lose 10% (= 2% x 5) from their final course grade, significantly lowering their grade one letter grade, say from B to C. Or, if students continued to ignore follow-up questions for the first five discussion issues, they would lose 1% (= 0.2% x 5) of their final course grade, possibly lowering their grade from B- to C+, for example. Students who realized the importance of discussions to their final course grades began to take them more seriously. Those who seriously engaged in online discussions and answered follow-up questions received the benefit of the instructor’s effective engagement. This could explain why the significant effect started with the later exam, Exam 4, rather than with Exam 1 or earlier, and why the significance level was only at the weaker 10% level rather than the stronger level, such as the 5% or 1% level.

Moreover, as mentioned earlier, the exam questions and the discussion issues were different. Therefore, students’ engagement in discussions did not directly benefit their exam performance. In other words, students who did not effectively utilize the opportunity to learn about topics and gain knowledge and take advantage of the instructor’s effective engagement in students’ online discussions, were less likely to perform well on exams and receive good course grades.

Furthermore, the reduced significant effect could possibly result from the instructor. For example, the instructor might not spend sufficient time designing discussion questions that motivated students to learn about topics or directly influenced students’ exam preparation. For example, an economics instructor (not the same instructor involved in this research) wrote the following discussion question — “Describe how economics reflects on your life”. This type of question is too broad: it does not focus on a specific topic, making discussion difficult, and does not reflect exam questions. Students who receive these types of questions may struggle to learn about a topic because the discussions related to it are too broad or unhelpful. Additionally, the use of exam performance as a proxy for students’ online learning performance may not be useful because it may not precisely mirror a student’s learning performance. Therefore, it might be an alternative idea that we could use the instructor’s observation of students’ online discussions, motivation, and participation as part of the qualitative results to complement the quantitative results and to better study the issue and the expressions of different groups of students, which might avoid the problem of using exam grades as the only variable to be measured for students’ online learning performance. Doing so would possibly make our conclusions more reliable and comprehensive.

Consequently, as shown in our explanations, the instructor’s effective engagement in students’ online discussions did not exert a significant effect at Exam 1 or earlier (it started later at Exam 4) and was only significant at the 10% level (weak significance) for Exams 4 and 5.

Conclusion

In this research, we investigated whether online discussions can significantly benefit students’ learning in online courses. We designed an experiment in which we used “Introduction to Microeconomics” as a case study. Our empirical evidence demonstrated that online discussions can significantly benefit students’ exam performance in online courses only when instructors effectively engaged in students’ online discussions. Absent this engagement, online discussions did not exert a significant effect on exam performance even though peer-responses were required. In other words, the engagement of the instructor played an essential role in students’ online discussions, because it facilitated the students’ thinking and reading of the textbook and thus increased students’ learning performance. In addition, although the effect was positive and significant when instructors effectively engaged in students’ online discussions, the significant effect was weakly significant (at the 10% level of significance) and began with the later exam (Exam 4) rather than the first exam or earlier. Here, we provide thorough and detailed discussions and offer possible reasons for these findings.

In the discussion, we identified several possible reasons for the restraint in the positive effect. These reasons are briefly summarized below:

1. Since each discussion issue weighted less for the final course grade, several students exhibited little interest in the online discussions. They either did not participate or participated but ignored the instructor’s constructive feedback by not responding to the instructor’s follow-up questions or did not seriously respond to them.

2. Since exam questions were not the same as online discussion questions, students did not use online discussions to prepare for exams and did not read the instructor’s comments and correct answers to learn about topics and gain knowledge.

3. If online discussion issues were not well designed by the instructor, students did not find them interesting and useful.

4. Exam performance might not be a perfect proxy and/or precisely reflect a student’s learning performance.

The evidence still showed that online discussions could significantly benefit students’ learning in online courses when instructors effectively engaged in students’ online discussions; however, the benefit was not as large as what we initially expected. Therefore, to enlarge the benefit, instructors may need to add more weights for online discussions to the final course grade and write some exam questions that reflect the online discussion questions. Doing so would improve students’ engagement in online discussions, including participating in the discussions and answering instructors’ follow-up questions.

Furthermore, as mentioned earlier in the introduction section, online discussions would be the most time-consuming aspect of instructors’ activities if they engaged fully and effectively in students’ discussions and responded to each student’s discussion post. According to the instructor’s report, it took the instructor approximately 6–7 h to provide student feedback on each discussion issue. When the class size is around 30–35 students, the instructor needs about 6–7 h on each discussion issue. What if the class size is around 60–70 students? We may expect that it will take the instructor more than 12–13 h to respond to each student’s discussion post. If the instructor teaches 3–4 online classes each semester, what will the cumulative effect be on the instructor? With this level of class time and effort, it’s no wonder that so many instructors choose not to respond to students’ discussion posts or just briefly respond to a random selection. Our evidence shows that no feedback or just brief feedback to a random group will not have a significant effect on students’ exam performance, implying that online discussions may not benefit students’ learning in online courses and would probably just waste students’ time.

Therefore, to ensure that online discussions are helpful to students engaging in online learning, class size is one very important factor to consider. Based on the instructor’s experience regarding time spent on individual student feedback, we would highly suggest that the optimal size of an online class not exceed 20 students. Limiting class size would allow instructors to take good care of each student, including their discussion posts, effectively engage in students’ online discussions, and significantly facilitate students’ learning in online courses.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Tin-Chun Lin

Note

Author biography

Tin-Chun Lin is a Full Professor of Economics at Indiana University–Northwest, USA. He serves as an Editor-in-Chief for International Journal of Economic Issues; Asian Journal of Arts, Humanities and Social Studies; Studies in Economics and International Finance; and Asian Journal of Economics and Finance; and as a Guest Editor for Sustainability.

References

Aderibigbe

(2020) Online Discussions as an Intervention for Strengthening Students' Engagement in General Education. JOItmC 6(4): 98. DOI: 10.3390/joitmc6040098.

Aderibigbe

(2021) Can online discussions facilitate deep learning for students in general education? Heliyon 7(3), e06414. DOI: 10.1016/j.heliyon.2021.e06414.

Akhtar

Hussain

Afzal

, et al. (2019) European Academic Research 7(2): 1201–1222.

Beaudoin

(2002) Learning or lurking? The Internet and Higher Education 5(2): 147–155. DOI: 10.1016/s1096-7516(02)00086-6.

Brophy

(1981) Teacher praise: A functional analysis. Review of Educational Research 51(1): 5–32. DOI: 10.3102/00346543051001005.

Cheng

Paré

Collimore

, et al. (2011) Assessing the effectiveness of a voluntary online discussion forum on improving students' course performance. Computers & Education 56(1): 253–261. DOI: 10.1016/j.compedu.2010.07.024.

Cho

M-H

Tobias

(2016) Should instructors require discussion in online courses? Effects of online discussion on community of inquiry, learner time, satisfaction, and achievement. The International Review of Research in Open and Distributed Learning 17(2): 123–140. DOI: 10.19173/irrodl.v17i2.2342.

Cohn

Johnson

(2006) Class attendance and performance in principles of economics. Education Economics 14(2): 211–233. DOI: 10.1080/09645290600622954.

Comer

Lenaghan

(2012) Enhancing Discussions in the Asynchronous Online Classroom. Journal of Management Education 37(2): 261–294. DOI: 10.1177/1052562912442384.

10.

Cortright

LujanLujan

Cox

, et al (2011) Does sex (female versus male) influence the impact of class attendance on examination performance? Advances in Physiology Education 35(4): 416–420. DOI: 10.1152/advan.00021.2011.

11.

Davidson-Shivers

(2009) Frequency and types of instructor-interactions in online instruction. Journal of Interactive Online Learning 8(1): 23–40.

12.

Davies

Graff

(2005) Performance in e-learning: Online participation and student grades. Br J Educ Technol 36(4): 657–663. DOI: 10.1111/j.1467-8535.2005.00542.x.

13.

Dickie

(2006) Do classroom experiments increase learning in introductory microeconomics? The Journal of Economic Education 37 (3): 267– 288. DOI: 10.3200/jece.37.3.267-288.

14.

Ebner

Holzinger

Catarci

(2005) Lurking: An underestimated human-computer phenomenon. IEEE Multimedia 12(4): 70–75. DOI: 10.1109/mmul.2005.74.

15.

Frank

(1997) The Impact of Classroom Experiments on the Learning of Economics: an Empirical Investigation. Economic Inquiry 35 (4): 763–769. DOI: 10.1111/j.1465-7295.1997.tb01962.x.

16.

Gan

Liu

(2021) Teacher Feedback Practices, Student Feedback Motivation, and Feedback Behavior: How Are They Associated With Learning Outcomes? Front. Psychol. 12: 1–14. DOI: 10.3389/fpsyg.2021.697045.

17.

Goshtasbpour

Swinnerton

Pickering

(2022) Twelve tips for engaging learners in online discussions. Medical Teacher 44(3): 244–248. DOI: 10.1080/0142159x.2021.1898571.

18.

Hawkins

Heflin

(2011) Increasing Secondary Teachers' Behavior-Specific Praise Using a Video Self-Modeling and Visual Performance Feedback Intervention. Journal of Positive Behavior Interventions 13(2): 97–108. DOI: 10.1177/1098300709358110.

19.

Kern

Clemens

(2007) Antecedent strategies to promote appropriate classroom behavior. Psychol. Schs. 44: 65–75. DOI: 10.1002/pits.20206.

20.

Kwon

Shin

Park

(2018) Effects of graphic organizers in online discussions: comparison between instructor-provided and student-generated. Education Tech Research Dev 66: 1479–1503. DOI: 10.1007/s11423-018-9617-7.

21.

Lin

(2013) Student performance and subsequent effort increment investment: Do students behave like producers? International Journal of Education Economics and Development, 4(3): 219–232.

22.

Lin

(2016) Effects of different types of quizzes on student effort investment behavior and learning outcomes. International Journal of Education Economics and Development, 7(1/2): 33–52.

23.

Lin

(2018) Using classroom game-play in introductory microeconomics to enhance business student Learning and Lecture Attendance. Journal of Education for Business, 93(7): 294–302.

24.

Lin

Dunphy

(2013) Using the crossword puzzle exercise in introductory microeconomics to accelerate business student learning. Journal of Education for Business, 88: 88–93.

25.

Lin

(2022) Modeling the economic reciprocal relationship between student learning and professor teaching: An economic theoretical study and application. International Journal of Education Economics and Development, 13(3): 191–215.

26.

Mamoon-Al-Bashir

Kabir

Rahman

(2016) The value and effectiveness of feedback in improving students’ learning and professionalizing teaching in higher education. Journal of Education and Practice, 7(16): 38–41.

27.

Markovic

(2023). Why Giving Instant Feedback Is Important for Effective Learning. Edu Me. London, UK and Santa Monica, CA, USA. https://www.edume.com/blog/role-of-feedback-in-improving-learning.

28.

Morgan

Sharp

Grogan

(2020) So you want to run a classroom experiment online? The good, the bad, and the different. Applied Economics Teaching Resources 2(5): 1–11.

29.

Nugent

(2009) The Impact of Teacher-Student Interaction on Student Motivation and Achievement. Electronic Theses and Dissertations, 2004–2019. https://stars.library.ucf.edu/etd/3860.

30.

Paisey

(2004). Student attendance in an accounting module – reasons for non-attendance and the effect on academic performance at a Scottish University. Accounting Education, 13 (Supplement 1), 39–53. DOI: 10.1080/0963928042000310788.

31.

Palmer

Holt

Bray

(2008) Does the discussion help? The impact of a formally assessed online discussion on final student results. British Journal of Educational Technology 39(5): 847–858. DOI: 10.1111/j.1467-8535.2007.00780.x.

32.

Rahman

Ali

Faiz

, et al. (2020) Impact of teacher–student interaction on student motivation and achievement. Globus Journal of Progressive Education 10(2): 1–6. DOI: 10.46360/globus.edu.220201002.

33.

Rovai

(2007) Facilitating online discussions effectively. The Internet and Higher Education 10(1): 77–88. DOI: 10.1016/j.iheduc.2006.10.001.

34.

Troussas

Krouska

Sgouropoulou

(2020) Collaboration and fuzzy-modeled personalization for mobile game-based learning in higher education. Computers & Education 144, 103698. DOI: 10.1016/j.compedu.2019.103698.

35.

Woodfield

Jessop

McMillan

(2006) Gender differences in undergraduate attendance rates. Studies in Higher Education 31(1): 1–22. DOI: 10.1080/03075070500340127.