Emotions,Motivation,Cognitive–Metacognitive Strategies,and Behavior as Predictors of Learning Performance in Blended Learning

Abstract

Several studies have focused on identifying the significant behavioral predictors of learning performances in web-based courses by examining the log data variables of learning management systems, including time spent on lectures, the number of assignments submitted, and so forth. However, such studies fail to quantify the impact of emotional, motivational, behavioral, and cognitive–metacognitive factors simultaneously. This research was an attempt to understand the relations between students’ motivation, cognitive–metacognitive strategies, behavior, and learning performance in the context of blended courses in higher education. Then, relevant predictors are used to obtain a model to classify the students’ performance and to identify those who are at risk of failing the course. The authors conducted an empirical study in a higher educational course with 137 Mexican students. Nineteen variables related to emotions, motivation, cognitive–metacognitive strategies, and behavior. Only six were found to be significant. These variables explain approximately 67% of the variance between each student’s overall grade. The model, based on those variables, correctly classifies 96% of the students.

Keywords

blended learning higher education emotions motivation cognitive–metacognitive strategies learning performance

Introduction

Efficiency levels in higher education have been criticized for their graduation and abandonment rates. The Internet has become an important tool to increase the number of enrolled students in higher education. Thus, online and blended learning are two solutions to the issues of graduation and abandonment rates. Approximately 504,000 Mexican students pursue a bachelor’s degree by means of online or blended programs (Secretaría de Educación Pública, 2015). However, only 42% of students conclude their career on time (Asociación Nacional de Universidades e Instituciones de Educación Superior, 2015). Students fail or drop out of the courses. This delays their graduation or, in the worst-case scenario, causes them to abandon their studies.

Most universities around the world offer online courses and careers. Learning management systems (LMS) share learning resources through a system to monitor student learning progress (You, 2015). Several empirical studies have examined LMS log data variables (in the context of distance learning), including time spent on lectures, the number of assignments submitted, and so forth. However, the studies fail to quantify the impact of emotional, motivational, and cognitive–metacognitive factors at the same time.

Research has recognized the importance of emotions in educational settings. Several studies have focused on emotions’ relationships to learning achievements. According to Pekrun (2006) and Pekrun, Frenzel, Goetz, and Perry (2007), emotions related to learning activities and outcomes can impact learning achievements. Based on the control-value theory, Pekrun (2006) proposed a theoretical framework to describe the effects of these emotions on cognitive resource, motivation, self-regulation, strategy, and achievements. Motivation plays an important role in learning achievements. It is also a variable influenced by the educational context, subject area, and task (Linnenbrink-Garcia, Patall, & Pekrun, 2016). Motivation can deactivate negative emotions. On the contrary, negative emotions, including test anxiety, can prevent failure by increasing intrinsic motivation and the effort. The relationship between emotions and motivation is ambivalent and complex (Hembree, 1988; Pekrun et al., 2007).

The goal of this research is to understand relationships between students’ emotions, motivation, cognitive–metacognitive strategies, behavior, and learning performance in the context of blended learning in higher education. Relevant predictors are to obtain a model to forecast students’ performance and identify those who are at risk of failing the course. In the next section, the authors provide a description of relevant literature. Later, the method and results are presented. Finally, the discussion, conclusions, and further topics for investigation are considered.

Related Works

Blended Learning on LMS

Blended learning combines online and face-to-face instruction (Rooney, 2003; Young, 2002). Thus, this method emphasizes the central role of computer-based technologies. Face-to-face learning environments rely on human–human interaction. Distributed learning environments place an emphasis on learner–material interactions (Graham, 2005). Learner–material interactions can easily be tracked by means of LMS. Thus, several distance learning research studies have focused on these interactions. On the other hand, emotions, motivations, and cognitive–metacognitive strategies have not been considered.

Most universities around the world have adopted LMS (such as Blackboard and Moodle) in blended courses to facilitate content distribution and monitor students’ learning progress (You, 2015). Also, the teacher can create a learning sequence (Hirumi, 2002). This means that the sequence in which students are invited to review the course material modules is suggested but not required (Graham, 2005). Other advantages include the cultivation of students’ self-discipline (Harding, Kaczynski, & Wood, 2005) and the enhancement of students’ motivation in comparison with traditional face-to-face classes (Woltering, Herrler, Spitzer, & Spreckelsen, 2009).

Emotions, Motivation, and Regulation

Emotion is a feeling directed toward a real or unreal person, thought, or situation (Pons, Rosnay, & Cuisinier, 2011). Matthews (1997) stated that emotion may be conceptualized as a dependent variable (influenced by processes such as appraisal) or an independent variable (influencing information processing and cognition).

Research shows the importance of emotions in educational settings. Pekrun (2006) and Pekrun et al. (2007) identified a set of emotions related to educational activities and outcomes. These emotions (or achievement emotions) are the rationale of the control-value theory. Achievement emotions are organized in a three-dimensional taxonomy (Pekrun, Goetz, Titz, & Perry, 2002): (a) valence (positive or negative), (b) object focus (activity or outcome), and (c) level of activation implied (activating or deactivating). According to the control-value theory, achievement emotions are instigated by the appraisal of both control and value. For example, a student will enjoy a topic if they are interested in linear regression and feel capable of handling the work. However, the student will be frustrated if the valued topic has obstacles without a sufficient control.

Activating positive emotions (e.g., task enjoyment) preserves cognitive resources and promotes the interest and intrinsic motivation. Conversely, deactivating positive emotions (e.g., relaxation) reduces task attention and leads to superficial information processing. Activating negative emotions (e.g., test anxiety) reduces cognitive resources available for task purpose and intrinsic motivation. On the other hand, they can produce a strong motivation. Deactivating negative emotions reduces cognitive resources and both extrinsic and intrinsic motivation (Pekrun & Stephens, 2010). This complex relationship between positive and negative emotions may explain the low correlation between test anxiety and academic achievement (Hembree, 1988; Ma, 1999). It allows us to speculate about the circular causal relationship between emotion and cognition (Kim, Park, & Cozart, 2014; Pekrun, 2006; Pons et al., 2011).

Traditional approaches to the analysis of online learning have been conducted for the purpose of explaining emotional ambivalence (Baker, D’Mello, Rodrigo, & Graesser, 2010; S. D’Mello, Lehman, Pekrun, & Graesser, 2014). For example, frustration was found to be less associated with poor learning. Also, confusion can be beneficial to learning if appropriately regulated and resolved. When monitoring emotions in different computer-based environments, several studies found that poor learning-related negative emotions (i.e., frustration, boredom, and confusion) less frequently impact performance (Baker et al., 2010; Craig, Graesser, Sullins, & Gholson, 2004; S. K. D’Mello, Craig, Witherspoon, McDaniel, & Graesser, 2008). Students in cognitive disequilibrium (i.e., incongruity, dissonance, and internal conflict) can create opportunities for the deep learning of difficult content (S. D’Mello et al., 2014). However, if they fail to restore equilibrium, this state triggers frustration and eventual boredom (S. D’Mello & Graesser, 2012; S. D’Mello et al., 2014; Lehman, D’Mello, & Graesser, 2012).

Motivation is a teleological process whereby goal-directed activity is instigated and sustained (Schunk, Pintrich, & Meece, 2008). Academic motivation is determined by both social factors and learners’ cognition (Anderman & Dawson, 2011). Cognitive emotion theories (e.g., control-value theory) state that emotions are related to motivation, learning strategies, cognitive strategies, self-regulation, and academic achievement (Pekrun et al., 2002). In addition, emotions are closely and reciprocally linked to motivational antecedents and their emotional effects (Pekrun, 2006).

Linnenbrink (2007) and Pekrun, Elliot, and Maier (2006) discussed two theoretical models that integrate emotions and motivation. Linnenbrink’s (2007) model depicts a relation between motivation and engagement mediated by emotions. It supports the hypothesis that a reduction in unpleasant emotions for motivated students may help to explain higher levels of learning. The model proposed by Pekrun et al. (2006) suggested that mastery goals are positively related to hope. It also noted that pride was negatively related to boredom and anger. While performance-approach goals were positive predictors of pride, performance-avoidance goals were positive predictors of anxiety, hopelessness, and shame. The theoretical relations of this model were extended to include academic performance. Empirical evidence shows that performance-approach goals were positive predictors of performance, and performance-avoidance goals were negative predictors of performance (Pekrun, Elliot, & Maier, 2009).

Several online and face-to-face empirical studies examined emotions, motivation, and academic outcomes. For example, Kim et al. (2014) found that motivation and emotions explain approximately 37% of the variance of students’ final score. Similarly, Elias, Mustaf, Roslan, and Noah (2011); Stegers-Jager, Cohen-Schotanus, and Themmen (2012); Mega, Ronconi, and De Beni (2014); and Kim, Park, Cozart, and Lee (2015) found that motivational variables explain approximately 25% of students’ performances. González, Fernández, and Paoloni (2017) found that hope and anxiety are mediators between motivation and learning performance. Hope was found to be a positive predictor of the latter.

The influence of metacognitive self-regulation on performance has been studied in both online and face-to-face courses. Kim et al. (2015) detailed a significant difference in emotion, motivation, and metacognitive self-regulation between students with low and high performance. Kramarski and Gutman (2006) and Azevedo, Moos, Greene, Winters, and Cromley (2008) found that metacognitive self-regulation is closely related to students’ achievements. On the other hand, Lynch (2010); Al-Harthy, Was, and Isaacson (2010); and Cho and Heron (2015) found different levels of correlation (modest correlation, low correlation, and noncorrelation, respectively) between organization, metacognitive self-regulation, and each student’s final grade.

Data Mining to Forecast Learning Performance

Educational data mining is an interdisciplinary research area dealing with the development of method to explore data originating in an educational context (Peña-Ayala, 2014; Romero & Ventura, 2010). Educational data mining is useful in student modeling, particularly in the prediction of performance. The forecast can be carried out by a regression data mining task (numerical/continuous variable) or classification task (discrete/categorical variable; Romero & Ventura, 2010).

Data mining techniques have advantages over statistical techniques, including linear and logistic regression (LR). Neural networks (NNs) and decision trees (DTs), unlike statistical techniques, address data sets with a mixture of numerical and discrete attributes. The data to be analyzed with statistical techniques must satisfy several assumptions, including normality, nonmulticollinearity, linearity, and homoscedasticity. Some of these assumptions are often violated, and the fit of the model is decreased or adjusted. On the other hand, data mining techniques are less susceptible to the effect of violations. Moreover, most data mining techniques deal with missing values or are used to replace it (e.g., expectation maximization) and enhance regression models (Horton & Kleinman, 2007).

There has been an increasing interest in data mining in online learning. The subject has been considered in several articles, most of which deal with the construction of models based on behavioral indicators extracted from LMS logs to predict students’ performance. Cerezo, Sánchez-Santillán, Paule-Ruiz, and Núñez (2016) found that three out of six variables are strongly related to students’ final marks: (a) time spent on practical tasks, (b) words posted in forums, and (c) delays in the completion of tasks. Similarly, You (2016) modeled self-regulated behavior using total viewing time, number of sessions, and number of late submissions as behavioral indicators. You (2016) found that the regression model explained 58% of the variance in the overall score for the course. The models for predicting students’ success or failure in a course are based on data extracted from online forums (Romero, López, Luna, & Ventura, 2013), activities (i.e., quiz views, assignment views, etc.; Lara, Lizcano, Martínez, Pazos, and Riera, 2014), and the historical grades of activities (Burgos et al., 2018).

In the context of blended courses, Zacharis (2015) used 29 variables extracted from LMS and found that four were good predictors: (a) the number of messages read and posted, (b) content creation, (c) the number of quizzes viewed, and (d) the number of files viewed. The predictors explained 50% of the variance in the course grade. The binary classification model (base on those predictors) correctly classified 81.3% of students as having failed or not failed the course. On the other hand, other factors have not been considered in these models, including emotion, motivation, cognitive–metacognitive strategies, and behavior. Traver, Volchok, Bidjerano, and Shea (2014) established that demographic, status, social presence, teaching presence, cognitive presence, and learning presence variables have not been correlated with dropout from or completion of a blended course. In face-to-face courses, socioeconomic, social, and family variables, along with historical grades, are used to build classification models. The model with an overall accuracy of 97% was selected (Marquez-Vera, Morales, & Soto, 2013).

This research was intended as an attempt to understand the relation between students’ motivation, cognitive–metacognitive strategies, behavior, and learning performance in the context of blended courses in higher education. Thus, the following research questions are posed:

What is the relationship between emotions, motivation, cognitive–metacognitive strategies, behavior, and learning performance in blended learning?

Which variables (ranging from emotions and motivation to cognitive–metacognitive strategies and behavior) are most effective for the construction of the best model for classifying student learning performance?

Method

Participants and Research Context

The data sets used in this research were gathered from a blended course in applied computing in biological sciences. The course enrolled 137 undergraduate university students. Included in a chemical biology degree program, it combined online and face-to-face sessions. The online sessions contained learning materials and learning activities delivered throughout the semester via the Moodle platform. The face-to-face sessions were intended for student feedback and deep explanations of learning activities. Thus, communication via forums and chats rarely took place.

A weekly online session was scheduled. Thus, the delivery of learning activities was provided the same week through Moodle. The students received feedback for all uploaded learning activities. The interactions between the students and Moodle were recorded in a log file. It provided information about the name, the time of each online session, the date of delivery of an activity, and the interaction with learning materials. The learning materials were designed to comply with the shareable content object reference model (SCORM) standard (Advanced Distributed Learning, 2004). Therefore, the completion of a given material and the number of topics reviewed could be tracked by Moodle. The learning materials or SCORM objects were organized in a hierarchical structure of topics and subtopics. Students were required to browse all topics and subtopics to consider a given learning material as reviewed.

Students needed to log into Moodle each time to review current session or previous session learning materials. They were not permitted to download learning materials or browse offline. The course contained three offline examinations, learning activities, and an end-term project. These were considered when computing each student’s overall grade. Hence, the overall grade represented a measure of learning performance.

Instruments

Students’ emotions and behaviors were assessed using adaptation of the questionnaire (Skinner, Furrer, Marchand, & Kindermann, 2008). It measured student engagement and disaffection in school (SEDS; Wellborn, 1991). It was administered at the end of the second face-to-face session and after the midterm and end-term examinations. Twelve items referring to negative emotions (i.e., boredom, anxiety, frustration, etc.) measured emotional disaffection. Similarly, five items relating to positive emotions (or behavioral engagement) measured emotional engagement, and positive behaviors (i.e., attention, effort, etc.) were measured by five items. Finally, the behavioral disaffection was measured by five items.

Students’ answers were based on a 5-point Likert scale of agreement where 1 meant strongly agree and 5 meant strongly disagree. The emotional disaffection questions involved negative emotions. Therefore, low emotional disaffection values meant that they experienced negative emotions. For example, a 1-point answer to the question “when we work on something in class, I feel bored” meant that a student experienced boredom. The reliability of scores on these subscales ranged from .871 to .96. The results of the three instruments administered throughout the semester computed the average of the four subscales.

The motivated strategies for learning questionnaire (MSLQ) assessed motivation (i.e., self-efficacy, test anxiety, etc.) and cognitive–metacognitive strategies (i.e., organization, metacognitive self-regulation, etc.; Pintrich, 1991; Pintrich & de Groot, 1990). The questions in this survey were answered using a Likert scale ranging from 1 not at all true of me to 7 very true of me. The instrument was administered midsemester. The reliability of scores on self-efficacy, test anxiety, metacognitive self-regulation, and organization ranged from .739 to .896.

Finally, Moodle’s log file computed the number of delivered learning activities, the number of learning materials reviewed, time spent interacting with Moodle, the number of partially viewed learning materials, and the number of missed learning activities.

Data Gathering and Data Analysis

The data used in the current study were collected from the second semester of 2016 scholastic year. At the end of the semester, the logs of students’ interactions with Moodle, results of SEDS and MSLQ, and overall grade of each student were used to build a data mining view with 19 variables, including 3 from Moodle, 4 from SEDS, 11 from MSLQ, and the overall grade. The data analysis process is illustrated in Figure 1.

Figure 1.

Data analysis process.

First, a correlation analysis was conducted to select the variables that correlated highly with the overall grade of each student. Then, hierarchical regression analysis was performed. This analysis had two goals: (a) to explain the relationship between emotions, motivation, cognitive–metacognitive strategies, behavior, and learning performance in blended learning and (b) to identify significant predictors (see Figure 1). Thus, from a data mining point of view, the hierarchical regression analysis acted as a feature selection task to improve the classification performance (Fayyad, Piatetsky-Shapiro, & Smyth, 1996). Finally, several classification models were built based on the full (F) data mining view (19 variables) and reduced (R) data mining view (6 variables selected by hierarchical regression analysis) in conjunction with the naïve Bayes (NB), NN, LR, and DT techniques implemented in Waikato Environment for Knowledge Analysis (WEKA; Hall et al., 2009) as NB, multilayer perceptron, logistic, and J48, respectively. In building classification models, an overall grade of less than six was considered as failing the course.

Comparing multiple classifiers over multiple data sets is a common task to evaluate the general performance of data mining techniques. Two-way analysis of variance (ANOVA) is the suitable statistical test for this situation. Unfortunately, due to the nature of the learning algorithms and data sets, the normality and homoscedasticity assumptions are often violated (Demˇsar, 2006). This is true in this case. Thus, a nonparametric test is typically more convenient. The accuracy of the classification of these models was compared by the Scheirer–Ray–Hare test (Scheirer, Ray, & Hare, 1976). Table 1 shows these models. They were named based on the data mining view used and the technique by which it was built. For instance, RNB was based on the R data mining view and its having been built by means of NB.

Table 1.

Classification Models Based on Two Views Using Naïve Bayes, Neural Network, Logistic Regression, and Decision Tree Techniques.

Data mining techniques
		Naïve Bayes	Neural network	Logistic regression	Decision trees
Data mining view	Full (19 variables)	FNB	FNN	FLR	FDT
Data mining view	Reduced (6 variables)	RNB	RNN	RLR	RDT

Results

Correlation Analysis

The descriptive statistics are presented in Table 2. The average number of learning activities is approximately 16, which means that the students turned in 66.26% of the activities. Although the number of viewed learning materials is low, the students partially viewed approximately 74% of the total materials available on the LMS. This means that the students skipped at least one topic of the learning material packaged as SCORM (Advanced Distributed Learning, 2004).

Table 2.

Descriptive Statistics of LMS Variables and Subscales of SEDS and MSLQ.

Variable	M	SD	Min–max
Turned in learning activities	16.566	6.052	0–25
Fully viewed learning materials	2.772	2.906	0–11
Missing learning activities	8.434	6.052	0–25
Partially viewed learning materials	8.228	2.900	0–11
Spent time (s)	335,165.007	682,598.454	4,000–302,039
Behavioral engagement	1.840	0.917	1–4.8
Emotional engagement	1.840	0.930	1–4.8
Behavioral disaffection	2.171	0.842	1–5.0
Emotional disaffection	2.324	0.878	1–5.0
Extrinsic goal orientation	4.542	2.022	1–7.0
Intrinsic goal orientation	4.599	1.999	1–7.0
Task value	4.871	2.072	1–7.0
Self-efficacy	4.635	1.942	1–7.0
Test anxiety	3.594	1.768	1–6.8
Control of learning beliefs	4.755	1.995	1–7.0
Metacognitive self-regulation	3.580	1.471	1–5.75
Rehearsal	4.234	1.857	1–7.0
Elaboration	4.057	1.776	1–7.0
Organization	4.138	1.883	1–7.0
Critical thinking	4.159	1.807	1–7.0
Overall grade	7.625	2.357	0–10

Note. LMS = learning management systems; SEDS = student engagement and disaffection in school; MSLQ = motivated strategies for learning questionnaire.

Pearson correlations, presented in Table 3, show that the overall grade of each student is negatively correlated with behavioral and emotional disaffection, test anxiety, and missing learning activities. However, the overall grade is positively correlated with extrinsic goal orientation, intrinsic goal orientation, task value, self-efficacy, control of learning beliefs, metacognitive self-regulation, rehearsal, elaboration, organization, and critical thinking. Motivation (i.e., extrinsic goal orientation, intrinsic goal orientation, task value, self-efficacy, test anxiety, and control of learning beliefs) and cognitive–metacognitive strategies (i.e., metacognitive self-regulation, organization, and critical thinking) are positively correlated with partially viewed learning materials but negatively correlated with missing learning activities. The correlations between each motivational variable (i.e., extrinsic goal orientation, intrinsic goal orientation, task value, self-efficacy, test anxiety, and control of learning beliefs) and each cognitive–metacognitive strategy (i.e., metacognitive self-regulation, rehearsal, elaboration, organization, and critical thinking) are also positive.

Table 3.

Correlation Analysis of LMS Variables and Subscales of SEDS and MSLQ.

	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19
1	1
2	−.403**	1
3	.162	.015	1
4	.146	.022	.858**	1
5	.309**	−.077	.678**	.652**	1
6	.418**	−.099	.486**	.489**	.690**	1
7	−.136	−.072	.024	−.012	.001	−.019	1
8	−.419**	.185*	−.136	−.050	−.194*	−.351**	−.083	1
9	−.416**	.185*	−.144	−.031	−.174*	−.380**	−.018	.869**	1
10	−.447**	.179*	−.158	−.060	−.219*	−.378**	.000	.892**	.926**	1
11	−.448**	.195*	−.125	−.050	−.189*	−.388**	−.029	.910**	.946**	.953**	1
12	−.316**	.199*	−.132	.016	−.214*	−.308**	−.112	.707**	.662**	.685**	.678**	1
13	−.413**	.210*	−.114	−.011	−.195*	−.355**	−.020	.873**	.920**	.926**	.931**	.749**	1
14	−.448**	.178*	−.151	−.056	−.208*	−.372**	−.050	.886**	.893**	.917**	.914**	.695**	.888**	1
15	−.403**	.133	−.166	−.067	−.217*	−.373**	−.032	.866**	.798**	.840**	.825**	.745**	.827**	.921**	1
16	−.413**	.147	−.141	−.033	−.186*	−.360**	−.068	.876**	.821**	.859**	.864**	.695**	.818**	.941**	.923**	1
17	−.455**	.207*	−.126	−.044	−.191*	−.360**	−.052	.881**	.829**	.865**	.863**	.714**	.827**	.916**	.915**	.931**	1
18	−.377**	.193*	−.233**	−.015	−.173*	−.324**	−.053	.852**	.870**	.878**	.888**	.712**	.855**	.917**	.895**	.907**	.892**	1
19	−.523**	.138	.088	−.157	−.382**	.536**	−.035	.653**	.705**	.704**	.732**	−.589**	.713**	.714**	.636**	.644**	.603**	.652**	1

Note. LMS = learning management systems; SEDS = student engagement and disaffection in school; MSLQ = motivated strategies for learning questionnaire. 1 = Missing learning activities; 2 = Partially viewed learning materials; 3 = Behavioral engagement; 4 = Emotional engagement; 5 = Behavioral disaffection; 6 = Emotional disaffection; 7 = Spent time (s); 8 = Extrinsic goal orientation; 9 = Intrinsic goal orientation; 10 = Task value; 11 = Self-efficacy; 12 = Test anxiety; 13 = Control of learning beliefs; 14 = Metacognitive self-regulation; 15 = Rehearsal; 16 = Elaboration; 17 = Organization; 18 = Critical thinking; 19 = Overall grade.

*p < .05. **p < .01.

The condition index, variance inflation factor, and tolerance were checked among the predictors to determine the existence of multicollinearity issues. This analysis shows multicollinearity issues between returned and missing learning activities and fully viewed and partially viewed learning materials. Thus, returned learning activities and learning materials are not included in the hierarchical regression analysis.

Hierarchical Regression Analyses

To perform the correlation analysis, those variables with a significant correlation with the student’s overall grade were selected (see Table 3). First, the missing learning activities variable was entered in Step 1. Extrinsic goal orientation, intrinsic goal orientation, task value, self-efficacy, test anxiety, and control of learning beliefs were entered in Step 2. Emotional disaffection was added in Step 3. Metacognitive self-regulation, rehearsal, elaboration, organization, and critical thinking were added in Step 4. Finally, behavioral disaffection was entered in Step 5.

The analysis strategy was based on the following considerations. First, the learning activities represent a percentage of the overall grade of each student. Second, emotions are often considered a result of motivational factors despite their bidirectional influence (Kim et al., 2014; Marchand & Gutierrez, 2012; Pekrun, 2006; Pekrun et al., 2002). Third, metacognitive self-regulation influences organization, elaboration, and so forth (Al-Harthy et al., 2010; Cho & Heron, 2015).

As presented in Table 4, the missing of learning activities explains approximately 26% of the overall grade. Results from the second model indicate that motivational factors (e.g., self-efficacy and test anxiety) increase the amount of variance explained by the predictors in the equation to approximately 59%. The third model added the emotional disaffection. This increased the amount of explained variance to 63.3%. Finally, the amount of explained variance by the fourth model is 67.9%. This percentage was increased by adding metacognitive self-regulation and organization. The behavioral disaffection variable (β = −0.067, p = .329) was not significant to predict the students’ overall grade.

Table 4.

Summary of Multiple Regression Analysis for Variables Predicting Students’ Overall Grade (n = 137).

Model	Predictor	B	SE	Β	R² (adjusted)	R ²
1	Missing learning activities	−0.198	0.028	−0.523**	.268	.274
2	Missing learning activities	−0.092	0.023	−0.242**	.590	.599
	Self-efficacy	0.620	0.096	0.513**
	Test anxiety	−0.219	0.099	−0.165*
3	Missing learning activities	−0.064	0.023	−0.168**	.633	.643
	Self-efficacy	0.562	0.092	0.464**
	Test anxiety	−0.195	0.094	−0.146*
	Emotional disaffection	0.644	0.158	0.241**
4	Missing learning activities	−0.072	0.022	−0.19**	.679	.693
	Self-efficacy	0.493	0.149	0.408**
	Test anxiety	−0.272	0.094	−0.205**
	Emotional disaffection	0.633	0.148	0.237**
	Organization	0.707	0.160	0.565**
	Metacognitive self-regulation	0.868	0.244	0.543**

*p < .05. **p < .01.

Multicollinearity is presented with conditions index values above 20 and variance proportions values above 0.90 (Meyers, Gamst, & Guarino, 2006). It is indicated for tolerance values less than 0.1 and variance inflation factor values above 10 (Meyers et al., 2006). The condition index of the fourth model predictors ranged from 1 to 17.6; variance proportion values associated with each predictor ranged from 0.001 to 0.87. Similarly, tolerance values ranged from 0.145 to 0.77. Variance inflation factor values ranged from 1.29 to 6.91. The results suggest that multicollinearity is not presented in the final model.

In brief, the missing learning activities, test anxiety, and emotional disaffection (or negative emotions) negatively affected students’ overall grades. Noticeably, the overall grade of a student who experiments a high level of emotional disaffection (low Likert value) will decrease. Thus, β value of emotional disaffection is positive (see Table 4). The SEDS instrument was collected after the midterm and end-term examinations. This fact may have affected students’ answers. One-way ANOVA was performed among emotional disaffection measures collected at the end of the second face-to-face session (M = 2.30, SD = 0.82) and after the midterm (M = 2.36, SD = 0.92) and end-term examinations (M = 2.21, SD = 0.84). The result shows that the effect of the examinations on emotional disaffection measures was not significant (F = 0.874, p = .418). As one might have expected, an increase in the number of nondelivered learning activities decreases the students’ overall scores. On the other hand, self-efficacy, organization, and metacognitive self-regulation positively correlated with students’ performance.

Data Mining Models

For practical purposes, forecasting students’ risk of failing the course is more useful than predicting overall scores. Working under this assumption, the accuracy of several classification models is evaluated. The construction of models is based on F (19-variable) and R (6-variable) data mining views. As presented, six variables were selected by hierarchical regression analysis. To evaluate each model, a 10-fold cross-validation (repeated 10 times) was performed. The WEKA (Hall et al., 2009) data mining software was used to build the models. Those students with an overall grade of less than six (those who failed the course) were categorized as FA. Otherwise, students were categorized as AP. The classification accuracy data were not normal. Therefore, the Scheirer–Ray–Hare test was used in place of two-way ANOVA (Scheirer et al., 1976). Table 5 presents the descriptive statistics of the classification accuracy of the models.

Table 5.

Descriptive Statistics of the Classification Models (M, SD).

	NB	NN	LR	DT	Average
Full view (F)	93.34, 6.33	94.47, 6.19	89.30, 7.23	95.74, 5.38	93.21, 6.37
Reduced view (R)	91.87, 6.67	94.28, 6.48	95.59, 5.75	96.34, 4.84	94.52, 5.99
Average	92.61, 6.50	94.37, 6.34	92.45, 6.49	96.04, 5.11

Note. NB = naïve Bayes; NN = neural network; LR = logistic regression; DT = decision tree.

The Scheirer–Ray–Hare test results were evidence that the effects of data mining view (H = 7.050, p = 7.92E-03), data mining technique (H = 55.098, p = 6.54E-12), and Data Mining View × Data Mining Technique (H = 21.690, 7.57E-05) are significant (Scheirer et al., 1976). Thus, the models that relied on the R data mining view outperformed the classification accuracy of those models that relied on the F view. Finally, pairwise comparisons among data mining techniques were performed by means of Mann–Whitney–Wilcoxon, with Bonferroni correction being used to adjust the significance level (Rovai, Baker, & Ponton, 2013).

Table 6 presents pairwise comparisons by the data mining technique. From this data, DT is shown to have outperformed NN, LR, and NB. Thus, the model based on the R view and the DT technique (RDT) is the best (see Tables 5 and 7). The RTD model was tested in a real setting forecasting each student’s overall grade of the first semester of 2017. The prediction was performed after the answers to the two first SEDS and MSLQ measures had been collected. The RDT model correctly forecasted 94% of the overall grades. The RDT failed to identify four students who did not pass the course or three students who had been predicted as being at risk of failing. On the other hand, the RDT correctly identified 27 students who did not pass the course and 85 who approved it.

Table 6.

Pairwise Comparison Among Data Mining Techniques.

Pair	U	Sig
DT-NN	17,152.50	4.47E-02*
DT-LR	14,457.00	1.95E-06*
DT-NB	13,914.00	1.44E-07*
NN-LR	17,182.50	3.21E-01
NN-NB	16,786.50	2.23E-02*
LR-NB	19,851.00	1

Note. NB = naïve Bayes; NN = neural network; LR = logistic regression; DT = decision tree.

*p < .05.

Discussion

Relationship Between Emotions, Motivation, Cognitive–Metacognitive Strategies, Behavior, and Learning Performance

In this report, the authors provided a comprehensive identification of the relationship between emotions, motivation, behavior, and cognitive–metacognitive strategies, as well as their effects on students’ performance. Negative emotions (or emotional disaffection) were reported by the participants in this research. Although emotional ambivalence was reported (Baker et al., 2010; S. D’Mello et al., 2014), the presented results show that each students’ overall grade was negatively affected by both negative emotions and test anxiety. Also, positive emotions (or emotional engagement) were not significantly correlated with overall grade. Thus, they were not included in the hierarchical regression analysis. This suggests that negative emotions play a more important role than that of positive ones.

Recent efforts by several researchers in the field of online learning have demonstrated that motivation plays an important role in learning performance. The results show that self-efficacy (motivational factor) positively affect the overall grade. On the other hand, extrinsic goal orientation, intrinsic goal orientation, task value, and control of learning beliefs are not significant predictors of learning performance. In this research, emotions were considered a result of motivational factors despite their bidirectional influences (Kim et al., 2014; Marchand & Gutierrez, 2012; Pekrun, 2006; Pekrun et al., 2002). The correlation analysis shows that motivation (extrinsic goal orientation, intrinsic goal orientation, task value, self-efficacy, test anxiety, and control of learning beliefs) is significant and negatively correlated with emotional disaffection. Thus, motivation may reduce negative emotions. Similarly, motivation may increase the use of cognitive and metacognitive strategies.

Differences between students with low and high performances in metacognitive self-regulation have been found in previous studies. Contrary findings of other investigations have yielded modest to low correlation or noncorrelation between metacognitive self-regulation and students’ overall grades. It suggests complex relationships. The results of the authors show that metacognitive self-regulation and organization significantly correlate with students’ performance and play an important role in blended learning. These results support the findings of previous research studies (Kim et al., 2015).

In this study, Moodle logs were analyzed to extract indicators of students’ behaviors. The variable measure of whether learning activities had been missed (missing learning activities) was included in the model to explain 19% of the variation in the overall grade (Zacharis, 2015). Other studies revealed that several LMS indicators were highly correlated with learning performance (Cerezo et al., 2016; Lara et al., 2014; Romero et al., 2013; You, 2016; Zacharis, 2015). Yet, the variables of partially viewed learning materials and time spent were excluded from the model at the first step of the hierarchical regression analysis. Finally, students’ self-reported behaviors (behavioral engagement and behavioral disaffection) were not good predictors of learning performance. In blended learning, these results suggest that negative emotions, motivation, and cognitive–metacognitive factors are more important than LMS indicators of students’ behaviors.

Classification Models

Several classification models were compared based on two data mining views and four classification techniques. The Scheirer–Ray–Hare test shows a significant difference between R and F views. The average of R view is higher than F view. Thus, R view enhances the classification performance of DT and LR (see Table 5). This evidence shows that the hierarchical regression analysis is a valuable feature selection task to improve classification rate. A feature selection task aims to select attributes with a high correlation value with the class (in this case the learning performance) and a low correlation value among them. In the hierarchical regression analysis, these goals are accomplished by correlation analysis and eliminating multicollinearity issues.

The averages of DT data mining technique outperform the others (see Table 5). The interaction Data Mining View × Data Mining Technique (H = 21.690, 7.57E-05) was significant; thus, the best model is RDT. It was built by the DT technique and regression model predictors. The model correctly classified the pass/fail status of 96% of students. This indicates the adequacy of using data regarding missed learning activities, along with self-efficacy, test anxiety, emotional disaffection, organization, and metacognitive self-regulation to discriminatively predict students’ performance.

Limitations of the Present Study

The correlation analysis shows that relationships between motivation and cognitive–metacognitive strategies and motivation and missing learning activities were significant. However, the causation cannot be established by this analysis. Thus, structural equation modeling is appropriate. The hierarchical regression analysis identifies direct relations from the predictors to overall grade. To avoid multicollinearity, the predictors with low correlation are preferable. Variables that play a mediator role may be neglected.

Conclusion and Future Work

In this article, empirical evidence supports the hypothesis that emotions, motivation, cognitive–metacognitive strategies, and missed learning activities (as an indicator of student behavior) predict students’ performance. Self-reported behaviors (time spent on LMS and partially viewed learning materials) are unrelated to students’ overall grades. Thus, empirical evidence suggests that emotions, motivation, and cognitive–metacognitive strategies in blended learning are closely correlated with student performance.

These findings, as well as the need to identify students at risk of failing the course, encouraged the building of a data mining model to correctly classify 96% of the students. This model paves the way for enhanced LMS use in blended learning by monitoring emotions, motivation, and cognitive–metacognitive strategies to forecast the learning performance.

The present research shows the importance of emotion, motivation, and cognitive–metacognitive strategies in predicting students’ learning performance. However, the causal relationships among them have been undervalued. The outcome of investigating more complex relationships in the context of blended learning in higher education remains an open question. For example, the authors plan to examine the relationship between emotions, motivation, and personality. The impacts of a learning environment (face-to-face, blended, and online) on emotions, motivation, and cognitive–metacognitive strategies will be evaluated.

Footnotes

Acknowledgments

The authors thank the referees for comments that greatly improved the article.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was partially supported by grants numbers 20170922 and 20171422 from SIP programs at Instituto Politécnico Nacional.

Author Biographies

Aldo Ramirez-Arellano received the PhD degree from the National Polytechnic Institute, Mexico in 2017. His main fields of interest are educational technology and data mining.

Juan Bory-Reyes received the PhD degree in mathematics in 1988 and the Dr. Sc. Habilitation degree in 2008 from University of Oriente, Cuba. He is a research professor in the Master and Doctorate programs in Engineering of Systems within the Superior School of Mechanical and Electric Engineering of the National Polytechnic Institute.

Luis Manuel Hernández-Simón, PhD, is a professor in the Department of Systems Engineering, ESIME-Zacatenco, National Polytechnic Institute, Mexico City. His main fields of research are knowledge management in education, statistical models, and quality management systems.

References

Advanced Distributed Learning. (2004). SCORM (sharable content object reference model) 4th edition. Retrieved from http://adlnet.gov/adl-research/scorm/scorm-2004-4th-edition/.

Al-Harthy

I. S.

Was

C. A.

Isaacson

R. M.

(2010) Goals, efficacy and metacognitive self-regulation: A path analysis. International Journal of Education 2(1): 1.

Anderman

E. M.

Dawson

(2011) Learning with motivation. In: Mayer

R. E.

Alexander

P. A.

(eds) Handbook of research on learning and instruction, New York, NY: Routledge, pp. 219–241.

Asociación Nacional de Universidades e Instituciones de Educación Superior (2015) Anuario Estadístico 2014-2015, Mexico City, Mexico: Author.

Azevedo

Moos

D. C.

Greene

J. A.

Winters

F. I.

Cromley

J. G.

(2008) Why is externally-facilitated regulated learning more effective than self-regulated learning with hypermedia? Educational Technology Research and Development 56(1): 45–72. doi:10.1007/s11423-007-9067-0.

Baker

R. S. J. D.

D’Mello

S. K.

Rodrigo

M. M. T.

Graesser

A. C.

(2010) Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies 68(4): 223–241. doi:10.1016/j.ijhcs.2009.12.003.

Burgos

Campanario

M. L.

Peña

D. D. L.

Lara

J. A.

Lizcano

Martínez

M. A.

(2018) Data mining for modeling students’ performance: A tutoring action plan to prevent academic dropout. Computers & Electrical Engineering. 66, 541–556. doi:10.1016/j.compeleceng.2017.03.005.

Cerezo

Sánchez-Santillán

Paule-Ruiz

M. P.

Núñez

J. C.

(2016) Students’ LMS interaction patterns and their relationship with achievement: A case study in higher education. Computers & Education 96: 42–54. doi:10.1016/j.compedu.2016.02.006.

Cho

M.-H.

Heron

M. L.

(2015) Self-regulated learning: The role of motivation, emotion, and use of learning strategies in students’ learning experiences in a self-paced online mathematics course. Distance Education 36(1): 80–99. doi:10.1080/01587919.2015.1019963.

10.

Craig

Graesser

Sullins

Gholson

(2004) Affect and learning: An exploratory look into the role of affect in learning with AutoTutor. Journal of Educational Media 29(3): 241–250. doi:10.1080/1358165042000283101.

11.

Demˇsar

(2006) Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research 7. (2006), 1–30.

12.

D’Mello

Graesser

(2012) Dynamics of affective states during complex learning. Learning and Instruction 22(2): 145–157. doi:10.1016/j.learninstruc.2011.10.001.

13.

D’Mello

Lehman

Pekrun

Graesser

(2014) Confusion can be beneficial for learning. Learning and Instruction 29: 153–170. doi:10.1016/j.learninstruc.2012.05.003.

14.

D’Mello

S. K.

Craig

S. D.

Witherspoon

McDaniel

Graesser

(2008) Automatic detection of learner’s affect from conversational cues. User Modeling and User-Adapted Interaction 18(1): 45–80. doi:10.1007/s11257-007-9037-6.

15.

Elias

Mustaf

S. M. S.

Roslan

Noah

S. M.

(2011) Motivational predictors of academic performance in end year examination. Procedia - Social and Behavioral Sciences 29: 1179–1188. doi:10.1016/j.sbspro.2011.11.352.

16.

Fayyad

Piatetsky-Shapiro

Smyth

(1996) The KDD process for extracting useful knowledge from volumes of data. Communications of the ACM 39(11): 27–34. doi:10.1145/240455.240464.

17.

González

Fernández

M.-V. C.

Paoloni

P.-V.

(2017) Hope and anxiety in physics class: Exploring their motivational antecedents and influence on metacognition and performance. Journal of Research in Science Teaching 54(5): 558–585. doi:10.1002/tea.21377.

18.

Graham, C. R. (2005). Blended learning systems: Definition, current trends, and future directions. In C. J. Bonk & C. R. Graham (Eds.), The handbook of blended learning: Global perspectives, local designs (pp. 3–21). San Francisco, CA: Pfeiffer Publishing.

19.

Hall

Frank

Holmes

Pfahringer

Reutemann

Witten

I. H.

(2009) The WEKA data mining software: An update. ACM SIGKDD Explorations Newsletter 11(1): 10–18. doi:10.1145/1656274.1656278.

20.

Harding, A., Kaczynski, D., & Wood, L. (2005). Evaluation of blended learning: Analysis of qualitative data. Paper presented at the UniServe Science Blended Learning Symposium, September, Sydney, Australia.

21.

Hembree

(1988) Correlates, causes, effects, and treatment of test anxiety. Review of Educational Research 58(1): 47–77. doi:10.3102/00346543058001047.

22.

Hirumi

(2002) The design and sequencing of e-learning interactions: A grounded approach. International Journal on E-Learning 1: 19–27.

23.

Horton

N. J.

Kleinman

K. P.

(2007) Much ado about nothing: A comparison of missing data methods and software to fit incomplete data regression models. The American Statistician 61(1): 79–90. doi:10.1198/000313007X172556.

24.

Kim

Park

S. W.

Cozart

(2014) Affective and motivational factors of learning in online mathematics courses. British Journal of Educational Technology 45(1): 171–185. doi:10.1111/j.1467-8535.2012.01382.x.

25.

Kim

Park

S. W.

Cozart

Lee

(2015) From motivation to engagement: The role of effort regulation of virtual high school students in mathematics courses. Educational Technology & Society 18(4): 261–272.

26.

Kramarski

Gutman

(2006) How can self-regulated learning be supported in mathematical E-learning environments? Journal of Computer Assisted Learning 22(1): 24–33. doi:10.1111/j.1365-2729.2006.00157.x.

27.

Lara

J. A.

Lizcano

Martínez

M. A.

Pazos

Riera

(2014) A system for knowledge discovery in e-learning environments within the European Higher Education Area – Application to student data from Open University of Madrid, UDIMA. Computers & Education 72: 23–36.

28.

Lehman

D’Mello

Graesser

(2012) Confusion and complex learning during interactions with computer learning environments. The Internet and Higher Education 15(3): 184–194. doi:10.1016/j.iheduc.2012.01.002.

29.

Linnenbrink

E. A.

(2007) Chapter 7 – The role of affect in student learning: A multi-dimensional approach to considering the interaction of affect, motivation, and engagement. In: Schutz

P. A.

Pekrun

(eds) Emotion in education, Burlington, NJ: Academic Press, pp. 107–124.

30.

Linnenbrink-Garcia

Patall

E. A.

Pekrun

(2016) Adaptive motivation and emotion in education. Policy Insights from the Behavioral and Brain Sciences 3(2): 228–236. doi:10.1177/2372732216644450.

31.

Lynch

D. J.

(2010) Motivational beliefs and learning strategies as predictors of academic performance in college physics. College Student Journal 44: 920–927.

32.

(1999) A meta-analysis of the relationship between anxiety toward mathematics and achievement in mathematics. Journal for Research in Mathematics Education 30: 520–540.

33.

Marchand

G. C.

Gutierrez

A. P.

(2012) The role of emotion in the learning process: Comparisons between online and face-to-face learning settings. The Internet and Higher Education 15(3): 150–160. doi:10.1016/j.iheduc.2011.10.001.

34.

Marquez-Vera

Morales

C. R.

Soto

S. V.

(2013) Predicting school failure and dropout by using data mining techniques. Tecnologias del Aprendizaje, IEEE Revista Iberoamericana de 8(1): 7–14. doi:10.1109/rita.2013.2244695.

35.

Matthews

(1997) An introduction to the cognitive science of personality and emotion. In: Stelmach

G. E.

Vroon

E. A.

(eds) Advances in psychology Vol. 124, Amsterdam, the Netherlands: Elsevier.

36.

Mega

Ronconi

De Beni

(2014) What makes a good student? How emotions, self-regulated learning, and motivation contribute to academic achievement. Journal of Educational Psychology 106(1): 121–131. doi:10.1037/a0033546.

37.

Meyers

L. S.

Gamst

Guarino

A. J.

(2006) Applied multivariate research: Design and interpretation, Thousand Oaks, CA: SAGE Publications.

38.

Pekrun

(2006) The control-value theory of achievement emotions: Assumptions, corollaries, and implications for educational research and practice. Educational Psychology Review 18(4): 315–341. doi:10.1007/s10648-006-9029-9.

39.

Pekrun

Elliot

A. J.

Maier

M. A.

(2006) Achievement goals and discrete achievement emotions: A theoretical model and prospective test. Journal of Educational Psychology 98(3): 583–597. doi:10.1037/0022-0663.98.3.583.

40.

Pekrun

Elliot

A. J.

Maier

M. A.

(2009) Achievement goals and achievement emotions: Testing a model of their joint relations with academic performance. Journal of Educational Psychology 101(1): 115–135. doi:10.1037/a0013383.

41.

Pekrun

Frenzel

A. C.

Goetz

Perry

R. P.

(2007) The control-value theory of achievement emotions: An integrative approach to emotions in education. In: Schutz

P. A.

Pekrun

(eds) Educational psychology series. Emotion in education, Burlington, NJ: Academic Press, pp. 13–36.

42.

Pekrun

Goetz

Titz

Perry

R. P.

(2002) Academic emotions in students’ self-regulated learning and achievement: A program of qualitative and quantitative research. Educational Psychologist 37(2): 91–105. doi:10.1207/s15326985ep3702_4.

43.

Pekrun

Stephens

E. J.

(2010) Achievement emotions: A control-value approach. Social and Personality Psychology Compass 4(4): 238–255. doi:10.1111/j.1751-9004.2010.00259.x.

44.

Peña-Ayala

(2014) Educational data mining: A survey and a data mining-based analysis of recent works. Expert Systems with Applications 41(4, Part 1): 1432–1462. doi:10.1016/j.eswa.2013.08.042.

45.

Pintrich

P. R.

(1991) A manual for the use of the motivated strategies for learning questionnaire (MSLQ), Ann Arbor, MI: University of Michigan; Washington, DC: U.S. Dept. of Education, Office of Educational Research and Improvement, Educational Resources Information Center.

46.

Pintrich

P. R.

de Groot

E. V.

(1990) Motivational and self-regulated learning components of classroom academic performance. Journal of Educational Psychology 82(1): 33–40. doi:10.1037/0022-0663.82.1.33.

47.

Pons, F., Rosnay, M. D., & Cuisinier, F. (2011). Cognition and emotion. In V. G. Aukrust (Ed.), Learning and cognition in education (pp. 78–84). Amsterdam, the Netherlands: Elsevier.

48.

Romero

López

M.-I.

Luna

J.-M.

Ventura

(2013) Predicting students’ final performance from participation in on-line discussion forums. Computers & Education 68: 458–472. doi:10.1016/j.compedu.2013.06.009.

49.

Romero

Ventura

(2010) Educational data mining: A review of the state of the art. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 40(6): 601–618. doi:10.1109/TSMCC.2010.2053532.

50.

Rooney

J. E.

(2003) Blending learning opportunities to enhance educational programming and meetings. Association Management 55(5): 26–32.

51.

Rovai

A. P.

Baker

J. D.

Ponton

M. K.

(2013) Social science research design and statistics: A practitioner’s guide to research methods and IBM SPSS, Chesapeake, VA: Watertree Press.

52.

Scheirer

C. J.

Ray

W. S.

Hare

(1976) The analysis of ranked data derived from completely randomized factorial designs. Biometrics 32(2): 429–434. doi:10.2307/2529511.

53.

Schunk

D. H.

Pintrich

P. R.

Meece

J. L.

(2008) Motivation in education: Theory, research, and applications, Upper Saddle River, NJ: Pearson/Merrill Prentice Hall.

54.

Secretaría de Educación Pública. (2015). Sistema Educativo de los Estados Unidos Mexicanos. Principales Cifras Ciclo escolar 2014–2015 [Educational System of the United States of Mexico. Main Figures of the 2014-2015 school year]. Mexico City, Mexico: Secretaría de Educación Pública.

55.

Skinner

Furrer

Marchand

Kindermann

(2008) Engagement and disaffection in the classroom: Part of a larger motivational dynamic? Journal of Educational Psychology 100(4): 765–781. doi:10.1037/a0012840.

56.

Stegers-Jager

K. M.

Cohen-Schotanus

Themmen

A. P. N.

(2012) Motivation, learning strategies, participation and medical school performance. Medical Education 46(7): 678–688. doi:10.1111/j.1365-2923.2012.04284.x.

57.

Traver

A. E.

Volchok

Bidjerano

Shea

(2014) Correlating community college students’ perceptions of community of inquiry presences with their completion of blended courses. The Internet and Higher Education 20(2014): 1–9. doi:10.1016/j.iheduc.2013.09.001.

58.

Wellborn

J. G.

(1991) Engaged and disaffected action: The conceptualization and measurement of motivation in the academic domain, Rochester, NY: University of Rochester.

59.

Woltering

Herrler

Spitzer

Spreckelsen

(2009) Blended learning positively affects students’ satisfaction and the role of the tutor in the problem-based learning process: Results of a mixed-method evaluation. Advances in Health Sciences Education 14(5): 725. doi:10.1007/s10459-009-9154-6.

60.

You

J. W.

(2015) Examining the effect of academic procrastination on achievement using LMS data in e-learning. Educational Technology & Society 18(3): 64–74.

61.

You

J. W.

(2016) Identifying significant indicators using LMS data to predict course achievement in online learning. The Internet and Higher Education 29: 23–30. doi:10.1016/j.iheduc.2015.11.003.

62.

Young

(2002) Hybrid teaching seeks to end the divide between traditional and online instruction. The Chronicle of Higher Education: Information Technology 48(28): A33–A34.

63.

Zacharis

N. Z.

(2015) A multivariate approach to predicting student outcomes in web-enabled blended learning courses. The Internet and Higher Education 27: 44–53. doi:10.1016/j.iheduc.2015.05.002.