College Students’ Cognitive Learning Outcomes in Technology-Enabled Active Learning Environments: A Meta-Analysis of the Empirical Literature

Abstract

Technology-enabled active learning environments (TE-ALEs) have attracted considerable research interest, particularly in higher education. However, research shows inconsistent results describing the influence of TE-ALEs toward students’ cognitive learning outcomes. This study was designed to identify high-quality empirical research examining college students’ cognitive learning outcomes and to utilize meta-analysis to determine the overall effectiveness of TE-ALEs. A systematic literature search identified 31 high-quality peer-reviewed journal articles that met the inclusion criteria. Meta-analysis showed that the calculated effect size of TE-ALEs more positively influenced students’ cognitive learning than traditional lecture-based environments. Moderator variable analysis suggested that social context, study design, and sample size were significant factors that influence the effectiveness of TE-ALE. TE-ALEs were found more effective when instructors employed individualized learning contexts as well as when bias was reduced in randomized controlled trials. TE-ALEs were also found to be more effective in small courses rather than in large courses.

Keywords

technology-enabled active learning active learning environment cognitive learning outcomes meta-analyses effect size

Introduction

Technology-enabled active learning environments (TE-ALEs) were developed to enable blended learning approaches that support active learning processes and activities which promote instructional interaction and engagement during in-class sessions. TE-ALEs offer student-centered experiences designed to equalize student and teacher control over technology integration (Beichner, 2014; MacLeod, Yang, Zhu, & Li, 2018). For example, as a part of TE-ALEs, the technology-enabled active learning (TEAL) classroom has been recognized by the EDUCAUSE Center for Analysis and Research as a top strategic technological investment for higher education institutions (Brooks, 2017). Active learning environments supported by technology began gaining widespread recognition in higher education after pioneer research at the Massachusetts Institute of Technology documented new processes in introductory freshman electromagnetism courses (Dori & Belcher, 2005; Dori et al., 2003) and other programs such as the Student-Centered Activities for Large Enrollment Undergraduate Programs Project (Beichner et al., 2007). These initiatives represent a transformational shift in how technology is used to encourage more social, flexible, and student-centered active learning experiences in higher education with the goal of achieving better communication, collaboration, creativity, and higher order thinking skills (Kovalchick & Kawson, 2004; Partnership for 21st Century Learning, 2007; Ruiz-Primo, Briggs, Iverson, Talbot, & Shepard, 2011). Nowadays, TE-ALEs refer not only the physical classroom spaces but also the broader classification of virtual and cloud learning spaces which heavily rely upon technology to deliver active learning pedagogy.

A close review of the broader topic of active learning also shows that research is not always in support of active learning pedagogy stimulating stronger cognitive learning outcomes (e.g., Andrews, Leonard, Colgrove, & Kalinowski, 2011). In addition, active learning pedagogies are most prominently researched within science-related educational disciplines (Ruiz-Primo et al., 2011) and often lack well-documented integration beyond these fields of instruction. While several meta-analyses, especially of STEM (science, technology, engineering, and mathematics) courses (e.g., Freeman et al., 2014; Springer, Stanne, & Donovan, 1999), explored the effectiveness of active learning pedagogy on student cognitive learning in undergraduate education, little research exists on the implementation of TE-ALEs across a wide range of educational disciplines.

Furthermore, research examining the effectiveness of TE-ALEs to improve college students’ cognitive learning outcomes has demonstrated mixed results. For instance, numerous studies have reported that TE-ALEs significantly improved students’ cognitive learning compared with counterparts in traditional learning environments (e.g., Blázquez et al., 2019; Parishan, Jafari, & Nosrat, 2011; Welch, 2013). Meanwhile, research has also shown that TE-ALEs do not lead to significant differences in college students’ cognitive learning outcomes (e.g., Baepler, Walker, & Driessen, 2014; Mendini & Peter, 2019; Shieh, Chang, & Liu, 2011). Mixed and unexpected results even exist within studies. For example, a study of a university-level physics classroom showed that students’ cognitive learning improved significantly during the first semester, but not in the second semester (Shieh, Chang, & Tang, 2010). Such inconsistencies provide a rationale for deeper investigative methods like meta-analyses, which can synthesize the findings of many different observations of similar contexts.

Thus, despite being the focus of much research, the extent to which TE-ALEs influence students’ cognitive learning remains unclear. This study was designed to provide a new quantitative perspective via meta-analyses of available high-quality empirical research. In particular, this study (a) explores the effectiveness of TE-ALEs across disciplines in higher education compared with traditional learning environments and (b) investigates what moderators may influence the effectiveness of TE-ALEs.

Background

Active Learning and TE-ALE

Active learning is an umbrella term generally defined as any instructional method that actively engages students in the learning process, encouraging them to participate in meaningful learning activities and think about what they are doing (Bonwell & Eison, 1991). With student activity and engagement as the core elements, active learning aims to activate or develop students’ higher order thinking in the learning process and often involves group work (Hung, 2015). Active learning is often contrasted with traditional lecture-based instructional methods, where students passively receive information from the instructor (Prince, 2004).

Due to the infusion and diffusion of a variety of technologies in the educational context, a vast range of innovative active learning pedagogies and instructional strategies can be implemented in TE-ALEs to foster participation and engagement with the material being presented, rather than passive reception through listening (Green & Repetti, 2015). These may include different social contexts such as individualized active learning or collaborative active learning (e.g., peer learning, cooperative learning, and team-based learning; Hung, 2015; Prince, 2004). TE-ALEs can also take a variety of forms, such as those in the models of flipped classrooms and smart classrooms, or through workshops, group problem-solving, and just-in-time teaching (e.g. Aşıksoy & Özdamli, 2016; Seery, 2015; Shi, Peng, Wang, & Yang, 2018; Mazur, 2004). A growing body of literature has focused on teaching and learning in TE-ALEs. For instance, numerous studies have noted that TE-ALEs offer potential benefits for the teaching and learning process across various disciplines and educational levels (e.g., Lewis, Chen, & Relan, 2018; Mueller, Knobloch, & Orvis, 2015; Shieh, 2012). However, as Alcalde and Nagel (2019) argued, although studies consistently show that students exhibit greater satisfaction with active learning methods, the results on student achievement are mixed: sometimes positive and sometimes noneffect.

Meta-Analysis and Related Works

Meta-analysis is a quantitative statistical method for synthesizing and integrating descriptive statistics reported in multiple relevant published and unpublished primary research testing the same conceptual question and hypothesis (Glass, 1976; Hedges & Olkin, 1985).

Several meta-analytic studies examined to the broader topic of active learning (e.g., Baranowski & Weir, 2015; Freeman et al., 2014; Kalaian & Kasim, 2014; Naing et al., 2015; Prince, 2004; Springer et al., 1999). For instance, Freeman et al. (2014) meta-analyzed 225 studies across different scientific disciplines, study designs, and class sizes to determine whether active learning improves examination scores when compared with traditional lecture-based instruction. Freeman et al. found that active learning interventions increased students’ examination performance by 0.47 standard deviations across the STEM disciplines. In addition, the active learning interventions tended to increase students’ scores on concept inventories more than on course examinations and appear effective across all class sizes. Thus, active learning seems to positively impact on students’ cognitive learning outcomes, suggesting it to be a preferred, empirically validated teaching practice in regular face-to-face environments. In a similar meta-analytic study published over a decade earlier, Springer et al. (1999) investigated the effects of small-group learning on undergraduates enrolled in STEM courses. Springer et al.’s results showed that various forms of small-group learning were effective in promoting greater academic achievement. These two meta-analyses painted a positive image for active learning; however, they only examined courses in STEM disciplines and did not focus on the more specific context of TE-ALEs.

Three other meta-analyses specifically examined the cognitive learning outcomes of active learning environments in subject-specific samples of college-level participants within the disciplines of medical education (Naing et al., 2015), statistics (Kalaian & Kasim, 2014), and political science (Baranowski & Weir, 2015). While these publications provide valuable new perspectives for these disciplinary demographics, they do not contribute knowledge to clarify the research inconsistencies surrounding TE-ALEs. Therefore, the influence of TE-ALEs toward college students’ cognitive learning outcomes remains unclear and is an important topic for conducting additional meta-analytic research.

Research Questions

To explore the status of college students’ learning outcomes in TE-ALEs more deeply, the following research questions were proposed to guide this study design and analysis:

How does the effectiveness of the TE-ALEs compare to traditional learning environments?

Which moderator variables influence the effectiveness of TE-ALEs?

Methodology

Conceptual Framework

Cognitive learning is a clear indicator for evaluating the quality of education (Michaelowa, 2010), and it is commonly measured either through examinations or through continuous assessments. Inspired by the framework proposed by Means, Toyama, Murphy, and Baki (2013) and a review of the moderator variables included in other recent meta-analyses (Hew & Lo, 2018; Shi, Ma, MacLeod, & Yang, 2019), the conceptual framework for this study was guided both by data extraction of the studies included in the meta-analysis and a follow-up moderator analysis, as shown in Figure 1. The framework comprises three categories: (a) instructional experiences of TE-ALEs, including social context, information and communication technology (ICT) features, and treatment duration; (b) the conditions of the studies, including the year of publication and subject matter; and (c) features of the research methodology, including study design, sample size, and instructor equivalence.

Figure 1.

Conceptual framework. ICT = information and communication technology.

All variables related to instructional experiences, study conditions, and research methodology were extracted, and a variable was analyzed as a potential moderator of the effect size only if an adequate number of studies with the necessary data for the variable was available. It should be noted that, among the variables, the social context is perhaps the most important, as it has been shown to have a significant influence on students’ learning outcomes. The different social context may promote different types of learning experiences as well as the nature of learners’ activities and engagement (Ryan & Reid, 2016). In this study, social context refers to the degree of interpersonal communication between students. Two major social contexts commonly described in the literature are individualized learning (a single independent person actively learning) and collaborative learning (some form of the interdependent group actively learning).

In this study, individualized learning refers to any instructional method that requires students to be active participants in the learning process while in a learning environment, but in ways that do not require peer coordination or communication with others to accomplish the given tasks (Crimmins & Midkiff, 2017; Prince, 2004). Comparatively, collaborative learning refers to the implementation of interpersonal interaction (Bouroumi & Fajr, 2014) and emphasizes student communication and pro-social behaviors as a method of engagement in a learning environment (Gilies, Ashman, & Terwel, 2008; Prince, 2004). In contrast to both of these active social contexts, the traditional learning environment renders students as passive listeners and notetakers (Wilson, 2012), only receiving information from the instructors.

Search and Selection Process

As shown in Figure 2, the survey of related studies included an initial search and screen, selection criteria, and a final selection and data extraction. Two stages of screening were conducted before a decision was made to include studies for final review. First, the title and abstract of studies resulting from the initial electronic database searches were screened to establish a preliminary set of articles for potential final review. Thereafter, the full text of each of the articles in the preliminary set was screened to confirm the relevance of the assessed studies. This two-stage approach has proven to be an efficient way to avoid losing any potentially relevant and high-quality studies (Means et al., 2013).

Figure 2.

Flow diagram showing the journal article selection process.

Initial search and screen

A comprehensive search of publicly available literature was carried out to identify relevant studies for meta-analysis, to expand the depth of coverage, and to provide a more explicit quantitative synthesis of the available research. The first step of this process entailed searching prominent research journals for major contributions, such as Brooks (2011) in The British Journal of Educational Technology; Baepler et al. (2014) and Shieh (2012) in Computers & Education; and Lin (2019) in Computers in Human Behavior. The second step was to examine major contributions in order to trace the research chronologically, backward from the most prominent papers. This approach has also been described as a “snowball search” (Cruz, da Silva, & Capretz, 2015, p. 94). After identifying the major contributions and conducting a “snowball search,” key concepts and search terms were developed. The specific search phrases utilized in this study were as follows: “active learning classroom” OR “ALC” OR “technology enable active learning classroom” OR “technology enhanced active learning environment” OR “TEAL” OR “smart classroom.”

Databases, including EBSCO, the Education Resources Information Center, Elsevier Science Direct online, ProQuest, Springer, Taylor & Francis online, and Web of Science, were selected as data sources. After the initial electronic search, the reference lists of the eligible articles were also searched manually. Two researchers searched the databases together and independently selected eligible articles. To reach consensus, a third researcher was brought in to discuss and resolve any discrepancies between the extracted data. As a result, the search yielded 3,154 relevant articles and, after removal of 553 duplicates, 2,601 residual references were reviewed based on their title and abstract. The review of titles and abstracts honed the literature search to 147 potentially relevant articles that required deeper investigation in comparison to the selection criteria.

Selection criteria

Studies were examined to ascertain whether they met the following selection criteria: (a) the use of a rigorous research design (e.g., randomized controlled trials, controlled quasi-experimental design); (b) the report of quantitative data on student cognitive learning outcomes via objective-based assessments, such as tests or final examinations; (c) at least one comparison between a TE-ALE and a traditional learning environment; and (d) written in an English peer-reviewed journal. Studies that did not meet these four criteria were excluded.

Final selection and data extraction

To extract relevant data, information was collected from each of the studies, based mainly on study design method (such as experimental or quasi-experimental design), the treatment and duration, measurements for cognitive learning outcomes (such as examinations), as well as sample size, subject matter(s), author(s) of the study, and year of publication. Data extraction was independently performed by two researchers. Discrepancies between the extracted data were resolved via consensus between the two researchers. One hundred sixteen journal articles were rejected from the initial pool of 147 potentially relevant publications. The rationale was that there was no comparison between a TE-ALE and traditional learning environment (n = 42), or authors did not use an experimental or quasi-experiment research design (n = 16), or authors did not report empirical research with sufficiently quantitative learning outcome data (n = 58). As a result, a total of 31 eligible journal articles were included in the final meta-analysis.

Data Analysis

To test the data for heterogeneity and thus conduct the meta-analysis, Review Manager 5.3 software (Cochrane Collaboration, 2014) was used. Based on the sample size and the 95% conﬁdence intervals, the standardized mean difference (SMD) from the means and standard deviations of student cognitive learning outcome data (e.g., examination scores and posttest scores) were calculated for the effect size for each study. Moreover, variance analysis was employed for pooled studies (Shi et al., 2019). A two-sided p value below .05 was regarded as signiﬁcant for all analyses.

The ﬁxed-effect model or the random-effects model would be used depending on the heterogeneity of the data. The presence of heterogeneity (i.e., the degree of inconsistency in the studies’ results) was detected by the I² test (Higgins, Thompson, Deeks, & Altman, 2003). If there was heterogeneity, then a sensitivity analysis was used to assess whether this heterogeneity signiﬁcantly altered the results of the meta-analysis. Publication bias occurs when researchers only publish favorable results (Peplow, 2017), and the publication bias was assessed by observing the shapes of funnel plots and by calculating the fail-safe number (Nfs), which can be used together to determine whether there is a publication bias. In addition, to assess which instructional experiences, study conditions, and research methodology influence the results of the meta-analysis, moderator analyses were conducted to examine preset variables according to the proposed conceptual framework (Shi et al., 2019).

Characteristics of Included Studies

A review of the 31 journal articles resulted in 33 independent effect sizes (two articles each reported two independent studies). Table 1 provides a descriptive summary of the main elements of the 33 independent effect sizes included in the meta-analysis. These studies were published from 2006 through 2019. The sample sizes ranged from 21 to 865 participants. Nearly half of the studies were modest in scope with a sample size below 100. It is worth noting, however, that large-scale applications of TE-ALEs have emerged; 10 studies in the meta-analysis corpus included more than 300 learners. The pooled sample size in the review was 7,045 (experimental group = 3,461 and control group = 3,584). The subjects of studies included biology, chemistry, English, marketing, mathematics, mechanics, nursing, pharmacy, physics, politics, and psychology.

Table 1.

Characteristics of the 33 Independent Studies Included for Analysis.

No.	Study (year)	Sample size (E/C)	Subject	Method	Treatment duration	Measurements
1	Alcalde and Nagel (2019)	78 (39/39)	Mathematics	RCT	1 semester	Pretest, quizzes, M and F examination
2	Aşıksoy and Özdamlı (2016)	66 (36/30)	Physics	RCT	8 weeks	Pretest and posttest
3	Baepler et al. (2014)	620 (316/304)	Chemistry	QED	2 semesters	Posttest only
4	Baranowski (2006)	66 (38/28)	Politics	QED	1 semester	Pretest and posttest
5	Baytiyeh and Naja (2017)	45 (21/24)	Engineering	RCT	1 semester	Quizzes, M and F examination
6	Blázquez et al. (2019)	110 (60/50)	Social psychology	RCT	6 weeks	Theoretical examination
7	Brooks (2011)	86 (43/43)	Biological science	QED	1 semester	Course grades
8	Chiu and Cheng (2017)	306 (29/277)	General education	QED	2 years	Grade point average
9	Cotner, Loper, Walker, and Brooks (2013)	263 (102/161)	Biology (nonmajors)	QED	1 semester	Pretest and posttest
10	Cotta et al. (2016)	316 (151/165)	Pharmacy	QED	10 weeks	Posttest only
11	Crimmins and Midkiff (2017)	766 (395/371)	Chemistry	QED	2 semesters	Pretest and posttest
12	Dori, Hult, Breslow, and Belcher (2007)	172 (120/52)	Electromagnetism	RCT	1 semester	Pretest and posttest
13	Entezari and Javdan (2016)	66 (32/34)	Anatomy and physiology	QED	12 weeks	Three examinations
14	Ghorbani and Ghazvini (2016)	21 (11/10)	Physiology	RCT	3 weeks	Pretest and posttest
15	Green and Repetti (2015)	228 (128/100)	Financial accounting	QED	3 semesters	Final grades
16	He, Holton, Farkas, and Warschauer (2016)	677 (334/343)	Chemistry	RCT	1 semester	M and F examination
17	Heuett (2017)	82 (52/30)	Math	RCT	1 semester	Three examinations
18	Hung (2015)	51 (26/25)	English	QED	8 weeks	Posttest only
19	Krain and Lantis (2006)	50 (24/26)	Not subject based	QED	1 semester	Pretest and posttest
20	Krain and Lantis (2006)	49 (27/22)	Not subject based	QED	1 semester	Pretest and posttest
21	Lewis, et al. (2018)	200 (98/102)	Surgery clerkship	ESD	12 weeks	Subject examination
22	Lin (2019)	34 (19/15)	Software engineering	QED	10 weeks	Pretest and posttest
23	McCarroll, Pohle-Krauza, and Martin (2009)	40 (21/19)	Kinesiology	QED	1 semester	Final grades
24	Mendini and Peter (2019)	54 (30/24)	Marketing	QED	1 semester	Final grades
25	Poirier and Feldman (2007)	865 (447/418)	Psychology	RCT	unknown	Examination grades
26	Rau, Kennedy, Oxtoby, Bollom, and Moore (2017)	333 (66/267)	Chemistry	QED	16 weeks	Course total grades
27	Sevindik (2010)	66 (33/33)	Internal diseases	RCT	4 weeks	Pretest and posttest
28	Shieh et al. (2011)	410 (281/129)	Physics	QED	2 semesters	Pretest and posttest
29	Shieh et al. (2010)	113 (45/68)	Mechanics	QED	2 semesters	Pretest and posttest
30	Shieh et al. (2010)	386 (159/227)	Electromagnetism	QED	2 semesters	Pretest and posttest
31	Tune, Sturek, and Basile (2013)	27 (13/14)	C, R, and R, physiology	RCT	1 year	Pretest and final course grade
32	Welch (2013)	105 (53/52)	Nursing	QED	1 year	Pretest and posttest
33	Wilson (2012)	394 (255/139)	Business	QED	6 semesters	Final examination

Note. E/C = experimental TE-ALE group/control group or traditional lecture-based model; RCT = randomized controlled trials; QED = quasi-experiment design; ESD = explanatory sequential design; C, R, and R physiology = cardiovascular, respiratory, and renal physiology; M and F examination = midterm examination and final examination.

Results

The present meta-analysis included 33 eligible independent effect sizes, involving 3,461 subjects exposed to the TE-ALEs and 3,584 subjects exposed to the traditional learning environments. Review Manager 5.3 software was used to test the data for heterogeneity and to conduct the meta-analysis. Substantial heterogeneity was found (I²= 91%, p < .00001). Thus, a random-effects model was used to pool the data. Figure 3 shows the 33 effect sizes derived from the 31 journal articles. Two references appear twice in Figure 3 because multiple effect sizes were extracted from the same article. Among the 33 individual contrasts between TE-ALEs and traditional learning environments, 18 were significantly positive effects favoring the TE-ALEs, while 5 were negative effects favored the traditional learning environments. Overall, the pooled effect size showed a statistically significant difference in learning outcomes in favor of TE-ALEs as opposed to traditional learning environments (SMD = 0.58, 95% conﬁdence interval [0.40, 0.75], p < .00001), as shown in Figure 3. A sensitivity analysis was employed to verify the reliability of the obtained results using the sequential omission of individual studies one-by-one. A pooled effect size, which favored the TE-ALEs, did not change the effects observed in the initial analysis. Thus, these results indicate that TE-ALEs effectively improved college students’ cognitive learning outcomes.

Figure 3.

Forest plot of effect sizes (standardized mean difference) using the random effect model. SD = standard deviation; CI = confidence interval; IV = inverse variance.

A funnel plot of the 33 effect sizes with examination scores is shown in Figure 4. Visual inspection of Figure 4 shown that the shape of this funnel plot is approximate symmetry, as the 33 effect sizes were approximately symmetrical distributed around the vertical axis of SMD (0.58) with an inverted funnel shape, which indicated that there is no signiﬁcant publication bias. Moreover, the fail-safe number (Nfs), which estimates the number of unretrieved studies averaging null results needed to bring the overall combined effect size at a nonsignificant level (Rosenthal, 1979), was calculated at the significant level of .05 by using the formula: ${Nfs}_{0.05} = (\frac{\sum Z}{1.645})^{2} - k$ , where the Z is the Z value of each independent effect sizes and k is the numbers of included studies. The large the Nfs is, especially when Nfs is significantly exceeded the critical value 5 k + 10 (Rosenthal, 1991), then it can be concluded that the mean effect size can be considered robust, which suggest that there is no indication of publication bias (Hoeve et al., 2012). In this study, the k = 33, $\sum Z = 119.36$ , and ${Nfs}_{0.05} = (\frac{\sum Z}{1.645})^{2} - k$ = (119.36/1.645)²− 33 = 5,231.85 > 175 (5*33 + 10), which also suggested that there is no indication of publication bias.

Figure 4.

Funnel plot showing no significant publication bias. SE = standard error; SMD = standardized mean difference.

Moderator Analysis

Due to the high level of heterogeneity among studies in terms of effect sizes, moderator analyses were conducted using the random-effects mode to explore possible causes for the significant heterogeneity and investigate the influence of possible moderator variables. Data from each study in the meta-analysis were extracted and coded according to three types of moderator variables: instructional experiences, study conditions, and research methodology. Previous meta-analyses have been criticized for not providing sufficient information about features considered to be potential moderator variables (Bernard et al., 2004), and the moderator analysis for a certain variable had to exclude such studies. Moreover, it is generally accepted that the identified variable should have at least two contrasting subsets of studies and that each subset contains six or more study effects (Means et al., 2013). Instructional experiences, study conditions, and research methodology variables, of which study subsets met the size criterion, were included in the search for moderator variables. The results of moderator analysis on variables are shown in Table 2.

Table 2.

Tests of the Moderator Variables (Instructional Experiences, Study Conditions, and Research Methodology).

Category	Variable	N	SMD	95% CI	Z	I² (%)	p
Instructional experiences	Social context^a					88.6	.003**
	Collaborative	10	0.26	[−0.02, 0.54]	1.84
	Individualized	19	0.84	[0.58, 1.09]	6.42***
	Treatment duration^a					0	.35
	≤One semester	22	0.68	[0.41, 0.95]	5.01***
	>One semester	10	0.50	[0.21, 0.78]	3.38***
	ICT features					0	.90
	Basic ICT	23	0.58	[0.35, 0.80]	5.06***
	Advanced programs/ software	10	0.60	[0.29, 0.91]	3.84***
Study conditions	Year of publication					58.8	.12
	2006–2012	12	0.42	[0.17, 0.67]	3.28***
	2013–present	21	0.70	[0.45, 0.94]	5.54***
	Subject matter^a					28.2	.25
	Medical sciences	8	0.75	[0.37, 1.13]	3.84***
	STEM	15	0.64	[0.34, 0.93]	4.18***
	Social sciences	7	0.38	[0.11, 0.66]	2.71**
Research methodology	Study design^b					74.1	.05*
	Randomized controlled trial	11	0.91	[0.52, 1.30]	4.56***
	Quasi-experiment design	21	0.47	[0.26, 0.68]	4.35***
	Sample size					83.5	.002**
	<60	9	0.81	[0.32, 1.29]	3.24***
	60–200	12	0.96	[0.51, 1.41]	4.21***
	>200	12	0.23	[0.04, 0.41]	2.42*
	Instructor equivalence^a					67.5	.08
	Same instructor	13	0.53	[0.28, 0.78]	4.14***
	Different instructor	9	0.25	[0.07, 0.43]	2.71**

Note. N = number of studies; SMD = standardized mean difference; 95% CI = 95% confidence interval; ICT = information and communication technology; STEM = science, technology, engineering, and mathematics.

^aThe moderator analysis excluded several studies because they did not report information about this feature.

^bThe moderator analysis excluded one study with an explanatory sequential design.

p < .05; **p < .01; ***p < .001.

Table 2 shows the variation in effectiveness associated with eight variables from three categories. In the category of instructional experiences, social context, treatment duration, and ICT features were tested as moderator variables. The influence of the social context of the study was examined by dividing studies into two subsets: individualized learning and collaborative learning. The social context was found to moderate the size of the TE-ALEs effect significantly (I²= 88.6%, p < .05). The effect size for TE-ALEs with individualized learning contrasted against traditional environments was larger than that for TE-ALEs with collaborative learning contrasted against traditional environments.

The treatment duration was coded at a relatively coarse level (not more than one semester vs. more than one semester). The results suggest that treatment duration is not a significant moderator variable (I²= 0%, p > .05). The impact of the ICT features was examined by splitting studies into two subsets according to whether TE-ALEs studied used basic ICT or advanced programs and software. Both of the two kinds of ICT use in TE-ALEs significantly improved college students’ cognitive learning outcomes when compared with traditional learning environments. However, no significant difference was found between the two subsets, suggesting that the ICT features did not moderate the effectiveness of TE-ALEs significantly.

In the study conditions category, both the year of publication and the subject matter were tested as moderator variables. The year of publication was used to estimate the potential sophistication of TE-ALE pedagogy by splitting the study sample into two subsets and contrasting studies published from 2006 through 2012 against those published from 2013 through 2019. As mentioned, studies covered a wide range of subject matter. To investigate whether TE-ALEs are more advantageous for specific subject matter, the studies were divided into three subsets: medical science (human anatomy and physiology, internal diseases, kinesiology and applied anatomy, nursing, surgery Clerkship, etc.), STEM (biology, chemistry, electromagnetism, mathematics, physics, software engineering, etc.), and social science (business, financial accounting, marketing, social psychology, physiology, politics, etc.). Most eligible studies belonged to STEM disciplines. Neither publication period (I²= 58.8%, p > .05) nor subject matter (I²= 28.2%, p > .05) emerged as statistically significant moderators of the effectiveness of TE-ALEs. To this end, the overall effect size for student learning outcome data appears to be robust to varying study conditions rigor of eligible studies, regardless of differences in the year of publication and subject matter.

In the research methodology category, the variables of study design, sample size, and instructor equivalence were tested. There is controversy whether the inclusion of poorly designed and small-sample studies in a meta-analysis corpus might lead to spurious effects (Mean et al., 2013). To address this issue, study method variables were examined as potential moderators. A comparison of both subsets of study design (randomized controlled trial vs. quasi-experimental research design) indicated that the study design was a significant moderator variable, with evidently higher positive effects found in studies designed with randomized control trials (Z = 4.56, p < .001) compared with those with quasi-experimental research designs (Z = 4.35, p < .001). The influence of the study’s sample size was examined by dividing studies into three subsets according to the number of participants involved. The meta-analysis revealed significantly higher positive effects in studies with samples from 60 to 200 (Z = 4.21, p < .001) and fewer than 60 (Z = 3.24, p < .001), when compared with samples above 200 (Z = 2.42, p < .05). Thus, sample size was found to be a statistically significant moderator of the effectiveness of TE-ALEs (I²= 83.5%, p < .05). However, the test for subgroup differences (I²= 0, p = .65 > .05) indicated that no significant difference was found between studies with less than 60 samples and those with samples from 60 to 200 regarding the effect size on college student’s cognitive learning outcomes. Therefore, there is no evidence that the inclusion of small-sample studies in the meta-analysis was responsible for the overall finding of a positive outcome for the TE-ALEs. Finally, moderator analyses showed that effect sizes did not vary depending on whether the same instructor was used across the treatment and control groups (I²= 67.5%, p > .05).

Discussion

The corpus of the effect sizes extracted from the 31 selected journal articles sufficiently demonstrated strong evidence that, overall, TE-ALEs have been documented as more effective than traditional learning environments in improving college students’ cognitive learning outcomes. The results are consistent with some previous studies which stated that the broader conceptualization of active learning has a more positive effect on college students’ cognitive learning outcomes. For example, Freeman et al. (2014) found that active learning interventions increased undergraduate students’ examination performance in the STEM disciplines compared with traditional lecture-based instruction. To the best of our knowledge, the results of this meta-analysis currently provide the broadest synthesis of evidence describing the influence of TE-ALEs toward college students’ cognitive learning outcomes.

The test for homogeneity showed significant variability in the effect sizes for the different studies, justifying a search for moderator variables that could explain differences in learning outcomes between TE-ALEs and traditional learning environments. Eight potential moderator variables were tested from three categories, including instructional experiences, study conditions, and research methodology. A moderator variable analysis identified three moderators (social context, study design, and sample size) that were significant at p < .05. These results are consistent with parallel research topics. For instance, a meta-analysis regarding the effectiveness of online and blended learning, conducted by Means et al. (2013), indicated that the social context was a moderator variable of the online and blended learning effect size. Another meta-analysis regarding the effectiveness of flipped classroom instruction, conducted by Shi et al. (2019), also indicated that the social context was a moderator variable of the flipped classroom instructional effect size.

Effect sizes were larger for individualized learning approaches than collaborative learning approaches employed in TE-ALEs. The results are partially consistent with the findings of Riley and Ward (2017), who examined the effectiveness of active learning versus passive learning methods in an accounting information systems course. The two social contexts (individualized and collaborative) of TE-ALEs and traditional environments (control group) were compared in terms of each groups’ students’ cognitive learning. The results showed that students who worked individually in an active environment performed significantly better in examinations, but no statistical difference was found between students in the collaborative group and the traditional environment. The authors concluded that active learning enhances student outcomes, particularly for those who work individually in TE-ALEs. This pattern of significant moderator variables is also consistent with the interpretation that the potential advantage of active learning methods originates from aspects of the instructional practice that emphasize student-centered participation and engagement as well as higher order thinking (Prince, 2004). Although active learning pedagogies—such as peer learning, team-based learning, and inquiry learning—can certainly work in traditional environments (Deslauriers, Schelew, & Wieman, 2011; Lyon & Lagowski, 2008; Mazur, 2009) with fixed seat settings, which makes peer collaboration difﬁcult and awkward, a better space for these instructional processes would be an active learning classroom designed speciﬁcally for student interaction and engagement (Chiu & Cheng, 2017).

Compared with the traditional environments, the context used in active learning requires students to be engaged learners, and to participate in meaningful learning activities (Prince, 2004) and the learning process, in order to achieve higher order thinking and the ability to construct complex knowledge (Chiu & Cheng, 2017; Kovalchick & Kawson, 2004). Thus, it is not surprising that TE-ALEs yielded better cognitive learning outcomes compared with traditional environments. The treatment duration of the research was not found to be a significant moderator of effects in this meta-analysis of TE-ALEs. This finding is consistent with the results of similar studies conducted by Means et al. (2013) and Shi et al. (2019), which reported that treatment duration caused no significant difference to the effectiveness of other, broader approaches to online/blended learning or flipped classroom instruction on student cognitive learning outcomes.

In this meta-analysis of TE-ALEs, both variables in the category of study conditions (i.e., year of publication and subject matter) were not found to be significant moderators of effects. It is generally assumed that more recent TE-ALEs have introduced more sophisticated pedagogical technologies and more complex instructional practices to the research field. However, the meta-analysis did not show significant differences in the effect size between studies published before 2012 and those published from 2013 through 2019. There was also no significant difference associated with the nature of the subject matter involved. Our findings regarding the subject matter are consistent with the findings of similar studies conducted by Hew and Lo (2018) and Shi et al. (2019), which both stated that variation was not significant when comparing different subjects with flipped classroom studies.

The examination of the influence of research methodology showed that effect sizes varied significantly according to the sample size based on the three groups of classifications (<60, 60–200, and >200). No significant difference was evident between studies with samples fewer than 60 and samples from 60 to 200 regarding effect size. However, the effect sizes of these two subsets were significantly higher than those with a sample size above 200. It seems that active learning in large courses is not as effective as in smaller or more modest counterparts. The results also conform to the argument that enhancing lectures with active learning can be challenging for instructors teaching in large lecture halls with hundreds of students, whose mastery of course content may be compromised (Poirier & Feldman, 2007). Patterson, Kilpatrick, and Woebkenberg (2010) also pointed out that the ability to engage all students actively in large lectures is increasingly more difficult. Active learning methods are often limited by the large lecture format because of the physical structure of the room, or that instructors are unable to hear student responses, and so on (Herreid, 2006). To this end, a variety of new technological strategies, such as audience response systems (Efstathiou & Bailey, 2012) and clickers or student response systems (Green & Repetti, 2015; Patterson et al., 2010), have been introduced into large lectures to increase student engagement, active learning, and deeper learning.

Instructor equivalence (whether the instructor was the same in both the control and the treatment groups) was not found to be a significant moderator, although the effect size was larger when the same instructors were employed. This finding is in accordance with the results of Means et al. (2013) and Shi et al. (2019), which demonstrated that effect sizes did not vary depending on whether or not the same instructor(s) taught in the traditional or treatment conditions. Both randomized controlled trials and quasi-experimental research designs indicated that students in TE-ALEs had significantly higher cognitive learning achievements than students in traditional learning environments. However, the study was found to be a significant moderator of effects in the present meta-analysis, with significantly higher positive effects in studies designed with randomized controlled trials compared with those with a quasi-experimental design. The result suggests students who have the same chance of being assigned to a TE-ALE more consistently achieve better cognitive learning outcomes than students who are not randomly assigned to a TE-ALE.

Conclusion

TE-ALEs have been widely studied in terms of its effect on the cognitive learning outcomes of college students, with particular focus on the disciplines of medical education and STEM education. This study synthesized the findings of 31 high-quality peer-reviewed, empirical research articles to provide clarity on the influence of TE-ALEs toward students’ cognitive learning outcomes in higher education. Overall, the meta-analysis shows that TE-ALEs significantly improve college students’ cognitive learning outcomes when compared with traditional learning environments.

In addition, TE-ALEs are observed as most effective when instructors employ individualized learning contexts and when they use the most rigorous research methodology. As shown in Table 2, individualized learning has a more positive effect on students’ cognitive learning outcomes when compared with that of collaborative learning approaches. Given that this study examines TE-ALEs, this is probably due to the instructors’ strategy of engaging every single student with technology, rather than grouping individuals in a way which may allow for social loafing. Alternatively, collaborative learning may result in a lower overall percentage of per-capita technology engagement because the technology is being shared among individuals. Furthermore, studies with randomized control trials showed an overall more positive effect on students’ cognitive learning outcomes when compared with quasi-experimental research designs. As a result, it was concluded that when potential for bias is reduced to the fullest extent possible, TE-ALEs have been observed most effectively.

The implications of this study build on the recognition that active learning processes, whether individualized or collaborative, require higher degrees of engagement and participation in the classroom and result in more positive cognitive learning outcomes. However, when relying upon TE-ALEs, individualized technology application may be more influential toward students’ cognitive learning. This key finding can be interpreted in at least three ways. (a) From a researcher’s perspective, inconsistent findings within this topic are likely due to variability in the social context. In other words, the simple implementation of a TE-ALE is not guaranteed to improve students’ learning outcomes when instructors’ do not customize pedagogy appropriately. (b) From a practitioner’s perspective, it is critical that emphasis is placed on individualized active learning approaches, which force every student to be satisfactorily engaged and participating with the technology. In addition, instructors should embed a wide variety of learning activities requiring more engagement and high-order thinking skills into their existing curricula, regardless of whether they are implementing individualized or collaborative TE-ALEs. (c) From the perspective of academic administrators, institutions need to organize and align opportunities for professional development and continued education related to the implementation of active learning methods. Research has shown that, despite much positive recognition, active learning methods are not the most common approaches being utilized in higher education (Stains et al., 2018). Therefore, administrators should be working actively to facilitate the cultivation of initiatives such as classroom observation, on-the-job training, and professional research groups, which have all been linked with instructional success and improved student learning outcomes (Jackson & Bruegmann, 2009).

The effectiveness of the TE-ALEs in large courses has been particularly topical, especially in higher education. Because the effectiveness of the TE-ALEs in large courses is significantly lower than in relatively small courses, there is a need for more concrete technologies and technological approaches that focus on the facilitation of individualized active learning. Active learning methods are important even in large environments, as reflected in the study conducted by Hake (1998). To this end, more empirical research is required that considers appropriate active learning methods in the implementation of TE-ALEs into large courses in order to provide references and suggestions for the improvement of TE-ALEs.

However, the results of this meta-analysis are subject to some limitations. First, this review focused on college student participants. To broaden the scope of such a review, future meta-analyses should isolate and examine different participant demographics, such as participants from primary and secondary education. Second, this study examined the topic from the perspective of cognitive learning and did not consider other potentially important perspectives of learning outcomes, such as affective or behavioral outcomes, or other positively related developmental attributes, such as knowledge retention, motivation, or long-term variables of degree completion. Third, several of the selected studies had insufficient information to be included with accuracy in the moderator analysis. To this end, future research should include more detailed reporting processes to support moderator variable analysis, enabling the expansion of knowledge in the field through future meta-analytic research on this topic. In addition, the difficulty in defining a TE-ALE as well as the tendency of publications to report only those with positively significant effects made it difficult to completely reduce the potential for publication bias.

Due to the significance of TE-ALEs, it is expected that high-quality, well-designed studies will continue to emerge. This study provides an initial, multidisciplinary snapshot of evidence, which suggests that researchers, practitioners, and policy-makers should continue to support and study TE-ALEs, due to the positive influence between TE-ALEs and students’ cognitive learning. However, the topic of technological learning environments is a dynamic and multifaceted issue. With increasing research prevalence, subject-specific and learning model-specific findings can, and should, be examined periodically to maintain objective and systematic understandings of these phenomena.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by the Ministry of Education of the Humanities and Social Science project (Project No.: 18YJC880074).

Yinghui Shi is a lecturer of National Engineering Research Centre for E-Learning at Central China Normal University. Dr. Shi's main areas of interest are interactive media, information and communication technology in education, e-learning, and information literacy.

Huiyun Yang is a graduate student at Central China Normal University. Her research interests include information and communication technology in education and e-learning.

Jason MacLeod is the associate vice president for operations and administration at D’Youville College. He is currently studying with Professor Harrison Hao. His research interests include learning environment design, technology acceptance, and e-learning experience management.

Jingman Zhang is a graduate student at Central China Normal University. Her research interests include information and communication technology in education and e-learning.

Harrison Hao Yang is a professor of the School of Education at the State University of New York at Oswego, NY. Dr. Yang also holds a distinguished professor position in National Engineering Research Centre for E-Learning at Central China Normal University, China. His research specialties include assessment and e-folios, distance/flexible education, information literacy, information technology diffusion/integration, e-learning and learning communities, and so on.

References

References marked with an asterisk indicate studies included in the meta-analysis.

*Alcalde

Nagel

(2019) Why does peer instruction improve student satisfaction more than student performance? A randomized experiment. International Review of Economics Education 30: 100149 . doi:10.1016/j.iree.2018.10.001.

Andrews

T. M.

Leonard

M. J.

Colgrove

C. A.

Kalinowski

S. T.

(2011) Active learning not associated with student learning in a random sample of college biology courses. CBE-Life Sciences Education 10: 394–405. doi:10.1187/cbe.11-07-0061.

*Aşıksoy

Özdamli

(2016) Flipped classroom adapted to the arcs model of motivation and applied to a physics course. Eurasia Journal of Mathematics, Science & Technology Education 12(6): 1589–1603. doi:10.12973/eurasia.2016.1251a.

*Baepler

Walker

J. D.

Driessen

(2014) It’s not about seat time: Blending, flipping, and efficiency in active learning classrooms. Computers & Education 78: 227–236. doi:10.1016/j.compedu.2014.06.006.

*Baranowski

M. K.

(2006) Single session simulations: The effectiveness of short congressional simulations in introductory American government classes. Journal of Political Science Education 2(1): 33–49. doi:10.1080/15512160500484135.

Baranowski

M. K.

Weir

K. A.

(2015) Political simulations: What we know, what we think we know, and what we still need to know. Journal of Political Science Education 11(4): 391–403. doi:10.1080/15512169.2015.1065748.

*Baytiyeh

Naja

M. K.

(2017) Students’ perceptions of the flipped classroom model in an engineering course: A case study. European Journal of Engineering Education 42(6): 1048–1061. doi:10.1080/03043797.2016.1252905.

Beichner

R. J.

(2014) History and evolution of active learning spaces. In: Baepler

Brooks

D. C.

Walker

J. D.

(eds) New directions for teaching and learning, special issue: Active Learning Spaces, San Francisco, CA: Jossey-Bass, pp. 9–16.

10.

Beichner, R. J., Saul, J., Abbott, D., Morse, J. J., Deardorff, D. L., Allain, R., … Risley, J. S. (2007). The student-centered activities for large enrollment undergraduate programs (SCALE-UP) project. Retrieved from https://www.compadre.org/Repository/document/ServeFile.cfm?ID=4517&DocID=183.

11.

Bernard

R. M.

Abrami

P. C.

Lou

Borokhovski

Wade

Wozney

Huang

(2004) How does distance education compare with classroom instruction? A meta-analysis of the empirical literature. Review of Educational Research 74(3): 379–439. doi:10.3102/00346543074003379.

12.

*Blázquez

B. O.

Masluk

Gascon

Díaz

R. F.

Aguilar-Latorre

Magallón

I. A.

Botaya

R. M.

(2019) The use of flipped classroom as an active learning approach improves academic performance in social work: A randomized trial in a university. PLoS One 14(4): e0214623 . doi:10.1371/journal.pone.0214623.

13.

Bonwell, C. C., & Eison, J. A. (1991). Active learning: Creating excitement in the classroom (ASHEERIC Higher Education Report No. 1). Washington, DC: The George Washington University.

14.

Bouroumi

Fajr

(2014) Collaborative and cooperative e-learning in higher education in Morocco: A case study. International Journal of Emerging Technologies in Learning 9(1): 66–72.

15.

*Brooks

D. C.

(2011) Space matters: The impact of formal learning environments on student learning. British Journal of Educational Technology 42(5): 719–726. doi:10.1111/j.1467-8535.2010.01098.x.

16.

Brooks, D. C. (2017). Active learning classrooms: The topic strategic technology for 2017 (EDUCAUSE Center for Analysis and Research Report). Retrieved from https://er.educause.edu/blogs/2017/3/active-learning-classrooms-the-top-strategic-technology-for-2017.

17.

*Chiu

P. H. P.

Cheng

S. H.

(2017) Effects of active learning classrooms on student learning: A two-year empirical investigation on student perceptions and academic performance. Higher Education Research & Development 36: 269–279. doi:10.1080/07294360.2016.1196475.

18.

Cochrane Collaboration. (2014). Review manager (Version 5.3) [Computer software]. Retrieved from https://community.cochrane.org/help/tools-and-software/revman-5/revman-5-download.

19.

*Cotner

Loper

Walker

J. D.

Brooks

(2013) “It’s not you, it’s the room”—Are the high-tech, active learning classrooms worth it? Journal of College Science Teaching 42(6): 82–88. doi:10.2505/4/jcst13_042_06_82.

20.

*Cotta

K. I.

Shah

Almgren

M. M.

Macias-Moriarity

L. Z.

Mody

(2016) Effectiveness of flipped classroom instructional model in teaching pharmaceutical calculations. Currents in Pharmacy Teaching and Learning 8: 646–653. doi:10.1016/j.cptl.2016.06.011.

21.

*Crimmins

M. T.

Midkiff

(2017) High structure active learning pedagogy for the teaching of organic chemistry: Assessing the impact on academic outcomes. Journal of Chemical Education 94(4): 429–438. doi:10.1021/acs.jchemed.6b00663.

22.

Cruz

da Silva

F. Q. B.

Capretz

L. F.

(2015) Forty years of research on personality in software engineering: A mapping study. Computers in Human Behavior 46: 94–113. doi:10.1016/j.chb.2014.12.008.

23.

Deslauriers

Schelew

Wieman

(2011) Improved learning in a large-enrollment physics class. Science 332(6031): 862–864. doi:10.1126/science.1202043.

24.

Dori

Y. J.

Belcher

(2005) How does technology-enabled active learning affect undergraduate students’ understanding of electromagnetism concepts? The Journal of the Learning Science 14(2): 243–279. doi:10.1207/s15327809jls1402_3.

25.

Dori

Y. J.

Belcher

Bessette

Danziger

McKinney

Hult

(2003) Technology for active learning. Materials Today 6(12): 44–49. doi:10.1016/S1369-7021(03)01225-2.

26.

*Dori

Y. J.

Hult

Breslow

Belcher

J. W.

(2007) How much have they retained? Making unseen concepts seen in a freshman electromagnetism course at MIT. Journal of Science Education and Technology 16(4): 299–323. doi:10.1007/s10956-007-9051-9.

27.

Efstathiou

Bailey

(2012) Promoting active learning using Audience Response System in large bioscience classes. Nurse Education Today 32(1): 91–95. doi:10.1016/j.nedt.2011.01.017.

28.

*Entezari

Javdan

(2016) Active learning and flipped classroom, hand in hand approach to improve students learning in human anatomy and physiology. International Journal of Higher Education 5(4): 222–231. doi:10.5430/ijhe.v5n4p222.

29.

Freeman

Eddy

S. L.

McDonough

Smith

M. K.

Okoroafor

Jordt

Wenderoth

M. P.

(2014) Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences of the United State of America 111(23): 8410–8415. doi:10.1073/pnas.1319030111.

30.

*Ghorbani

Ghazvini

(2016) Using paper presentation breaks during didactic lectures improves learning of physiology in undergraduate students. Advances in Physiology Education 40(1): 93–97. doi:10.1152/advan.00137.2015.

31.

Gilies

R. M.

Ashman

A. F.

Terwel

(2008) The teacher’s role in implementing cooperative learning in the classroom, New York, NY: Springer.

32.

Glass

G. V.

(1976) Primary, secondary, and meta-analysis. Educational Researcher 5: 3–8. doi:10.3102/0013189X005010003.

33.

*Green

A. J.

Repetti

(2015) Clickers: A strategy for active learning in a hospitality classroom. Journal of Hospitality & Tourism Education 27(4): 171–179. doi:10.1080/10963758.2015.1089513.

34.

Hake

(1998) Interactive engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. American Journal of Physics 66(1): 64–74. doi:10.1119/1.18809.

35.

*He

Holton

Farkas

Warschauer

(2016) The effects of flipped instruction on out-of-class study time, exam performance, and student perceptions. Learning & Instruction 45: 61–71. doi:10.1016/j.learninstruc.2016.07.001.

36.

Hedges

L. V.

Olkin

(1985) Statistical methods for meta-analysis, Orlando, FL: Academic Press.

37.

Herreid

C. F.

(2006) “Clicker” cases: Introducing case study teaching into large classrooms. Journal of College Science Teaching 36(2): 43–47.

38.

*Heuett

W. J.

(2017) Flipping the math classroom for non-math majors to enrich their learning experience. Problems, Resources, and Issues in Mathematics Undergraduate Studies (Primus) 27(10): 889–907. doi:10.1080/10511970.2016.1256925.

39.

Hew

K. F.

C. K.

(2018) Flipped classroom improves student learning in health professions education: A meta-analysis. BMC Medical Education 18(1): 38 . doi:10.1186/s12909-018-1144-z.

40.

Higgins

J. P.

Thompson

S. G.

Deeks

J. J.

Altman

D. G.

(2003) Measuring inconsistency in meta-analyses. British Medical Journal 327(7414): 557–560.

41.

Hoeve

Stams

G. J.

van der Put

C. E.

Dubas

J. S.

van der Laan

Gerris

J. R. M.

(2012) A meta-analysis of attachment to parents and delinquency. Journal of Abnormal Child Psychology 40(5): 771–785. doi:10.1007/s10802-011-9608-1.

42.

*Hung

H. T.

(2015) Flipping the classroom for English language learners to foster active learning. Computer Assisted Language Learning 28(1): 81–96. doi:10.1080/09588221.2014.967701.

43.

Jackson

C. K.

Bruegmann

(2009) Teaching students and teaching each other: The importance of peer learning for teachers. American Economic Journal: Applied Economics 1(4): 85–108. doi:10.1257/app.1.4.85.

44.

Kalaian, S. A., & Kasim, R. M. (2014). A meta-analytic review of studies of the effectiveness of small-group learning methods on statistics achievement. Journal of Statistics Education, 22(1). doi:10.1080/10691898.2014.11889691.

45.

Kovalchick

Kawson

(2004) Education and technology: An encyclopedia (Vol. 1), Santa Barbara, CA: ABC-CLIO.

46.

*Krain

Lantis

(2006) Building knowledge? Evaluating the effectiveness of the global problems summit simulation. International Studies Perspectives 7(4): 395–407. doi:10.1111/j.1528-3585.2006.00261.x.

47.

*Lewis

C. E.

Chen

D. C.

Relan

(2018) Implementation of a ﬂipped classroom approach to promote active learning in the third-year surgery clerkship. The American Journal of Surgery 215: 298–303. doi:10.1016/j.amjsurg.2017.08.050.

48.

*Lin

Y.-T.

(2019) Impacts of a flipped classroom with a smart learning diagnosis system on students’ learning performance, perception, and problem solving ability in a software engineering course. Computers in Human Behavior 95: 187–196. doi:10.1016/j.chb.2018.11.036.

49.

Lyon

D. C.

Lagowski

J. J.

(2008) Effectiveness of facilitating small-group learning in large lecture classes. Journal of Chemical Education 85(11): 1571–1576. doi:10.1021/ed085p1571.

50.

MacLeod

Yang

H. H.

Zhu

(2018) Understanding students’ preferences toward the smart classroom learning environment: Development and validation of an instrument. Computers & Education 122: 80–91. doi:10.1016/j.compedu.2018.03.015.

51.

Mazur

(2004) Interactive teaching: Promoting better learning using peer instruction and just-in-time teaching, Upper Saddle River, NJ: Prentice Hall.

52.

Mazur

(2009) Farewell, lecture? Science 323(5910): 50–51. doi:10.1126/science.1168927.

53.

*McCarroll

M. L.

Pohle-Krauza

R. J.

Martin

J. L.

(2009) Active learning in the classroom: A muscle identiﬁcation game in a kinesiology course. Advances in Physiology Education 33(4): 319–322. doi:10.1152/advan.00013.2009.

54.

Means

Toyama

Murphy

Baki

(2013) The effectiveness of online and blended learning: A meta-analysis of the empirical literature. Teachers College Record 115(3): 1–47. doi:10.1080/10888438.2011.624566.

55.

*Mendini

Peter

P. C.

(2019) Research note: The role of smart versus traditional classrooms on students’ engagement. Marketing Education Review 29(1): 17–23. doi:10.1080/10528008.2018.1532301.

56.

Michaelowa

(2010) Primary education quality in francophone Sub-Saharan Africa: Determinants of learning achievement and efficiency considerations. World Development 29(10): 1699–1716. doi:10.1016/S0305-750X(01)00061-4.

57.

Mueller

A. L.

Knobloch

N. A.

Orvis

K. S.

(2015) Exploring the effects of active learning on high school students’ outcomes and teachers’ perceptions of biotechnology and genetics instruction. Journal of Agricultural Education 56(2): 138–152. doi:10.5032/jae.2015.02138.

58.

Naing

Wai

V. N.

Durham

Whittaker

M. A.

Win

N. N.

Aung

Mak

J. W.

(2015) A systematic review and meta-analysis of medical students’ perspectives on the engagement in research. Medicine 94(28): e1089 . doi:10.1097/MD.0000000000001089.

59.

Parishan

Jafari

E. M.

Nosrat

(2011) The effect of technology enabled active learning (TEAL) method in biology on the academic achievements of students. Procedia—Social and Behavioral Sciences 28: 542–546. doi:10.1016/j.sbspro.2011.11.104.

60.

Partnership for 21st Century Learning. (2007). Framework for 21st century learning. Retrieved from http://www.battelleforkids.org/networks/p21/frameworks-resources.

61.

Patterson

Kilpatrick

Woebkenberg

(2010) Evidence for teaching practice: The impact of clickers in a large classroom environment. Nurse Education Today 30: 603–607.

62.

Peplow, M. (2017). Social Sciences suffer from severe publication bias. Retrieved from https://www.nature.com/news/social-sciences-suffer-from-severe-publication-bias-1.15787.

63.

*Poirier

C. R.

Feldman

R. S.

(2007) Promoting active learning using individual response technology in large introductory psychology classes. Teaching of Psychology 34(3): 194–196. doi:10.1080/00986280701498665.

64.

Prince

(2004) Does active learning work? A review of the research. Journal of Engineering Education 93(3): 223–231. doi:10.1002/j.2168-9830.2004. tb00809.x.

65.

*Rau

M. A.

Kennedy

Oxtoby

Bollom

Moore

J. W.

(2017) Unpacking “active learning”: A combination of flipped classroom and collaboration support is more effective but collaboration support alone is not. Journal of Chemical Education 94(10): 1406–1414. doi:10.1021/acs.jchemed.7b00240.

66.

Riley

Ward

(2017) Active learning, cooperative active learning, and passive learning methods in an accounting information systems course. Issues in Accounting Education 32(2): 1–16. doi:10.2308/iace-51366.

67.

Rosenthal

(1979) The file-drawer problem and tolerance of null result. Psychological Bulletin 86(3): 638–641. doi:10.1037/0033-2909.86.3.638.

68.

Rosenthal

(1991) Meta-analytic procedures for social research (Vol. 86), Newbury Park, CA: SAGE.

69.

Ruiz-Primo

M. A.

Briggs

Iverson

Talbot

Shepard

L. A.

(2011) Impact of undergraduate science course innovations on learning. Science 331(6022): 1269–1270. doi:10.1126/science.1198976.

70.

Ryan

M. D.

Reid

S. A.

(2016) Impact of the flipped classroom on student performance and retention: A parallel controlled study in general chemistry. Journal of Chemical Education 93(1): 13–23. doi:10.1021/acs.jchemed.5b00717.

71.

Seery

M. K.

(2015) Flipped learning in higher education chemistry: Emerging trends and potential directions. Chemistry Education Research and Practice 16(4): 758–768. doi:10.1039/C5RP00136F.

72.

*Sevindik

(2010) Future’s learning environments in health education: The effects of smart classrooms on the academic achievements of the students at Health College. Telematics and Informatics 27(3): 314–322. doi:10.1016/j.tele. 2009.08.001.

73.

Shi, Y. H, Ma, Y. Q, MacLeod, J., & Yang, H. H. (2019). College students’ cognitive learning outcomes in flipped classroom instruction: A meta-analysis of the empirical literature. Journal of Computers in Education. Advance online publication. doi:10.1007/s40692-019-00142-8.

74.

Shi, Y. H, Peng, C. L., Wang, S. M., & Yang, H. H. (2018). The effects of smart classroom-based instruction on college students’ learning engagement and internet self-efficacy. In S. Cheung, L. Kwok, K. Kubota, L. K. Lee, & J. Tokito (Eds.), 2018 International conference on blended learning (pp. 263–274). New York, NY: Springer. doi:10.1007/978-3-319-94505-7_21.

75.

Shieh

R. S.

(2012) The impact of technology-enabled active learning (TEAL) implementation on student learning and teachers’ teaching in a high school context. Computers & Education 59: 206–214. doi:10.1016/j.compedu.2012.01.016.

76.

*Shieh

R. S.

Chang

Liu

E. Z.-F.

(2011) Technology enabled active learning (TEAL) in introductory physics: Impact on genders and achievement levels. Australasian Journal of Educational Technology 27(7): 1082–1099. doi:10.14742/ajet.905.

77.

*Shieh

R. S.

Chang

Tang

(2010) The impact of implementing technology-enabled active learning (TEAL) in university physics in Taiwan. The Asia Pacific Education Researcher 19(3): 401–415. doi:10.3860/taper.v19i3.1850.

78.

Springer

Stanne

M. E.

Donovan

S. S.

(1999) Effects of small-group learning on undergraduates in science, mathematics, engineering, and technology: A meta-analysis. Review of Educational Research 69(1): 21–51. doi:10.3102/00346543069001021.

79.

Stains

Harshman

Barker

M. K.

Chasteen

S. V.

Cole

DeChenne-Peters

S. E.

Levis-Fitzgerald

(2018) Anatomy of STEM teaching in North American universities. Science 359(6383): 1468–1470. doi:10.1126/science.aap8892.

80.

*Tune

J. D.

Sturek

Basile

D. P.

(2013) Flipped classroom model improves graduate student performance in cardiovascular, respiratory, and renal physiology. Advances in Physiology Education 37(4): 316–320. doi:10.1152/advan.00091.2013.

81.

*Welch

(2013) Effectiveness of classroom response systems within an active learning environment. Journal of Nursing Education 52(11): 653–656. doi:10.3928/01484834-20131014-01.

82.

*Wilson

L. V. E.

(2012) An assessment of learning outcomes based on a comparison of active learning and traditional lecture pedagogical styles in a legal environment classroom. Southern Journal of Business & Ethics 4: 101–110.