Interactive Response System to Promote Active Learning in Intelligent Learning Environments

Abstract

This research used a comparative quasi-experimental design to investigate the impacts of an IRS in the ILE on students’ academic performance, cognitive load, and satisfaction with the lesson. A total of 31 middle school students were divided into the experimental group and the control group. Mann–Whitney U tests yielded three major results. (1) Significant differences in academic performance were noted between the two groups. The knowledge test scores of comprehensive questions showed significant differences between the two groups but those of the strict memorization questions did not. Students in the experimental group had overall higher knowledge test scores than those in the control group. (2) Significant differences in cognitive load were observed between the two groups; specifically, students in the experimental group had lower total cognitive load than those in the control group. (3) Significant differences in satisfaction with the lesson were observed between the two groups; specifically, students who studied with the IRS were more satisfied with the lesson than those who studied without the IRS. These findings imply that the IRS is a useful tool that could promote active learning amongst students in ILEs and help to promote teaching and learning efficiency in ILEs.

Keywords

Intelligent learning environments (ILEs)interactive response system (IRS)active learning academic performance cognitive load satisfaction

Introduction

Developments in new technology, such as artificial intelligence, the internet of things, and big data technology, have enabled the use of intelligent learning environments (ILEs) equipped with smart devices in education to promote student learning (Ouf et al., 2017; Lee, 2020). Many schools have proposed strategies to promote the application of ILEs and optimize the presentation of teaching content, facilitate the acquisition of learning resources, and promote classroom interaction (Visvizi et al., 2018; Leem & Sung, 2019). ILEs could lead to good performance and high learning efficiency amongst students when used properly (Liu, 2016). However, several problems remain in the application of ILEs. For example, instructors often do not fully utilize the possibilities of the technology, which leads to low performance and wastage of smart resources (Kuang et al. 2018). Moreover, classes often remain teacher-centered rather than student-centered. In teacher-centered class, teachers are always the knowledge provider and students have limited time to explore by themselves, as a result, students are passive in teacher-centered classes and the interaction between teachers and students is monotonous and shallow (Li et al., 2018). On the contrary, student-centered classes can provide multimodality and deep interaction for students to promote their interests and initiative; as a result, students are positive in class (Yang & Wang, 2021; Cai et al., 2021). The present study incorporates an interactive response system (IRS) into an ILE to solve these problems and to promote active learning.

An IRS is a teaching application that provides real-time feedback to instructors and learners. It helps instructors to fully evaluate student comprehension of the learning materials based on big data and it provides different teaching approaches supported by technology, such as educational games for learners (Slain et al., 2004). An IRS combines several different technologies, such as learning analysis and big data in ILEs, thereby leading to the enhanced use of smart resources. It is also a useful tool to enhance interactions between students and instructors, students and students, and students and techniques (Li et al., 2018). This feature renders the class more student-centered and increases the opportunities and channels through which students can interact, communicate, and discuss in class. In addition, students can give feedback to instructors through a clicker without raising their hands (Kioutsiouki & Demetriadis, 2014), which could make them feel more at ease when engaging in classroom discussions. Reports from as early as the 1970s on the use of IRSs in educational programs indicated that such systems could serve as useful tools in traditional large classrooms (Littauer, 1972; Shieh & Chang, 2013; Fuad et al., 2018) and promote students’ learning interests and satisfaction with the lesson (Slain et al., 2004).

Therefore, this study chose IRS as a tool to solve the problems in ILEs mentioned above and to investigate the effectiveness of the IRS in ILEs to improve students’ academic performance, control cognitive load, and promote satisfaction with the lesson. In the literature review that follows, a description of ILEs will be given, followed by a section on IRSs, and a section on cognitive load. The literature review ends with the research questions that guided this study and the hypotheses.

Literature Review

Intelligent Learning Environments

ILEs are also called smart learning environments. Whilst many researchers have focused on defining ILEs, no unified definition of this term currently exists (Zhu et al. 2016). Shannon et al. (2012) proposed that ILEs could provide self-learning, self-motivated, and personalized services. In such an environment, learners could access personalized learning content according to their requirements; thus, learners were the central focus of ILEs. Koper (2014) defined ILEs as physical environments equipped with digital, context-aware, and adaptive devices to promote better and faster learning. Huang et al. (2012) summarized five technical characteristics of ILEs:

(1) Tracking of the learning process: ILEs are often equipped with several webcams, which can capture students’ behavior, facial expression, eye movements, and interactions automatically over a 360° point of view. Students’ learning progress, academic achievements, system logins, access to learning resources, terms searching, and engagement can be recorded whilst using smart mobile devices for learning. Additionally, all of the data are synchronized to the cloud to provide an important basis for learning analysis.

(2) Recognition of learning scenarios: ILEs can recognize learners’ personal information, including prior knowledge, learning styles, learning abilities, cognitive features, interests, emotional state, scenario information, learning time, place, partners, and activities. Such functions in ILEs can be achieved by wireless sensor networks (Alassery, 2019), learner modeling (Lin et al., 2019), learning analysis (Chiu & Tseng, 2021), and affective computing (Popescu et al., 2018). The ILEs can then provide different learning resources and materials that are suitable for the learner to realize personalized learning.

(3) Awareness of the physical environment: ILEs often apply sensor technology to monitor air, temperature, illumination, sound, smell, and other physical environmental factors to provide a comfortable physical environment for learners.

(4) Connection of learning communities: ILEs can establish learning communities based on specific learning situations and provide technological support for learners to communicate and interact with other learners in these communities effectively.

(5) Easy, engaged, and effective learning: The aim of ILEs is to promote relaxed, committed, and effective learning for learners based on the above functions. In addition, ILEs must include various devices, such as smartphones, laptops, Google Glasses, and wearable devices (Naimi & Westreich, 2014) and can apply new instructional styles and technological support to immersive learning, cooperative learning, inquiry-based learning, mobile learning, and any other types of learning approach (Gros, 2016). A typical ILE is shown in Figure 1.

Figure 1.

Intelligent learning environment.

However, since ILEs are complex and unfamiliar to instructors and students, some problems may arise. First, the associated technologies and devices may not always be fully utilized due to the limited ability in information technology applications of instructors; therefore, they often only apply interactive whiteboards as an electronic blackboard to present slides and videos in ILEs. Unfortunately, these whiteboards do not facilitate deep interactions with learners or provide real-time feedback based on learners’ results and personal characteristics (Li et al., 2018). In addition, the data of learners provided by the technologies in ILEs is not always used by instructors. Second, since instructors often take dominant positions in the class setting, 78.57% of the technologies in ILEs are used by instructors to explain knowledge and only 21.43% is used by students in specific educational practice (Li et al., 2018). Therefore, learners cannot achieve deep interactions with instructors, their partners, or technologies. Third, because the smart equipment, such as cameras and microphones, could capture their behaviors and collect their voice, learners may sometimes feel nervous and shy when they are learning in ILEs. Some students also have a fear of speaking in public (Jiang, 2016). Therefore, developing methods to achieve deep interactions and take advantage of the available technologies to promote student learning is of vital importance for the instructional design of ILEs to improve learning efficiency.

Interactive Response Systems

IRSs have been widely used as popular teaching aids to enhance interactions between students, instructors, and technologies and promote active learning since the 1970s. Recent research found that learning with such a system was effective for students’ learning processes (Camacho-Miñano & del Campo, 2016) and could result in higher test scores (Lantz & Stawiski, 2014). A typical IRS usually includes a computer and specific software, a projector and screen to present questions, a radio signal receiver unit or a directly wired receiver unit and a personal clicker which is used by individual students to respond to questions. In ILEs, IRSs can be smarter than typical response systems. Mobile devices are used as clickers; meanwhile, test questions are presented on these devices, and students submit their answers, opinions, and comments on the questions by using mobile devices (Bonaiuti et al., 2015). IRSs support several types of questions, such as yes/no questions, single and multiple-choice questions, subjective questions, application questions, and creative works. Then, individual responses to questions can be aggregated as bar graphs or percentages and be made available on the screen of the interactive whiteboard for viewing by the instructor and attendees (Slain et al., 2004). When students respond to subjective questions or creative works, instructors and other students can check these answers and discuss interesting ideas online. Additionally, students can ask questions online via the IRS without raising hands whenever they are confused and instructors can respond to them in real time. The IRS can be used as a game-based technology to select students randomly to answer questions, establish competitions, and give ranks. The IRS can overcome the limitations of time and space to achieve interactions between instructors and students, so it can be used inside and outside the class. Before class, instructors can submit learning materials online and students can discuss their questions or interests with the instructors and other students through the IRS. This application of IRS can raise teaching and learning quality and improve communication (Caldwell, 2007). After class, the IRS has tracking features that help instructors access to students’ answers on a cloud-based database. In this manner, they can fully understand students’ comprehension of the learning content for the next stage of instructional design (Pirahandeh & Kim, 2017).

Cognitive Load

Cognitive load theory is based on the idea that the cognitive capacity of working memory is limited, that learning activities consume cognitive resources, and that learning is impeded if a task asks for an amount of cognitive resources that exceeds the limited capacity (Sweller & Chandler, 1991). Cognitive load is defined as the total amount of cognitive resources needed from working memory during a specific period of time (Sweller, 1988). There are four factors that affect cognitive load: prior knowledge, the complexity of the learning materials, the organizational form, and the presentation mode of learning materials (Leahy et al., 2003). In ILEs, there are many instructional hardware and software smart devices and the application of such devices may cause changes in learners’ cognitive load by changing their prior knowledge, the complexity of learning materials, the organizational form and the presentation mode of learning materials (Gao et al., 2017). Therefore, we take learners’ cognitive load as an index to indicate learners’ learning effects when applying IRS in ILEs.

This Study

Because many studies have proven that IRSs are useful tools to improve students’ learning and enhance interactions between instructors and learners (Oigara, 2015; Huang, 2016) in traditional learning environments, this research applied an IRS to an ILE to solve the existing problems described earlier in ILEs and improve active learning by using a comparative quasi-experimental method. The following questions guided our analysis and are addressed in this study:

• What are the differences in students’ academic performance when they learn with and without an IRS in an ILE?

• What are the differences in students’ cognitive load when they learn with and without an IRS in an ILE?

• What are the differences in students’ satisfaction with the lesson when they learn with and without an IRS in an ILE?

Three hypotheses based on the literature review and research questions are postulated:

• The academic performance of students who learn with an IRS in the ILE will be higher than that of students learning without an IRS.

• The total cognitive load of students who learn with an IRS in the ILE will be higher than that of students learning without an IRS.

• Students will be more satisfied with the lesson when they learn with an IRS in the ILE than when they learn without an IRS.

Method

Research Design

This research adopted a comparative quasi-experimental design. Two classes were assigned to an experimental group or a control group. The experimental group was taught a lesson about the lever principle via an ILE with an IRS, and the control group was taught the same lesson in the same environment without an IRS. After learning, a knowledge test, a cognitive load test, and a satisfaction survey were administered. Also, an unstructured interview was conducted to collect supplementary information.

Participants

A total of 31 students (13 males and 18 females) from two seventh grade classes of a middle school in Changchun, China participated in this experiment. The average age of the participants was 14.05 years (SD = 0.71). All of the students had a half-year learning experience in ILEs. Students’ physics scores from the last 6 months were compared between the two classes by a Mann–Whitney U test and the results showed no significant differences. This finding, together with both classes being taught by the same instructor and using the same learning materials, indicated that the participants of the two classes were comparable and at the same physics level. Therefore, one class was assigned to the experimental group, which used an IRS in an ILE (n = 16), and the other was assigned to the control group using a traditional lecture format without an IRS in the ILE (n = 15).

Instruments

Knowledge Test

A knowledge test was conducted to assess participants’ strict memory of the concepts by using memorization questions and their analytical ability by using comprehensive questions. The knowledge test was based on the learning materials. Four test items were multiple-choice questions, three test items were yes/no questions, and three test items were fill-in-the-blank questions. Examples of questions are shown in Table 1. Students could earn 10 points for each question and obtain a maximum score of 100. The scoring scheme and all 10 knowledge test questions were examined via the Delphi method by two professors in the field of instructional technology and two middle school physics teachers to ensure the reliability of the test. All four specialists opined that none of the test questions should be deleted.

Table 1.

Examples of different categories of questions.

Categories of questions	Examples
Multiple-choice question	Which of the following item is Laborious leverage? A. Hammer B. Tweezers C. Scissors D. Mouse
Yes/No question	The use of chopsticks is based on the principle of leverage. Yes/No
Fill-in-the-blank question	The lever of the resistance arm equal to the power arm is _____ lever

NASA-TLX

The NASA-TLX was conducted to measure learners’ cognitive load. It is a classical and mature rating scale and considered to be an effective tool to measure learners’ cognitive load (Nygren, 1991; Hill et al., 1992). It has a high reliability and sensitivity and is more widely used than other cognitive load measurement scales (Sun & Liu, 2013). Xiao (2005, 2010) used the NASA-TLX twice to investigate individuals’ cognitive load in education and their results also showed that the scale had a good reliability and validity. The NASA-TLX included six components: mental demand (MD), physical demand (PD), temporal demand (TD), effort (EF), performance (OP), and frustration level (FR) and these six components can be considered appropriate indices to measure students’ cognitive load during their learning process (Hart & Staveland, 1988). The specific definitions of the six components are shown in Table 2. Prior to using the scale, the participants were given a specific description of each component and how to use it.

Table 2.

NASA-TLX rating scale definitions.

Components	Endpoints	Descriptions
Mental demand	Low/high	How much mental and perceptual activity was required (e.g., thinking, deciding, calculating, remembering, looking, and searching)? Was the task easy or demanding, simple or complex, exacting or forgiving?
Physical demand	Low/high	How much physical activity was required (e.g., pushing, pulling, turning, controlling, and activating)? Was the task easy or demanding, slow or brisk, slack or strenuous, restful or laborious?
Temporal demand	Low/high	How much time pressure did you feel due to the rate or pace at which the tasks or task elements occurred? Was the pace slow and leisurely or rapid and frantic?
Performance	Good/poor	How successful do you think you were in accomplishing the goals of the task set by the experimenter (or yourself)? How satisfied were you with your performance in accomplishing these goals?
Effort	Low/high	How hard did you have to work (mentally and physically) to accomplish your level of performance?
Frustration level	Low/high	How insecure, discouraged, irritated, stressed and annoyed versus secure, gratified, content, relaxed and complacent did you feel during the task?

Satisfaction Questionnaire

A satisfaction questionnaire was used to examine the learning satisfaction of the participants in ILEs with or without an IRS. This questionnaire included 13 items (Table 3) that were adopted from the satisfaction questionnaire of Slain et al. (2004). The original satisfaction questionnaire was used for a general classroom with an IRS but not specifically for ILEs. Therefore, we modified some items to suit the ILE. The questions in the modified questionnaire were evaluated on a five-point Likert-type scale, and the possible responses for the items included “strongly agree,” “agree,” “neutral,” “disagree,” and “strongly disagree,” which were scored 5, 4, 3, 2, and 1, respectively. The revised internal consistency, as measured with Cronbach’s alpha, was α = 0.92, which indicated the questionnaire was reliable.

Table 3.

Satisfaction questionnaire for ILEs.

Item	Description
1	I enjoyed the teaching/learning format used in the lesson
2	The lesson format encouraged active learning on the part of the students
3	I would like to provided sufficient feedback to the instructors during the lesson
4	The instructor often moved through concepts with an appropriate pace as I could understand them
5	A sufficient number of practice and case studies were used to emphasize important concepts
6	I interacted with teachers and others participants more than before
7	The amount of material covered in the lesson was appropriate
8	The before-class reading assignments and discussion were useful for me and I completed them
9	The questions I had about the material were answered instantly during class sessions
10	I was an active participant in the lesson
11	I had no difficulty paying attention in the lesson
12	I felt comfortable and with no fear when asking questions during the lesson
13	I had a good understanding of the learning material as the problems were being solved in the lesson

Unstructured Interviews

Unstructured interviews were applied as a supporting method to identify possible reasons for the differences in the knowledge test, cognitive load, and satisfaction of the learners in the two groups. The interview outline was examined by the Delphi method. The questions of the interview were established by two PhD students in this research field based on current research and the research questions of the study. Then two middle school teachers and two professors who majored in instructional technology applied the Delphi method to give suggestions on the interview questions to ensure the validity of the outline. After revising the questions, three categories of questions were established: (1) learners’ feelings toward the lesson, (2) effective functions and technologies in the ILE and why, and (3) learners’ suggestions for the lesson.

Features of the ILE

The ILE used in this research is a typical ILE that has since then been widely used in China named Changyan Intelligent Classroom. The ILE includes an interactive whiteboard, a touch projector, automatic cameras to record the behaviors of the learners, a sound receiving system, mobile devices, interactive recording and broadcasting systems, a basic network infrastructure, and flexible desks and chairs. This ILE is supported by various technologies, including cloud storage, artificial intelligence in education, student modeling technology, learning analysis technology, affective computing, and sensor networks. This ILE also has all functions described in the literature review section of general ILEs.

Both groups of participants learned in the same classroom. The IRS was used in the experimental group, and every participant had one individual mobile device to view learning materials, submit test answers, ask questions, complete competitions, and have discussions with others. In the control group, the participants did not have individual mobile devices; these students viewed the learning materials and tests through the interactive whiteboard and asked questions by raising their hands. The detailed differences between the two groups are shown in Table 4. The instructional designs of the two groups are also different and will be discussed in the next section.

Table 4.

Differences in available equipment, functions, presentation modes, and interactions between the experimental and control groups.

	Experiment group	Control group
Equipment
Computer	√	√
Projector	√	√
Interactive whiteboard	√	√
Camera	√	√
Mobile device for instructors	√	√
Mobile device for learners	√	×
Movable desks and chairs	√	√
Functions
Behavior recording	√	√
Intelligent management system	√	√
Interactive learning tools	√	×
Learning resource database	√	√
Cloud storage	√	√
Physical environment perception	√	√
GPS	√	√
Student modeling	√	√
Academic achievement recording	√	×
Instructional games	√	√
Formative assessment report	√	√
Give real-time feedback	√	×
Presentation mode
Learning materials	On both mobile devices and whiteboard	On whiteboard
Quizzes	On both mobile devices and whiteboard	On whiteboard
Response to questions	On whiteboard	×
Interactions
Ask questions	By IRS	Hand-raising
Discussion	Online and face to face	Face to face
Comments on others’ work	By IRS	×
Ask questions to students	By IRS	Oral questions
Show works to others	By IRS	To the front
Pre-class discussion	By IRS	×

Instructional Designs of the Lessons

In order to make clear the differences in instructional procedures of the experimental and control group, Tables 5 and 6 have been given. The test questions submitted by the IRS and the application mode are shown in Table 7. The tables show the instructional procedure for the two groups: Table 5 for the experimental group and Table 6 for the control group. They show the instructional activities that were given to the students before, during and after the lesson as well as the instructional procedures that the instructor followed, the activities the students did, the response of the instructor to that, and the technology that was used for that.

Table 5.

Instructional design of the experimental group.

Period	Instructional activities	Instructional procedures	Students’ feedback	Instructor’s feedback	Technology
Before class	Pre-assignment	The instructor uploaded learning materials online for students to study	Found their interests and confusions and discussed them with the instructor and other students online	Discussed with students	Online interactive system
Before class	Pre-test	The instructor submitted pre-test of the learning content to learn about student comprehension	Submitted answers and the results were shown to teachers	Adjusted teaching plan based on the results of the test	IRS and mobile devices
During class	New knowledge teaching: “Leverage in life”	The instructor began to show some leverage in life to students	Asked questions when they were confused without interrupting the class	Answered students’ questions instantly	IRS, interactive whiteboard, and mobile devices
	New knowledge teaching: “The structures of Leverage”	The instructor expressed the structures of leverage to students	Asked questions when they were confused without interrupting the class	Answered students’ questions instantly	IRS, interactive whiteboard, and mobile devices
	Quiz	The instructor submitted the test question about structures of leverage to students	Answered the questions and submitted answers through IRS	Explained students’ confusions based on the accuracy rate of students’ answers	IRS, interactive whiteboard, learning analysis, and mobile devices
	New knowledge teaching: “Types of Leverage”	The instructor introduced and expressed the different types of leverage to students	Asked questions when they were confused without interrupting the class	Answered students’ questions instantly	IRS, interactive whiteboard, and mobile devices
	Quiz	The instructor submitted quiz about the types of leverage to student	Answered the questions and submitted answers through IRS	Explained students’ confusions based on the accuracy rate of students’ answers	IRS, interactive whiteboard, learning analysis, and mobile devices
	Quiz	The instructor submitted the test question about the types of leverage to students, and the system selected one student to answer randomly	The specific student answered the question and other students could give comments by IRS	Explained students’ confusions instantly	IRS, interactive whiteboard, AI, and mobile devices
	Quiz	The instructor submitted the test question about the types of leverage by a game-based method	Answered the questions and the system gave the ranking of the students based on their speed	Explained students’ confusions instantly	IRS, interactive whiteboard, AI, and mobile devices
	Summary	The instructor summarized the concepts	Asked questions when they were confused without interrupting the class	Explained students’ confusions instantly	Interactive whiteboard
	Knowledge application	The instruct submitted one application question to students to finish	Submitted their work online and had a discussion on the work	Gave comments to students’ work	IRS, interactive whiteboard, AI, and mobile devices
After class	Personal Practice	The system submitted different exercises to different students based their performance in class	Finished the exercises online and discussed with the instructor	Explained students’ confusions instantly	IRS, AI, learning analysis, and mobile devices
After class	Summarized assessment	The system gave individual summarized assessment report to students and the instructor	Self-examination	Analyzed the report	AI and learning analysis

Table 6.

The instructional procedures of the control group.

Period	Instructional activities	Instructional procedures	Students’ feedback	Instructor’s feedback	Technology
Before class	Pre-assignment	The instructor uploaded learning materials online for students to study	Self-learning and no instant feedback	No feedback	Online learning system
Before class	Pre-test	The instructor showed pre-test on the interactive whiteboard for students to finish	No feedback	Explained some of the tests based on individual teaching experience	Interactive whiteboard
During class	New knowledge teaching: “Leverage in life”	The instructor began to show some leverage in life to students	No feedback	No feedback	Interactive whiteboard
	New knowledge teaching: “The structures of Leverage”	The instructor expressed the structures of leverage to students	No feedback	No feedback	Interactive whiteboard
	Quiz	The instructor presented the test question about structures of leverage on the interactive whiteboard for students to finish	Gave answers together by speaking out	Gave correct answer and explained the question based on individual teaching experience	Interactive whiteboard
	New knowledge teaching: “Types of Leverage”	The instructor introduced and expressed the different types of leverage to students	No feedback	No feedback	Interactive whiteboard
	Quiz	The instructor presented the test question about the types of leverage on the interactive whiteboard	Gave answers together by speaking out	Gave correct answer and explained the question based on individual teaching experience	Interactive whiteboard
	Quiz	The instructor presented the test question about the types of leverage on the interactive whiteboard	A specific student took the initiative to answer the question	Continued questioning the student	Interactive whiteboard
	Quiz	The instructor presented the test question about the types of leverage on the interactive whiteboard	Gave answers together by speaking out	Gave correct answer and explained the question based on individual teaching experience	Interactive whiteboard
	Summary	The instructor summarized the concepts	Asked questions by hand-raising	Gave comments to the questions	Interactive whiteboard
	Knowledge application	The instruct presented one applying question on the interactive whiteboard and students finished it on their own notebook	A specific student took the initiative to show his/her answer in public	Gave comments to the specific answer	Interactive whiteboard
After class	Practice	The system submitted the same exercises to all the students	Discussed on their confusions with the instructor next class	No instant feedback	Online learning system and mobile device
After class	Summarized assessment	The system gave individual summarized assessment report to students and the instructor	Self-examination	Analyzed the report	AI and learning analysis

Table 7.

Test questions submitted by IRS.

Item	Test content	Categories of the test	Application mode
1	Which of the following item is a leverage?	Single-choice	All students answer
2	Which of the following item is labor saving leverage?	Single-choice	All students answer
3	Which of the following item is laborious lever	Single-choice	Selected one student randomly
4	Which of the following item is equal-arm leverage?	Single-choice	Game-based ranking
5	The person uses one wooden sticks to carry heavy objects on his shoulder, please mark the five elements of leverage in the following pictures	Drawing	All students drawing on the pictures

Experiment Procedure

At the beginning of the experiment, the two classes were randomly assigned to the experimental group or the control group. A brief description of the experiment, including its aims, procedure and privacy protection measures, was given to the participants. The mobile devices were handed out to each participant in the experimental group to be used as IRS receivers to receive real-time feedback and to have interactions with the instructor and other learners. The two groups were taught the same lesson by the same instructor using the different instructional designs described earlier. The experimental group learned about the lever principle via the ILE with the IRS, and the control group learned in the same environment without the IRS. After learning, the knowledge test was immediately administered to the participants for 12 min to measure students’ learning outcomes. The cognitive load test was conducted for 9 min to measure students’ total cognitive load, and the satisfaction survey was carried out for 5 min to investigate students’ satisfaction. Finally, the participants were individually subjected to unstructured interviews to collect supplementary information. The interview lasted 372 min in total. The average interview time for each participant was 12 min (range: 6–19 min). Figure 2 outlines the overall experimental procedure.

Data Analysis

This study applied the grounded theory to analyze the interview text data (Chen, 2015) and it could be divided into three phases: (1) Open coding: In this phase, the researchers identified and named the categories and identified the categorical characteristic of the interview content. (2) Axial coding: In this phase, the researchers linked the various categories together, combined the data and verified the relationships between the categories. (3) Selective coding: In this phase, the core categories were established and systematically linked with other categories, the relationships between categories were verified and the conceptualization of categories that had not been fully developed was completed. The annotation function of WORD 2016 software was used to encode the original text line by line. Each coder carried out open coding twice. The first annotation extracted the main content from the original text data as detailed as possible. In order to ensure the integrity and authenticity of the content, the first annotation was recorded in original words of the interviewees. The second annotation adopted the reply function to reply to the first annotation by content extraction method. The coding of the interview text data was jointly completed by two researchers (Chen, 2015). Category agreement (CA) was used to evaluate the reliability among coders. Category agreement refers to the percentage of the same number of the codes by different coders of the same interview text data in the total number of the codes (Schultheiss & Brunstein, 2001). The result of CA was 0.86 and the total reliability coefficient was 0.92. These results indicated that the reliability of the coders was good.

The Mann–Whitney U test method was used to investigate the differences of the knowledge test scores, cognitive load and satisfaction of the lesson between the two groups in this research. In addition, common-factor analysis of variance and exploratory factor analysis were used to test the commonness of measurement items in the satisfaction questionnaire.

Results

Knowledge test scores

The mean (and SDs) of the knowledge test, strict memorization questions, comprehension questions, cognitive load, and satisfaction are shown in Table 8.

Table 8.

Mean (and SDs) of the knowledge test, strict memorization questions, comprehension questions, cognitive load, and satisfaction.

Measure	Experimental group	Control group
Knowledge test (max. 100 points)	89.38 (9.98)	78.00 (14.24)
Strict memorization questions (max. 50 points)	47.50 (4.47)	46.67 (6.17)
Comprehension questions (max. 50 points)	41.88 (9.11)	31.33 (11.25)
Cognitive load	9.95 (2.95)	12.63 (1.87)
Satisfaction	3.98 (0.60)	3.38 (0.64)

The Mann–Whitney U test results showed a significant difference in knowledge test scores between the experimental and control groups (Z = −2.197, p = .028). The results are shown in Table 9. Students using the IRS in the ILE achieved higher knowledge test scores than students who did not use this system. The Mann–Whitney U test results of the strict memorization questions showed no significant difference between the scores of the two groups (Z = −0.027, p = .836). The results also indicated a significant difference in the comprehension questions between the two groups (Z = −2.621, p = .011) and students using the IRS achieved higher scores than those who did not use the IRS.

Table 9.

Data analysis results of the knowledge test.

Measure	Z	Asymp. Sig. (2-tailed)
Knowledge test (max. 100 points)	−2.197	.028
Strict memorization questions (max. 50 points)	−0.027	.836
Comprehension questions (max. 50 points)	−2.621	.011

^aGroup variable: Group.

NASA-TLX data

The Mann–Whitney U test results showed that the total cognitive load of students in the two groups were significantly different (Z = −2.966, p = .003). Students who learned using the IRS had a lower total cognitive load than those who did not (Table 10).

Table 10.

Data analysis results of the cognitive load.

Measure	Z	Asymp. Sig. (2-tailed)
Cognitive load	−2.966	.003

^aGroup variable: Group.

Satisfaction Questionnaire

The Mann–Whitney U test results showed a significant difference (Z = −2.610, p = .009) in satisfaction between the two groups (Table 11). Students who learned with an IRS were more satisfied with the class than those students who did not learn with the IRS.

Table 11.

Data analysis results of the satisfaction.

Measure	Z	Asymp. Sig. (2-tailed)
Satisfaction	−2.610	.009

^aGroup variable: Group.

Unstructured Interviews

3247 words as text data have been categorized into 32 open codes. Axial coding is a more refined process to merge and classify the open coding based on the similarity of its meaning into new categories. And the axial coding revealed 12 axial codes. Selective coding has a high degree of generality and systematicity to indicate the common characteristics of axial coding and there were six selective codes. The axial coding and selective coding results of the knowledge test, cognitive load, and satisfaction are shown in Table 12.

Table 12.

Coding of the interviews.

Axial coding	Selective coding
Knowledge test factors	Students’ emotion and understanding
	Learning interests and concentration
	Data-based explanations
Comprehension questions factors	Comprehension and application ability
Strict memorization questions factors	No explanation
Cognitive load factors	Construct prior knowledge
	Presentation mode
	Interactions
Satisfaction factors	Promote the interactions
	Instant feedback
	Reduce students’ fear
Suggestions	More game-based instructional format
Suggestions	More time to view the comments in class

Knowledge Test Factors

Codes of the knowledge test factors are shown in Table 12 and the graphical representation of the opinions put forward by the interviewees of the two groups are shown in Figure 3 and Figure 4. The knowledge test factors are summarized through these opinions and revealed three main themes. The first theme concerned students’ emotion and understanding. A number of 11 students who learned with the IRS reported that they could ask the instructor questions without fear whenever they misunderstood or became confused about a topic. By contrast, a number of eight students who learned without the IRS said that they sometimes felt shy or nervous about expressing their thoughts in public. Thus, their questions could not be solved in class. The interviewed students revealed that the comprehension questions were fairly difficult because one question applied several concepts. A number of eight students of the experimental group said they had good comprehension of the learning contents; therefore, they could apply their knowledge well when the test questions were difficult. On the contrary, a number of six students of the control group said it was difficult for them to finish the test questions that needed deep comprehension of the learning content. The second theme concerned learning interest and concentration. A number of 15 students of the experimental group considered the IRS as a popular tool that could promote their learning interest and concentration. As such, a number of 12 students who learned with the IRS indicated that they participated in class actively and could focus on the learning content while nine students who learned without the IRS indicated that they could not participate in class actively or be concentrated in class. The third theme was data-based explanations. A number of 13 participants of the experimental group mentioned that the instructors explained the learning content and test questions with a focused goal based on the data provided by the IRS. Additionally, the typical false answers were presented to them to help them avoid such errors. By contrast, nine participants of the control group indicated that the instructor explained the test questions selectively based on the teaching experience to the class and such a method could not solve their problems and confusions properly.

Figure 2.

Experimental procedure.

Figure 3.

Opinions put forward by the interviewees of the experimental group.

Figure 4.

Opinions put forward by the interviewees of the control group.

Cognitive Load Factors

Codes of the cognitive load factors are shown in Table 12 and involved three themes: Construct prior knowledge: A number of eight students who learned with the IRS said the before-class assignment and discussion online helped them to obtain good prior knowledge on the learning content. Presentation mode: Half of the students found that the mobile device was better able to help them concentrate on the learning content than the interactive whiteboard, especially when students sat at the back of the classroom. Interactions: Students who learned with the IRS indicated that they could interact with other students more which helped them learn more relevant knowledge.

Satisfaction Factors

The participants gave three possible reasons for their satisfaction (Table 12): 14 students in the experimental group said the IRS promoted their interactions with the instructor and other students which made them feel satisfied with the class, while most of the participants in the control group felt the interactions in the class were insufficient. Nine participants in the experimental group mentioned the instant feedback by the IRS made them feel satisfied with the class. Half of the participants in the experimental group also indicated that they could ask questions and give their answers through the IRS without fear and this satisfied them very much.

The students who were interviewed also proposed several suggestions (Table 12) for the application of an IRS in ILEs. First, the students suggested teachers to use the game-based format more often than other learning formats. Second, they needed more time to review comments from the instructor and other students online during class.

Discussion

This research investigated the impact of an IRS in an ILE on students’ academic performance, cognitive load, and learning satisfaction by using comparative quasi-empirical methods. Although the sample size was small, the results seem to reflect the effectiveness of the IRS in ILEs.

Students who learned with an IRS in the ILE were expected to show better academic performance than students who learned without the IRS. The knowledge test was conducted to verify this supposition. Based on the results of the interview, students who learned with the IRS considered they could have a better understanding of the knowledge while the students who learned without the IRS did not think so. This difference can be explained by the fact that the individual questions of each student who learned with the IRS could be solved quickly, so that every student could identify questions and receive explanations in class without delay. This finding is in line with the research of Wu et al.(2012) that instant feedback could promote learning achievement and with the research of Contreras-Castillo et al.(2006) who concluded that instant messaging could be successfully integrated and used online. The results of the interviews and satisfaction questionnaires of the current study revealed that students learning with the IRS participated more actively in class and that the IRS made them feel more relaxed and comfortable because they could ask questions through the system without interrupting the lesson. Students responded to questions via the IRS, thereby reducing their fear and shyness. Numerous studies indicated that students’ learning performance could be improved by increasing levels of active participants (Francisco et al., 1998; Blasco-Arcas et al., 2013). The students also stated that they were interested in the class with IRS because they believed this system could help them better concentrate on the learning content. In addition, the game-based format of lesson offered by the IRS could attract the students. Wu et al. (2010) investigated the relationship between students’ learning interests and their physics learning performance. The result indicated a significant positive correlation between students’ learning interest in physics and their academic performance. These results support the finding of higher test scores in the experimental group compared with those in the control group. In addition, students who learned with the IRS performed better on both comprehension questions and memorization questions. However, differences in the comprehension questions between the two groups were significant while differences in strict memorization questions were not significant. This finding is in line with the research of Efstathiou and Bailey (2012) who found that increasing student participation could help students to understand complex concepts. Moreover, research by Slain et al. (2004) indicated that the IRS that they used in their research had no positive effect on students’ performance in strict memorization questions.

This study assumed that the cognitive load of the students who learned with the IRS was higher than that of the students who learned without the IRS. However, the results showed the opposite. According to the factors influencing cognitive load (Sweller & Chandler, 1991), increasing students’ prior knowledge could reduce students’ cognitive load. The instructional design of the experimental group included a before-class assignment. According to the results of the interviews and satisfaction surveys, students considered the before-class assignment very useful in providing them with prior knowledge of the learning content. Students’ prior knowledge of the learning content is a factor affecting their intrinsic cognitive load. Students with more prior knowledge experience lower intrinsic cognitive load than those with less prior knowledge (Sweller & Chandler, 1991). This result is in line with the lower cognitive load of the experimental group. Students in the experimental group learned the physics lesson and received the corresponding test questions through their mobile devices. Presentation mode was a factor influencing students’ extraneous cognitive load (Sweller & Chandler, 1991). According to the results of the interview, students believed that the presentation mode of the mobile devices helped them concentrate on the learning content and avoid distractions so that their extraneous cognitive load was reduced. Therefore, the total cognitive load of the experimental group could be considered to be lower than that of the control group. One limitation in the measurement of the cognitive load must be highlighted. The cognitive load of the participants was measured by using self-reported measurements. Thus, the results of cognitive load are subjective. In the future, other physiological index measurements can be used to measure students’ cognitive load and provide objective data to enhance the reliability of the results.

This study assumed that the students who learned with an IRS were more satisfied with the class than students who learned without an IRS. The results indicated that the IRS encouraged active learning, provided sufficient feedback to students, helped students speak up without fear, and provided more opportunities for them to have discussions with the instructor and other students. These findings are in line with the research of Slain et al. (2004). Most of the students being interviewed said that they enjoyed the application of the IRS and recommended using this system in all classes to enhance their learning and promote participation. However, the present research did not consider the satisfaction of instructors using the system; as a result, future research should also consider the satisfaction of instructors.

The application of the IRS also solved the problem of the lack of interaction in ILEs. Students who learned with an IRS had more opportunities to discuss the learning content with other students and to ask questions to the instructor. The game-based teaching format helped students to interact more with the technology and to participate more in class. The explanations of the instructor no longer took up most of the time and students had much more time to participate in class.

Conclusion

This research finds that the application of an IRS in an ILE has a positive impact on students’ learning and results in significantly higher academic performance, lower total cognitive load, and higher learning satisfaction compared with those learning in an ILE without an IRS. In addition, the results indicate that the IRS is a popular and effective tool to solve the three problems in the ILEs that were mentioned in the introduction: Firstly, instructors make the class smarter by using the IRS; Secondly, the system provides an enjoyable way to promote student participation, interaction which changes the teacher-centered structure of the class; Thirdly, it can also help students asking questions without fear and receiving feedback immediately, thereby enhancing their comprehension. These findings can help educators to apply IRS properly in ILEs to actively and effectively promote student learning.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Yue Wang

Tessa H.S. Eysink

References

Alassery

(2019). A smart classroom of wireless sensor networks for students time attendance system. IEEE integrated STEM education Conference (ISEC), Princeton, NJ, USA, March, 2019, (pp. 324–331). IEEE.

Blasco-Arcas

Buil

Hernández-Ortega

Sese

F. J.

(2013). Using clickers in class. The role of interactivity, active collaborative learning and engagement in learning performance. Computers & Education, 62(3), 102–110. https://doi.org/10.1016/j.compedu.2012.10.019.

Bonaiuti

Calvani

Piazza

(2015). Increasing classroom engagement and student comprehension through the use of clickers: An Italian secondary school experience. Research on Education and Media, 5(1), 95–108. http://doi.org/10.1187/cbe.13-05-0090.

Cai

Jiao

X. Y.

Yang

Jiang

L. F.

S. Q.

(2021). Practice of multimodality smart classroom under 5G environment. Modern Distance Education Research, 33(5), 10.

Caldwell

J. E.

(2007). Clickers in the large classroom: Current research and best-practice tips. CBE—Life Sciences Education, 6(1), 9–20. http://doi.org/10.1187/cbe.06-12-0205.

Camacho-Miñano

M. D. M.

del Campo

(2016). Useful interactive teaching tool for learning: Clickers in higher education. Interactive Learning Environments, 24(4), 706–723. https://doi.org/10.1080/10494820.2014.917108.

Cao

S. D.

Liu

(2019). A review of research on college English mobile learning in cognitive load theory cognitive. Journal of Heilongjiang College of Education, 38(1), 133–135.

Chen

X. M.

(2015). Critical application of grounded theory in Chinese education research. Peking University Education Review, 13(1), 2188–2115.

Chiu

C. K.

Tseng

J. C.

(2021). A bayesian classification network-based learning status management system in an intelligent classroom. Educational Technology & Society, 24(3), 256–267.

10.

Contreras-Castillo

Pérez-Fragoso

Favela

(2006). Assessing the use of instant messaging in online learning environments. Interactive Learning Environments, 14(3), 205–218. https://doi.org/10.1080/10494820600853876.

11.

Efstathiou

Nikolaos

Bailey

Cara

(2012). Promoting active learning using audience response system in large bioscience classes. Nurse Education Today, 32(1), 91–95. https://doi.org/10.1016/j.nedt.2011.01.017.

12.

Francisco

J. S.

Nicoll

Trautmann

(1998). Integrating multiple teaching methods into a general chemistry classroom. Journal of Chemical Education, 75(2), 210. https://doi.org/10.1021/ed075p210.

13.

Fuad

Deb

Etim

Gloster

(2018). Mobile response system: A novel approach to interactive and hands-on activity in the classroom. Educational Technology Research and Development, 66(2), 493–514. https://doi.org/10.1007/s11423-018-9570-5.

14.

Gao

Huang

Z. Z.

J. H.

Huang

R. H.

(2017). Analysis of cognitive load in smart learning environment. Open Education Research, 23(1), 56–64. http://doi.org/10.13966/j.cnki.kfjyyj.2017.01.007.

15.

Gros

(2016). The design of smart educational environments. Smart Learning Environments, 3(1), 15. https://doi.org/10.1186/s40561-016-0039-x.

16.

Hart

S. G.

Staveland

L. E.

(1988). Development of NASA-TLX (task load index): results of empirical and theoretical research. Advances in Psychology, 52(6), 139–183. https://doi.org/10.1016/S0166-4115(08)62386-9.

17.

Hill

S. G.

Iavecchia

H. P.

Byers

J. C.

Bittner

A. C.

Jr Zaklade

A. L.

Christ

R. E.

(1992). Comparison of four subjective workload rating scales. Human Factors, 34(4), 429–439. https://doi.org/10.1177/001872089203400405.

18.

Huang

R. H.

Yang

J. F.

Y. B.

(2012). From digital to smart: The evolution and trends of learning environment. Open Education Research, 1(1), 75–84.

19.

Huang

X. R.

(2016). The study on pupil's academic engagement and the influence factors in the smart classroom environment. Central China Normal University.

20.

Jiang

L. B.

(2016). The effectiveness on the application of information technology for classroom teaching in primary and secondary schools. Central China Normal University.

21.

Kasuya

(2001). Mann‐Whitney U test when variances are unequal. Animal Behavior, 61(6), 1247–1249. https://doi.org/10.1006/anbe.2001.1691.

22.

Koper

(2014). Conditions for effective smart learning environments. Smart Learning Environments, 1(1), 5. https://doi.org/10.1186/s40561-014-0005-4.

23.

Kuang

S. Y.

Huang

R. M

Shuang

(2018). Analysis and proposals of the status of informatization teaching practice in basic education ——Based on the investigation of video lessons of "Ministry-level excellent course. China Educational Technology, 380(09), 67–74. http://doi.org/10.3969/j.issn.1006-9860.2018.09.009.

24.

Lantz

M. E.

Stawiski

(2014). Effectiveness of clickers: Effect of feedback and the timing of questions on learning. Computers in Human Behavior, 31(1), 280–286. https://doi.org/10.1016/j.chb.2013.10.009.

25.

Leahy

Chandler

Sweller

(2003). When auditory presentations should and should not be a component of multimedia instruction. applied cognitive psychology. The Official Journal of the Society for Applied Research in Memory and Cognition, 17(4), 401–418. https://doi.org/10.1002/acp.877.

26.

Lee

R. S.

(2020). Smart education. Artificial intelligence in daily life (pp. 301–320). Springer. https://doi.org/10.1007/978-981-15-7695-9_11.

27.

Leem

Sung

(2019). Teachers' beliefs and technology acceptance concerning smart mobile devices for SMART education in South Korea. British Journal of Educational Technology, 50(2), 601–613. https://doi.org/10.1111/bjet.12612.

28.

Lang

W. J.

Xue

(2018). Analysis of classroom interaction in smart classroom: A case study of math lesson at primary school. e-Education Research, 39(3), 115–121. http://doi.org/115-121. 10.13811/j.cnki.eer.2018.03.018.

29.

Lin

Xie

Xiao

Deng

Cai

(2019). Adaptive recommender system for an intelligent classroom teaching model. International Journal of Emerging Technologies in Learning, 14(5), 51–63. https://doi.org/10.3991/ijet.v14i05.10251.

30.

Littauer

(1972). Instructional implications of a low-cost electronic student response system. Educational Technology, 12(10), 69–71.

31.

Liu

B. Q.

(2016). Research on design and implementation strategy of smart class teaching in the Ara of “Internet +”. China Educational Technology, (10), 51–56. http://doi.org/10.3969/j.issn.1006-9860.2016.10.009.

32.

Nachar

(2008). The Mann-Whitney U: A test for assessing whether two independent samples come from the same distribution. Tutorials in Quantitative Methods for Psychology, 4(1), 13–20. https://doi.org/10.20982/tqmp.04.1.p013.

33.

Naimi

A. I.

Westreich

D. J.

(2014). Big data: A revolution that will transform how we live, work, and think. American Journal of Epidemiology, 179(9), 1143–1144. https://doi.org/10.1093/aje/kwu085.

34.

Nygren

T. E.

(1991). Psychometric properties of subjective workload measurement techniques: implications for their use in the assessment of perceived mental workload. Human Factors, 33(1), 17–33. https://doi.org/10.1177/001872089103300102.

35.

Oigara

(2015). Using smart response system in elementary common core mathematics Classroom. Society for Information Technology & Teacher Education International Conference, Waynesville, USA, 2015, (pp. 1691–1694). AACE.

36.

Ouf

Ellatif

M. A.

Salama

S. E.

Helmy

(2017). A proposed paradigm for smart learning environment based on semantic web. Computers in Human Behavior, 72, 796–818. https://doi.org/10.1016/j.chb.2016.08.030.

37.

Pirahandeh

Kim

D. H.

(2017). Energy-aware and intelligent storage features for multimedia devices in smart classroom. Multimedia Tools and Applications, 76(1), 1139–1157. https://doi.org/10.1007/s11042-015-3019-1.

38.

Popescu

Ponescu

Roibu

Popescu

L. C.

(2018). Smart classroom-affective computing in present-day classroom. 2018 28th EAEEIE Annual Conference IEEE, Hafnarfjordur, Iceland, 26-28 September, 2018, (pp. 1–9).

39.

Schultheiss

O. C.

Brunstein

J. C.

(2001). Assessment of implicit motives with a research version of the TAT: Picture profiles, gender differences, and relations to other personality measures. Journal of Personality Assessment, 77(1), 71–86. https://doi.org/10.1207/S15327752JPA7701_05.

40.

Shannon

Riecan

Orozova

Sotirova

Atanassov

Krawczak

, et al. (2012). Generalized net model of the process of selection and usage of an intelligent e-learning system. 2012 6th IEEE International Conference Intelligent Systems, Sofia, Bulgaria, 6-8 September, 2012, (pp. 233–236). IEEE

41.

Shieh

R. S.

Chang

(2013). Implementing the interactive response system in a high school physics context: Intervention and reflections. Australasian Journal of Educational Technology, 29(5). https://doi.org/10.14742/ajet.331.

42.

Slain

Abate

Hodges

B. M.

Stamatakis

M. K.

Wolak

(2004). An interactive response system to promote active learning in the doctor of pharmacy curriculum. American Journal of Pharmaceutical Education, 68(5), 1–5. https://doi.org/10.5688/aj6805117.

43.

Sun

C. Y.

Liu

D. Z.

(2013). Comparison of several rating scales of cognitive load. Journal of Phychological Science, 36(01), 194–201.

44.

Sweller

(1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285. https://doi.org/10.1016/0364-0213(88)90023-7.

45.

Sweller

Chandler

(1991). Evidence for cognitive load theory. Cognition and Instruction, 8(4), 351–362. https://doi.org/10.1207/s1532690xci0804_5.

46.

Visvizi

Lytras

M. D.

Daniela

(2018). (Re) defining smart education: Towards dynamic education and information systems for innovation networks. Enhancing knowledge discovery and innovation in the digital era. IGI Global.

47.

J. Y.

X. L.

Peng

J. S.

(2011). Relationship between Self-efficacy and Academic Performance of Physics of Senior Middle School Students. Education and Teaching Research, 25(2), 122–125.

48.

P. H.

Hwang

G. J.

Milrad

H. R.

Huang

Y. M.

(2012). An innovative concept map approach for improving students' learning performance with an instant feedback mechanism. British Journal of Educational Technology, 43(2), 217–232. https://doi.org/10.1111/j.1467-8535.2010.01167.x.

49.

Xiao

Y. M.

(2005). Study on evaluation and its application of mental workers's mental workload. Sichuan University.

50.

Xiao

M. Y.

Fan

G. Q.

Feng

Jiang

H. Y.

(2010). Reliability and validity evaluation of NASA-TLX among primary and middle school teachers. Chinese Journal of Public Health, 26(10), 1254–1255.

51.

Yang

Wang

X. Y.

(2021). Smart classroom helps the four-ring deep interactive teaching. Journal of the Chinese Society of Education, (08), 105.

52.

Zhang

Cui

Liu

H. Q.

, & Wang

Y. N.

(2020). Investigating high school students’ perceptions and presences under VR learning environment. Interactive Learning Environments, 28(5), 635–655. https://doi.org/10.1080/10494820.2019.1709211.

53.

Zhu

Z.-T.

M.-H.

Riezebos

(2016). A research framework of smart education. Smart Learning Environments, 3(1), 1–17. https://doi.org/10.1186/s40561-016-0026-2.