Game-based student response system: Revisiting its potentials and criticalities in large-size classes

Evolution of student response systems: toward a “bring your own device” approach and game-based learning

The interest toward the potential of student response systems to overcome traditional instructional approaches and increase students’ participation goes back to the 1970s. In recent years, these tools have been used in different contexts (Barber and Njus, 2007; Caldwell, 2007; Kay and LeSage, 2009) and have been referred to with different terms including personal response system, audience response system, and electronic voting system, while having similar architecture and affordances (Kay and LeSage, 2009). Instructor–student interaction is difficult to achieve, especially in large-size lectures, and student response systems can facilitate it by automatically collecting students’ responses to a posted question, and immediately tabulating and displaying results in chart form. This allows the instructor to gather information on students’ understanding and therefore regulate their teaching style, while the students can visualize their learning outcomes and discuss misconceptions with the lecturer.

On a technical level, a student response system consists of software to create and manage questions, and a combination of three hardware components: interactive remote controls, a receiver unit, and a classroom computer with a projection system. To give their responses, students use handheld devices called “clickers”. Each clicker unit has a unique ID so that the answer from each student can be identified, recorded, and graded. Student response systems have been increasingly adopted by numerous universities, but costs and maintenance of dedicated hardware probably limited their widespread distribution (Fotaris et al., 2016; Hussein, 2015). In addition, if the class sizes are large, in terms of setting up the class for these, it is just too time-consuming. The Bring Your Own Device (BYOD) wave (Attewell, 2015; Traxler, 2010) disclosed how new opportunities, such as using personal mobile devices, have proved to be more economically sustainable and enjoyable than clickers (Wang, 2015). And, they are far quicker and easier to use, particularly in a big class. A study showed that students prefer to use personal devices because they are already familiar with them and they do not waste time on distribution and collection of remote controls (Katz et al., 2017). Other studies reported students having frequent connectivity problems with clickers, but not being distracted by other functionalities during classroom polling (Stowell, 2015).

Many online applications have been developed to enable student interactions, brainstorming, or making comments through personal devices. Specifically, quiz applications such as Socrative, Poll Everywhere, and Kahoot! replicate the clickers’ functionality, introducing new features like gaming aspects, modalities, and design (Andres et al., 2015). By “gaming aspects”, this means the use of sounds and music playing in the back ground, timed questions and scores for the right answers, and rankings of the players. This could be said to increase the “fun” element, and therefore explain why it is termed a “game” and why they are called game-based student response systems. The beneficial potentials of videogames and game-based learning have received increased attention, and experiments showed positive results, especially because students were more motivated and involved in the learning process (Gee, 2003; Papastergiou, 2009). The main idea, indeed, is that playing games is so engaging that people end up learning without realizing it. For student response systems, the gamified component consists of response timeout, music and other media effects, rankings, and the possibility of working individually or by competing teams: gamification creates an amusing and engaging situation (Bian et al., 2011; Fotaris et al., 2016; Ismail and Mohammad, 2017; Premarathne, 2017; Wang and Lieberoth, 2016; Wang et al., 2016).

Student response systems and game-based student response systems: mode of use and effectiveness

Student response systems and game-based student response systems can be combined with teaching methods like peer instruction, cooperative learning, or flipped classrooms (Barrera Gomez, 2016; Brady et al., 2013; Caldwell, 2007). However, they are used mostly to increase students’ participation and provide formative feedback. Since these systems allow instructors to punctuate the lesson through questions, some studies (Caldwell, 2007; Lantz and Stawiski, 2014; Mayer et al., 2009) reflected on how to best formulate when and with what purposes to administer them. Caldwell (2007) summarized common uses of clicker questions such as starting a discussion, polling student opinion, probing and activating pre-existing knowledge, revealing students’ misunderstanding of lectures, and making formative assessments. But how does asking questions impact student learning? Mayer et al. (2009) found that trying to answer questions and then receiving immediate feedback may encourage the active cognitive process of learning in three ways: (1) before answering questions, students may be more attentive to the lecture material, (2) during question answering, students may work harder to organize and integrate the material, and (3) after receiving feedback, students may develop metacognitive skills to gauge how well they understood the lecture material and to know how to answer examination-like questions.

Although many studies have investigated the impact of student response systems on learning (Bruff, 2014; Caldwell, 2007; Castillo-Manzano et al., 2016; Han, 2014; Han and Finkelstein, 2013; Hunsu et al., 2016; Kay and LeSage, 2009), there are still several open issues related to their educational effectiveness and student engagement, especially for game-based student response systems (Wang, 2015). A systematic review by Kay and LeSage (2009) identified three types of student response system benefits: classroom environment benefits (increased frequency, level of attention, participation and engagement), learning benefits (interaction, discussion, contingent teaching, learning outcomes, and quality of learning), and evaluation benefits (feedback, formative assessment, and possibility of comparing responses with other students). As is well known, instructor–student interaction and feedback are crucial for educational quality, particularly to activate prior knowledge (Hattie, 2009), to reduce cognitive overload (Sweller, 1994), and the risk of misconceptions (Hattie and Yates, 2013). Student response systems and game-based ones provide an opportunity to address student misconceptions as part of formative assessment: giving further explanations and making the students aware of their advances and difficulties, in order to encourage or promote better learning strategies.

Cubric and Jefferies (2015) analyzed students’ perceptions of different modalities of clicker use such as monitoring of attendance, summative assessment, formative assessment, or a combination of these. The results showed that learning benefits were significantly lower when clickers were used only for summative tests, since students felt anxious about the reliability of the technologies used. In addition, student response systems and game-based ones ensure the possibility of having immediate feedback (Lantz and Stawiski, 2014). In the literature, it is underlined that the immediacy of feedback enhances student performance when compared to delayed feedback (Johnson, 2014; Lemley et al., 2007), and diminishes test anxiety (Gilbert et al., 2011; Way, 2012), facilitating students in better understanding their strengths and weaknesses (Wilson et al., 2011).

A meta-analysis of the literature comparing clicker-facilitated to non-clicker classrooms was carried out (Castillo-Manzano et al., 2016; Hunsu et al., 2016). Hunsu et al. (2016) found that the use of student response systems produced significant effects on cognitive outcomes such as knowledge transfer and final achievement, while the measures of retention indicate no significant difference between the experimental and control groups. Furthermore, this meta-analysis demonstrated that the student-response-system effect is minimal when compared to groups using a question-driven pedagogy, similar to the student-response-system approach. As for game-based ones, an interesting line of research focused on their efficacy for formative assessment when compared to other approaches. For example, Wang et al. (2016) compared game-based student response systems, traditional ones and paper-based tests, detecting increased motivation, engagement, enjoyment, and concentration for the gamified approach. On the other hand, the comparison between results of pre- and posttest did not reveal significant improvements in learning outcomes. Han’s (2014) review concluded that although learners’ perceptions were positive both in terms of engagement and learning, the effects on performances were less evident and that further studies were required.

The analysis of the literature has pointed out that most studies have been based on experiments mainly concentrated on small-scale implementation conducted on a limited set of technical and scientific subjects (Castillo-Manzano et al., 2016; Cubric and Jefferies, 2015; Kay and LeSage, 2009). Castillo-Manzano et al. (2016) found different effects depending on the disciplines: their meta-analysis revealed that the use of a student response system improves academic performance, especially with pure soft sciences or applied hard sciences. However, there is a need to look more closely at the effects, if any, of the use of game-based student response systems on students’ learning outcomes because, while these systems might be fun to use, as educators, we are interested in what our students actually learn. The following research questions were identified:

Method

Context and procedure

A quasi-experiment, intra-subject design, involving about 400 students was conducted within the undergraduate course of Educational Technologies (ET2017) at the University of Florence in Italy in 2016–2017. The aim was to use a game-based student response system in addressing the challenge of feedback in a large-size class. A quasi-experimental design encompasses a protocol to administrate an experimental treatment (in this case the adoption of a game-based student response system) in a natural environment like an ongoing class activity. While a “pure” experimental approach requires conditions which might not always be ensured within real educational contexts, an intra-subject pre- and posttest design (Arthur et al., 2012), quasi-experimental, could be a more feasible option in complex educational settings (Cohen et al., 2007; Hammersley, 2012).

The experimental treatment consisted of a number of lessons (three out of eight) where the game-based student response system, specifically Kahoot!, was delivered. Kahoot! allows students to answer through their personal devices. To start, students insert the game pin and choose a personal nickname. Kahoot! allows teachers to create three different games: (1) Quiz, that is, a set of multiple-choice questions with a correct answer, which provides points; (2) Survey, that is, multiple-choice questions aimed at collecting students’ opinions, without assigning points; (3) Discussion, that is, the same as a survey but with a single question. After each question, the app processes the distribution chart of answers; for the quiz typology, the system also provides the scoreboard of students and at the end shows the podium of best players.

Three types of questions were designed:

“Motivational question”—delivered at the beginning of the lesson to mobilize students’ preconceptions and engage them. The game typology is Discussion;

“Monitoring question”—delivered in the middle of the lesson to help students focus on the lesson’s key concepts. The game typology is Quiz, with one single question;

“Reflection questions”—a quiz game with a set of four questions. Delivered at the end of the lesson to stimulate students’ reflection on the overall content and prepare them for further independent study.

The ET2017 course included three main subjects: Theories of educational technologies (T1, four lessons), Social networks for learning (T2, two lessons), and Videogame education (T3, two lessons). The game-based student response system was added as an experimental activity in one of the lessons for each of the three topics mentioned above and consisted of three sessions of Kahoot! (motivational, monitoring, and reflection questions). Figure 1 summarizes the experimental design procedure (lessons with or without Kahoot!), while Table 1 shows the three questions proposed during the first lesson on Educational Technologies.

OPEN IN VIEWER

Figure 1.

Experimental Design Procedure. T1: Theories of Educational Technologies, T2: Social Networks in Education, T3: Videogames in Education. MotQ + F: Motivational question + feedback, MonQ + F: Monitoring question + feedback, RefQ + F: Reflection question + feedback.

OPEN IN VIEWER

Table 1.

Examples of types of questions used during the experimental lesson with Kahoot!

Question type and purpose	Timing	Kahoot typology	Examples
Motivational—raise preconceptions and engage students in the topic	Beginning of the lesson	Discussion	According to you, the use of technologies in teaching positively affects: • involvement and motivation of the students • student participation and interaction with teachers • quantity and quality of learning • no positive influence, indeed worsening
Monitoring—focus on the lesson’s key terms	Middle of the lesson	Quiz, single question	Education technology field deals with: • evaluate and certify the technical effectiveness of electronic tools • design, develop, manage, and evaluate teaching and learning processes • promote the dissemination of technologies in training processes (correct) • teach media and digital skills to new generations
Reflection—trigger reflection on contents and prepare for study	End of the lesson	Quiz, set of four questions True/false	1. For constructivism, technology enhances feedback (false) 2. For cognitivism, technology enhances cognitive processes (true) 3. For behaviorism, technology supports distributed learning (false) 4. For connectivism, technology enables the connection between nodes and information resources (true)

Data collection tools and analysis

The effects of the experimental treatment were investigated through two instruments: pre- and posttests, and a final online survey of students’ opinions about the course with specific questions on the system. The pre- and posttests were made up of 30 items relating to the three subjects that were covered in the lessons. The test items were connected either to the lessons where the system was used or to the traditional lessons, according to the topic. Through this design, it was possible to compare the items linked to the experimental procedure with the items that responded to traditional lecturing. Therefore, despite the intra-subject design, it was possible to make inter-item comparisons to measure the differences between experimental and control conditions in the pre- and posttests.

The final survey focused on five dimensions: students’ profiles, experience with the system, question typology, visualization of results, and the instructor’s further comments on results. Specifically, the first group of questions explored students’ characteristics (age, sex, prior education). The second group focusing on experience with the system gathered information on the forms of participation (using tablets, PCs, or personal smartphones), as well as perception of engagement, anxiety, frustration, and satisfaction with the BYOD approach. The third group related to perceptions of learning enacted with regard to the three types of questions, namely “motivational questions”, “monitoring questions”, and “reflection questions”. The fourth group of questions was devoted to the importance of visualizing the results of the participation in the game, which are the histograms of correct and incorrect responses. The last group of questions aimed at understanding to what extent the instructors’ comments on errors and misleading situations could have a positive effect on students’ perceived learning.

Data were collected immediately through the digital forms as raw datasets that were further polished and rearranged for data-processing and analysis.

Prior to administration and in addition to mapping particular test items to sections of the course, we collected additional validity evidence by conducting reliability analyses. The assessment test, measuring conceptual knowledge, was more difficult to structure. The Split-Half test was adopted being more appropriate for instruments measuring knowledge and learning. Since the initial item-analysis was rather low (>0.40), difficult and easy questions (above the 0.85 and below 0.15 thresholds) were removed. Therefore, the final Split-Half test (with the adjusted Spearman–Brown prophecy formula) on assessment test was 0.61 for the pretest and of 0.69 for the posttest. Moreover, the item-facility was controlled and balanced between the experimental and nonexperimental questions. With regard to the questionnaire adopted for the student-satisfaction survey, the Cronbach’s-Alpha was 0.91, a value normally associated with an instrument’s excellent internal consistency, based on the data collected in our specific case.

The data collected were organized aligning with the research questions. Table 2 shows the variables and their levels characterizing the design, the instruments used, and the inferential test applied after a first exploration of the descriptives.

OPEN IN VIEWER

Table 2.

Research design.

Independent variables	Dependent variables	Instruments for data collection	Statistics
Use of GSRS (presence/absence)	General students’ performance (scores 1/30)	General pre- and posttest	Paired t-test (Shapiro–Wilk test)
Thematic areas (T1, T2, T3)	Students’ performance per topic (scores 0/1)	“Experimental” questions within the pre- and posttest	Wilcoxon signed rank test
GSRS method (the game, BYOD)	Students’ opinions of the method (ordinal scale 1/4)	Final survey	Chi-squared goodness-of-fit Correlational analysis
Type of questions and related feedback (motivational, monitoring, reflection)	Students’ self-perception of achievement through the different forms of feedback (ordinal scale 1/4)

GSRS: game-based student response system; BYOD: Bring Your Own Device.

Overall the sample of 414 participants followed the observed patterns for a degree in Educational Sciences: females prevailing over males (91.1%); mean age of 20 years (with High School studies recently completed), and generally doing their first degree (84.1%).

After removing incomplete records, 399 cases with responses to pre- and posttest were obtained. However, the final dataset was selected according to the criteria of active participation, namely, having participated in more than one Kahoot! per lesson, having made at least 6 out of 9 sessions during the course and having completed the pre- and posttest. These criteria ensured the intra-subject design but provoked a 60% data loss: the convenience sample consisted of 186 cases. The situation could be explained by the irregular presence at face-to-face activities. However, another factor was a problem of correct identification of users, frequently found when using a game-based student response system via BYOD, since identification is accomplished through a nickname. In spite of the clear instructions at the beginning of every game, many students frequently changed their nicknames, making it impossible to trace their results. For the survey, all completed responses were collected (403 of 414).

To find out whether or not the use of a game-based student response system, as a means to provide several types of feedback, improve students’ learning outcomes, as a first step, the overall learning outcomes of the course were assessed. To this end, the pre- and posttest mean scores were compared. Conditions for a parametric test were met (Shapiro–Wilk normality test on residuals of pre- and posttest distributions with W = 0.99 and p value = 0.29; pretest kurtosis = 3.02 and skewness = −0.002; and posttest kurtosis = 2.33 and skewness = −0.11). It must be pointed out that the pre- and the posttest did not require prior study (personal study was controlled as a variable since both pre- and posttests were a “surprise”).

Successively, the test results for each thematic area (T1, T2, T3) were compared along the pre- and posttest mean scores. Within these three thematic groups, the items of lessons where no experimental activity was carried out, and items linked to the experimental condition were compared. Since these grouped subscores generated non-normal, Poisson distributions, the nonparametric Wilcoxon signed-rank test was adopted. A correction of continuity was applied for the paired comparisons (pre- and posttest), and a simple Wilcoxon test (paired = FALSE) was applied in the case of comparisons between items in the final posttest.

To address the question of whether or not the use of a game-based student response system increases students’ engagement and motivation to learn, the survey on students’ experience was analyzed.

Results

The results indicated a significant difference between the pretest mean score (M = 12.48, SD = 2.57) and the posttest score (M = 16.20, SD = 4.32), t(−2.472) = 185, p = 0.001. As for the Effect Size, measuring the dimension of the experimental method’s effect, the value obtained was Cohen’s d = 1.048 (large), which can be interpreted as a high difference between the average score obtained in the pretest and the average score in the posttest. Therefore, these results demonstrated a general improvement in knowledge, developed during the course.

The results showed an increased performance for the “experimental” items between the pre- and the posttest across the three topics: pre–post experimental T1 (V = 851.5, p = 0.001), T2 (V = 467, p = 0.001), and T3 (V = 420, p = 0.001). The effect size reported for the three conditions were: T1 (Cohen’s d = 0.49, small); T2 (Cohen’s d = 0.72, medium); and T3 (Cohen’s d = 0.27, small). This result supported the assumption that the experimental condition was specifically contributing to the general observed improvement. Therefore, we aimed at observing the differences between the contribution of the nonexperimental and the experimental items in the posttest. The results showed significant differences in the case of the T1 (W = 12,484, p = 0.0001), while for T2 and T3 no significant differences of mean scores between experimental and nonexperimental items were obtained (W = 18,214, p = 0.36 for T2, and W = 16,196, p = 0.27 for T3). As for the effect size, the size of the difference between the mean scores in experimental and nonexperimental questions was: T1 (Cohen’s d = 0.33, small); T2 (Cohen’s d = 0.05, negligible); and T3 (Cohen’s d = 0.05, negligible). In any case, it should be considered again that the respondents (subjects) were the same (not a control group) and that the positive effect on experimental topics could have been extended to the other questions within the same topic (by generalization of the learning effect as intervenient variable). In this regard, the overall positive pre–post test results support this assumption.

Students participated mainly through a BYOD approach (an overwhelming 93%, 374/403 participated through smartphone, while the rest used a tablet or PC) and using their own Internet connection (78%) instead of the university WiFi connection (16%) or another public WiFi (6%), X-squared 1128.8, df =4, p =< 0.01. In this regard, students declared having experienced technical problems during the game “always” (26%, 103/403), “frequently” (36%, 145/403), “sometimes” (30%, 122/403), the rest being distributed between “rarely” and “never” (9%, 35/403), which entails a significant difference for the given distribution (X-squared = 182.79, df = 4, p =< 0.01). This shows that the technological infrastructure did not respond effectively to the students’ expectations.

In spite of this situation, the analysis of distributions relating to the interface, the game rules and its dynamics, as well as the class atmosphere, showed highly positive results, balancing the negative issues encountered with technologies. As can be observed in Table 3, the students particularly agreed or completely agreed with the clearness and easiness of the game participation; a relevant number of students also agreed or completely agreed with the engaging game dynamic and good class atmosphere. By contrast, they overwhelmingly disagreed or completely disagreed with the fact that the system produced frustration. Anxiety appeared in some other cases, yet not significantly (completely disagree, 196 and disagree 73 of 403 cases). The BYOD approach, as expected, was irregularly appreciated, but positive perceptions still prevailed (completely agree, 139 and agree 98 of 403).

OPEN IN VIEWER

Table 3.

Students’ survey: dimensions of analysis and results.

	Com. disagree	%	Disagree	%	Neither disagree nor agree	%	Agree	%	Completely agree	%	X-squared N = 403, df = 4	p value >0 .01
1.1. GSRS experience/clear rules	7	1.7	15	3.7	6	1.5	94	23.3	281	69.7	690.14	<0.001***
1.2. GSRS experience/simple interface	5	1.2	11	2.7	12	3.0	64	15.9	311	77.2	851.43	<0.001***
1.3. GSRS experience/engaging dynamic	10	2.5	16	4.0	38	9.4	148	36.7	191	47.4	343.71	<0.001**
1.4. GSRS experience/stimulating class climate	12	3.0	23	5.7	59	14.6	139	34.5	170	42.2	246.81	<0.001*
1.5. GSRS experience/trigger anxiety	196	48.6	73	18.1	45	11.2	58	14.4	31	7.7	218.53	<0.001*
1.6. GSRS experience/trigger frustration	268	66.5	59	14.6	37	9.2	22	5.5	17	4.2	557.88	<0.001***
1.7. GSRS experience/liked BYOD	39	9.7	30	7.4	97	24.1%	98	24.3	139	34.5	102.65	<0.001
2.1. Motivational question/get into the topic	12	3.0	19	4.7	58	14.4	189	46.9	125	31.0	282.05	<0.001*
2.2. Motivational question/deepen on a topic	15	3.7	26	6.5	92	22.8	177	29.0	93	23.1	209.2	<0.001*
2.3. Motivational question/feeling of engagement	9	2.2	21	5.2	39	9.7	161	40.0	173	42.9	315.28	<0.001**
2.4. Monitoring question /focus on a topic	9	2.2	26	6.5	57	14.1	149	37.0	162	40.2	247.76	<0.001*
2.5. Monitoring question/improve attention	14	3.5	20	5.0	60	14.9	162	40.2	147	36.5	242.77	<0.001*
2.6. Reflection question/support to verify own knowledge	12	3.0	21	5.2	39	9.7	152	37.7	179	44.4	307.31	<0.001**
2.7. Reflection question /deepen achieved knowledge	19	4.7	23	5.7%	94	23.3	149	37.0	118	29.3	165.87	<0.001
2.8. Reflection question/enact reflection (metacognition)	11	2.7	21	5.2	65	16.1	163	40.4	143	35.5	239.74	<0.001*
3.1. Visualizing results/reflecting on own difficulties	19	4.7	30	7.4	76	18.9	167	41.4%	111	27.5	183.19	<0.001
3.2. Visualizing results/trigger a feeling of inadequacy	193	47.9%	72	17.9	69	17.1	46	11.4	23	5.7	215.35	<0.001*
3.3. Visualizing results/trigger a feeling of gratification	16	4.0	29	7.2	110	27.3	164	40.7	84	20.8	181.98	<0.001
4.1. Teachers’ comments/Clarifies the elements leading to errors	8	2.0	23	5.7	34	8.4	159	39.5	179	44.4	329.89	<0.001**
4.2. Teachers’ comments/put in relation key concepts	8	2.0	16	4.0	39	9.7	171	42.4	169	41.9	336.99	<0.001**
4.3. Teachers’ comments/support focus for further personal study activities	9	2.2	21	5.2	59	14.6	157	39.0	157	39.0	258.3	<0.001*
4.4. Teachers’ comments/enact peer-learning	19	4.7	28	6.9	95	23.6	159	39.5	102	25.3	165.92	<0.001

GSRS: game-based student response system; BYOD: Bring Your Own Device.

When coming to the question types, we observed that overall the students completely agree or agree with the fact that the system enhanced the learning experience. They mostly felt that the initial question helped them get into the topic and deepen their knowledge of it (see Table 3, 2.1 and 2.2). Students also considered monitoring questions positive to better focus on the topic and improve their attention (see Table 3, 2.4 and 2.5). With regard to the final set of questions, students mainly appreciated how they were supported in verifying their knowledge of the topic (see Table 3, 2.6). They also considered positively, although to a lesser extent, the effects of deepening their achieved knowledge (2.7), and on metacognition as the reflection on their learning process (2.8).

Visualizing was also deemed important, connected to a positive emotional regulation of learning; in fact, most students did not consider it “frustrating” to see their own errors (3.2) but they generally appreciated the feeling of gratification when the answers were correct (3.3). As for the teachers’ comments on the game results, they helped to clarify the elements leading to student errors (4.1) and to put key concepts in relation (4.2). Less clear, but still positive, was the influence of teachers’ comments to further organize independent study (4.3) and to enact peer-learning (4.4).

Discussion and conclusion

Game-based student response systems are suitable tools to engage students in university classes, but their value for educational effectiveness is still to be fully demonstrated (Castillo-Manzano et al., 2016; Han, 2014; Hunsu et al., 2016). Indeed, consistently with previous studies (Brady et al., 2013; Kay and LeSage, 2009), our findings show a general improvement of student performance, but to different degrees and with no significance for certain topics (Castillo-Manzano et al., 2016), while students’ satisfaction and engagement were high. Specifically, looking at the first research question on learning improvement, the comparison between the pre- and posttests indicates that students obtained better learning outcomes. However, focusing on the differences between nonexperimental and experimental items in the posttest, the use of Kahoot! was significant only for topics of a more theoretical nature, while it was less effective for those where they had to apply theory to practice.

Since the teaching style is a relevant factor influencing the quality of the learning process (Hattie, 2009), one might explain these differences referring to the three diverse teaching styles of the teachers involved. In our case, the game-based student response system was not the only factor influencing positive and negative results, excluding the teaching style and students’ personal baselines. In any case, the external, extraneous variables were controlled. Beyond the overall positive results, the specific results related to practical knowledge were always worse no matter who the teacher was. In addition, a common instructional format was shared by the three teachers to ensure a higher level of comparability. Another explanation might refer to the style of questioning adopted through the testing. Regardless of the subject, they aimed at activating, focalizing, and stimulating reflection on conceptual knowledge while knowledge to be applied to the more practical aspects might require different questions shifting the focus from conceptual knowledge to the application of knowledge learned. Since questioning is crucial in the use of such systems (Mayer et al., 2009), a better understanding of how different types of questions may impact the learning of different types of knowledge would help to improve their use and would make their adoption more effective. The observation that different types of questions should have been applied according to the type of knowledge leads us to another consideration. While the three teachers adopted a common instructional format for all topics, leveraging on teacher–student interaction and discussion, the results seem to indicate the limits of the teaching methods employed. In other words, if an interactive lecture addressing hundreds of students may be suitable to improve understanding and conceptual knowledge, when coming to issues around applicability in practice, which require learning-by-doing approaches like workshops, and so on, the limits of large-size classes return to the surface. Such systems may support more interactive lectures intensifying formative feedback and discussion but cannot transform a large-size class into a space for more learning-by-doing activities.

As far as the second research question is concerned, whether the use of game-based student response systems for teaching has a positive influence on students’ perceptions, the results confirm that Kahoot! was highly appreciated. Although technical troubles occurred during the lecture, as in other studies (Wang, 2015; Wang and Lieberoth, 2016), no feeling of frustration or anxiety toward technologies was recorded. Furthermore, students liked the type of questions marking the rhythm of the lesson, although to different degrees. While they fully acknowledged the first and the second questions, respectively, relating to mobilizing prior knowledge and monitoring students’ understanding, they were less enthusiastic about the value of the third question: not all students agreed that this last question facilitated reflection on personal learning and cognitive strategies, that is, processes that are metacognitive in nature.

This brings us to another consideration about what such systems could or could not support. As previously highlighted (Gee, 2003), (good) gaming entails several benefits for learning, including a strong learner commitment to achieving the task, increased interaction, greater agency and control, engagement with problem-solving processes, opportunities for situated learning, immediate feedback on learner performance, and the fact of being pleasantly frustrating (i.e. challenging but doable). Nonetheless, the speed of interaction, which characterizes the mechanisms of gaming, may contrast with the need for time that more reflective processes typically require. Metacognition has less to do with immediate responses and more with overthinking activities which can take place in a span of time. It might be that the pleasant and engaging atmosphere generated by the game through giving limited time to answer the questions, not always positively affected student self-perception of being able to rethink about knowledge learned (Brady et al., 2013).

Interestingly, students expressed appreciation for the possibility of visualizing the results on the screen after having answered, rather than feeling frustrated in the case of incorrect answers, they realized they were not alone in making mistakes (Caldwell, 2007). An understanding of visualized results prevailed as an opportunity to receive feedback on current processes, which is formative feedback (Kay and LeSage, 2009). This is consistent with another of our results referring to the teachers’ comments. They were highly appreciated, suggesting the importance of changing the focus from technological aspects to interaction and teaching dimension, especially looking at formative feedback.

At a methodological level, this study encompasses several limitations that should be carefully taken into account in further research. First, we are dealing with an intra-subject, quasi-experiment. What could be seen as a more ecological approach to educational research (respectful to the educational needs and the institutional activities without imposing rigid experimental conditions that would alter teaching and learning opportunities and conditions), also implies that the type of sample does not allow generalizations to other participants (even of similar characteristics). Second, the problem of identifying subjects (as an intra-subject design requires) with the connected loss of data, game-based student response systems run on an independent platform which ensures easy access; however, the lack of a unified identity (i.e. via login and password) generated the impossibility of checking who was who. This issue provoked a high loss of valid cases (nearly 60%). Third, the assessment test was generally difficult for the students. Fourth, while the survey was well structured and with good reliability, the questions explored complex constructs.

However, one important issue for self-report on learning processes or achievements connected to game-based student response systems would be to explore in-depth (through interviews or focus groups) the specific elements of the experience. Fifth, the technological infrastructures were not stable, generating problems with the WiFi connection and hence with the game dynamics. Other limitations include that the participants were from only one discipline (Educational Sciences), that they were predominantly female and that they were in their first year of study at undergraduate level. They were also all from only one university in one country/culture. As responses might vary according to level (postgraduates, for example, may have different responses), gender, discipline, and country/culture, future work is needed in these areas.

As for future developments, greater attention should be devoted to research about the circumstances under which a game-based student response system produces improved learning performance. In particular, as learning improvement proved to be differentiated between theoretical and topics of a more practical nature, it would be interesting to better investigate the impact of questioning in terms of learning and confirm the hypothesis that different types of knowledge require different types of questions. Furthermore, similar considerations could also be made about the impact of methods: while the questioning instructional format was adopted for all three topics, future research could address the use of game-based student response systems in combination with different teaching methodologies. For example, peer instruction could be used to support learning of topics that are more application based, asking students to discuss in small groups before answering the questions. It would be interesting to make a comparison of the same lesson supported by the use of a game-based student response system but with different methodologies, in order to evaluate the potential of different methods in terms of learning effectiveness. The same experiment could also highlight different outcomes of game-based student response system application in terms of metacognitive process: as said, not all students agreed that questioning facilitated metacognition. Therefore, it would be worth exploring in future research how to foster reflection on personal learning and cognitive strategies going beyond the opportunities offered by visualization and comment on results.