On-Task and Off-Task Behavior in the Classroom: A Study on Mathematics Learning With Educational Video Games

Abstract

When performing a task in the classroom, it is essential to place the focus on learning. In the classroom, it is possible to distinguish between time spent by students on-task and off-task. The former is the time in which the student is focused on the learning task; the latter is the remaining time in which they focus on other activities. Understanding the relationship between the two is a concern for teachers, especially for those who teach mathematics and other subjects that are often considered unattractive by students. Given the opportunity afforded by educational video games to motivate and engage math students, an educational drill-and-practice video game was used in this study to practice second-grade arithmetic and study the students' on-task and off-task behavior. We found that when practicing arithmetic using an educational drill-and-practice video game, time on-task decreases during an activity (30 minutes) as well as over the course of the school year (March to December). This study has implications for the length of mathematics classes at schools as well as the need to vary activities during a class.

Keywords

time on-task time off-task educational video game digital game learning arithmetic

Introduction

Learning in school depends not only on the contents of the curriculum and how relevant they are but also on the ability of the teacher; there are also other conditions which affect student performance. One of the conditions is how focused students are on performing their tasks (Duncan & Magnuson, 2011). Focus is defined in this case as the ability to select and concentrate exclusively on certain information (Cardoso-Leite & Bavelier, 2014). As this ability is limited, the brain develops a mechanism of selective attention in order to separate relevant information from irrelevant information (Cowan, 1988).

At 6 years of age, students are already capable of carrying out tasks independently and with higher levels of concentration. By the time they are 8 or 9 years, they are able to maintain their focus for longer periods (Cardoso-Leite & Bavelier, 2014).

Being able to focus on the learning process is essential. If a student is not focused on the task, the time spent on-task decreases, as does learning (Roberge, Rojas, & Baker 2012). It therefore becomes less likely that the student will develop the necessary skills for acquiring the target knowledge. We can therefore distinguish between two types of behavior in the classroom: on-task and off-task. Time on-task (Romero & Barberà, 2011) is the time spent focused on an activity and is a determining factor in student achievement (Usart, Romero, & Barberà, 2013). Time off-task is the time spent on things other than the learning task (Karweit & Slavin, 1982) and is associated with low academic performance (Rodrigo, Baker, & Rossi, 2013). Off-task behavior can be seen as a regulation device for students, where they take a break from the pedagogical activity (Sabourin, Rowe, Mott, & Lester, 2011). It includes actions such as talking with a classmate (or teacher), playing with other objects, or being disruptive (Allday & Pakurar, 2007). Off-task behavior is considered a significant problem in teaching, as well as a concern for teachers (Rodrigo et al., 2013). Off-task behavior that involves an academic conversation between students may benefit learning, despite interrupting the class (Godwin et al., 2016).

Lim (2015) suggests that incorporating technological innovation into education may improve the quality of the teaching. Given that children find video games attractive (Bassiouni & Hackley, 2016), their use has been introduced into the classroom (Usart et al., 2013) and has provided students with new ways of learning (Schaaf, 2012). Educational video games have become a very popular teaching tool due to their ability to increase student motivation (Sampayo-Vargas, Cope, He, & Byrne, 2013). However, different video games are more or less successful at attracting the students' attention, therefore affecting their motivation (Castell & Jenson, 2004).

There are several types of educational video games. Drill-and-practice is one such type and, based on behaviorist theory of learning (Dede, 2008), is considered one of the more traditional uses of Information and Communication Technology in education (Kuiper & de Pater-Sneep, 2014). However, there is still much debate regarding whether drill-and-practice applications are well-suited to the classroom. This is especially the case when they are compared with more conceptual, exploratory, innovative, and creative activities (Lim, Tang, & Kor, 2012). However, drill-and-practice continues to be a technique commonly used by teachers (Lim et al., 2012), especially in primary education (Kuiper & de Pater-Sneep, 2014) and for teaching mathematics (Lim et al., 2012).

Drill-and-practice applications are generally used for individual practice of basic skills based on the student's abilities. These applications provide immediate feedback (Kuiper & de Pater-Sneep, 2014) and a system for the teacher to monitor student progress (Lim et al., 2012). Furthermore, in order to engage the students, they also include common elements of a video game (Lim et al., 2012). Examples of these elements include graphics, animation, a narrative, and game mechanics. In general, drill-and-practice video games do not require the teachers to have a sophisticated knowledge of computers, and students of all ages can use them with ease (Kuiper & de Pater-Sneep, 2014).

Considering the trend of using digital resources to scaffold learning in the classroom and the importance on focusing on a task while learning, the aim of this research is to study on-task and off-task behavior in the mathematics classroom while using a single drill-and-practice educational video game during class sessions and throughout the school year (March to December). The corresponding research question is this: What changes in on-task behavior can be observed (over time) in a classroom using an educational drill-and-practice video game?

Methodology

Sample and Procedure

A study was conducted in two schools in Santiago, Chile, involving a total of 110 children aged between 8 and 9 from four second-grade classes (Table 1). This grade level was chosen for two main reasons. During this age frame, children acquire enough cognitive resources to spend a relevant amount of time on a single task, thus reducing the interference of limited attention spans in task measurements (Coll, Ullastres, & Palacios, 2001). The second reason is that this is when basic mathematical operations are introduced in the Chilean curriculum (MINEDUC, 2012). As a topic, this is often taught using drill-and-practice applications (Lim et al., 2012).

Table 1.

Class Characteristics.

	School 1 (S1)		School 2 (S2)
Classes	Class A (A1)	Class B (B1)	Class A (A2)	Class B (B2)
Participating students	28	33	23	26
Participating girls	11	21	13	11
Participating boys	17	12	10	15

Table 2.

Features of On-Task and Off-Task Behavior.

No.	On-task behavior	Off-task behavior
1	Commenting on achievements	Resting their head on one or both hands and looking at, or away from, the screen
2	Commenting when they make progress in the game	Looking around the room for something other than the game
3	Commenting on success/failure when completing an exercise	Doodling on a piece of paper/the desk
4	Asking for help from the teacher with a pedagogical task	Playing with other objects (phone, book, toy, etc.)
5	Asking for help from another student with a pedagogical task	Talking about things not related to the activity
6		Disrupting their classmates

Evidence suggests that students' digital skills are clearly influenced by their family's socioeconomic status whether their school is public or private (Román & Murillo, 2013). Thus, we chose two schools with students of different socioeconomic status. The first school (School 1) is a private school with middle/high socioeconomic status and an average monthly family income of over US$2,400. The second school (School 2) is a state-subsidized school with middle socioeconomic status and an average monthly family income between US$570 and US$1,100. The total student enrollment for each of the schools was 71 and 72, respectively, Table 1 only shows the students who took both the pretest and posttest (participating students), with the corresponding breakdown of participating boys and girls.

In order to analyze on-task and off-task behavior, said behaviors were measured on two levels: the micro level, that is, during a class, and the macro level, that is, across a whole school year. In Chile, the first semester of the school year starts in March and ends in July, while the second semester starts in August and ends in December.

Given the schools' availability based on their scheduled activities and public holidays, a total of 14 weekly sessions were conducted, spread evenly across the school year for each class. Each session lasted a maximum of 45 minutes (the duration of a class). Fifteen minutes were used for setting the system up and for the teacher to give basic instructions regarding the content or use of the system. The remaining 30 minutes were focused on the activity itself. Whenever it was deemed necessary, the students were allowed to use a pencil and paper to help them solve the exercises.

The teacher can have a positive or negative impact on a student's learning process. This impact is determined by the teacher's pedagogical and technological skills. Lim (2015) suggests that the use of technology represents an additional role for the teacher. Therefore, the teacher's technological skills become even more important when the teaching process involves technology.

In total, four teachers were involved in this study. In order to minimize the impact of their technological skills, these teachers received a series of four 60-minute practical training sessions. These sessions involved the teachers practicing how to use the video game in a simulated classroom environment. In order to minimize the impact of their pedagogical skills, a member of the research team was always present during the sessions with the video game. The role of this individual was to ensure that all students received the same level of support when faced with pedagogical or technological difficulties. Doing so, limited any bias related to the classroom teacher.

Teaching Tool

The aim of any pedagogical activity is to increase student learning, while considering the pace at which they learn (ChanLin, 2007) and engaging them in the learning process (Reeve, 2013). An educational drill-and-practice video game based on the Interpersonal Computer was used (Beserra, Nussbaum, & Grass, 2017), where each child works at their own pace on a shared screen (Kaplan et al., 2009). The interpersonal computer is a cheap alternative for introducing technology into the classroom (Pawar, Pal, & Toyama, 2006; Trucano, 2010). This is because it only requires one CPU, and the cost of maintenance and support is minimized (Alcoholado et al., 2012). Both technologies differ in the way they provide information. Information is provided privately on a laptop, via a personal display, while it is provided publicly on an interpersonal computer, via a shared display. In terms of learning, when comparing the two technologies using the same software, no significant difference is found (Alcoholado, Diaz, Tagle, Nussbaum, & Infante, 2014). For a detailed description of the game, see the Appendix. It is worth noting that the participating students only practiced arithmetic using the educational video game described in the Appendix.

Assessment

This study adopted both a qualitative and quantitative approach. Galliot and Graham (2016) report that various researchers have highlighted the advantages of adopting a mixed methods approach when it comes to assessment as it provides a broader and more comprehensive view of the research.

Quantitative assessment

To assess student knowledge, an individual assessment of each student was carried out (pretest and posttest) by adapting the instrument used by Beserra, Nussbaum, Oteo and Martin (2014). Before starting the study, this instrument was reviewed and validated by domain experts in terms of its pedagogical content and effectiveness for measuring knowledge acquisition. The group of experts comprised two candidates for a PhD in Education, seven candidates for a Master's degree in Education, and two Graduate-level Professors with expertise in measurement and assessment in education.

The paper-and-pencil test lasted for a maximum of 45 minutes and contained 45 questions. This test looked to identify second-graders' skills in addition, subtraction, and multiplication. These 45 questions included 25 additions, 13 subtractions, and 8 multiplications. The Chilean curriculum for second grade does not include divisions as one of its topics. The questions were ordered by the level of difficulty (from easy to difficult) so as to control for the item position effect and Differential Item Functioning or measurement bias (Thissen, Steinberg, & Wainer, 1993).

Validity and reliability are both fundamental elements of any measurement instrument (Tavakol & Dennick, 2011). Cronbach's alpha was used to guarantee the reliability of this instrument (Tavakol & Dennick, 2011) for each of the participating classes. The results for these using the pretest were all greater than .94 (Table 3). A Cronbach's alpha of more than .6 indicates that the test is an acceptable instrument for classifying students (Bland & Altman, 1997).

Table 3.

Pretest and Posttest Scores.

School	Class	N	t test	Pretest		Posttest		Learning gain	Cronbach's alpha	Cohen's d pre–post
School	Class	N	t test	M	SD	M	SD	Learning gain	Cronbach's alpha	Cohen's d pre–post
S1	A1	28	<.001	25.68	8.07	36.82	7.29	24.76%	0.94	1.48 (huge effect)
S1	B1	33	<.001	23.24	7.99	36.76	6.46	30.04%	0.98	1.89 (huge effect)
S2	A2	23	<.001	14.68	6.71	28.36	7.71	30.40%	0.97	1.94 (huge effect)
S2	B2	26	<.001	11.77	7.97	20.46	4.52	19.31%	0.96	1.37 (very large effect)

Qualitative assessment

Motivation

A questionnaire was used to determine how motivated the students were when playing the video game. This questionnaire included eight yes/no questions and was given to the students at the end of the study (Table 4). Dichotomous variables were used as children at that age struggle to differentiate between values on a multiple-point scale such as the Likert scale (Eiser, Mohay, & Morse, 2000).

Table 4.

Student Motivation.

No.	Questions	School 1 (N = 61)	School 2 (N = 49)
1	Time went quickly when I was playing	70%	76%
2	I forgot that I was in class while playing	72%	71%
3	I stopped hearing sounds from outside the classroom while playing	75%	78%
4	I was not thinking how long was left for recess	72%	86%
5	At the end of the session I wanted to keep playing	85%	86%
6	I enjoyed participating in activities using the game	89%	90%
7	It was fun to play the game every session	95%	96%
8	When I was playing, I only thought about the game	70%	92%

As with the instrument used by Appleton, Christenson, Kim, and Reschly (2006), this questionnaire looks to determine whether the students enjoyed the activity throughout the study. If the students enjoy, they do not feel tired and lose sense of both time (Brockmyer et al., 2009) and space (Table 4, Rows 1, 2, 3, and 8). Furthermore, if the students are motivated they should show a certain lack of interest in having recess (Table 4, Row 4). The questionnaire was also used to ask the students whether they would like to play the video game again at some stage (Table 4, Row 5). This is because the habit of playing the game is reinforced when the students have a pleasant gaming experience (Festl, Scharkow, & Quandt, 2013). Similarly, Przybylski, Rigby, and Ryan (2010) suggest that video games are played when they are considered to be a fun activity. We therefore asked the students whether they thought the game was fun to play, that is, engaging (Table 4, Rows 6 and 7).

The questionnaire was validated before being given to the students. This was done by the aforementioned experts in assessment. These experts were presented with the objectives and research question framing this study, after which they collectively revised and evaluated each item included in the instrument. After applying the revisions suggested by the group of experts, the test was implemented with a sample of 69 students, 11 of which were randomly selected for personal interviews and focus group, conducted by PhD students. The aim of this was to safeguard the use of appropriate vocabulary and question formulation. In addition to the obvious benefits for qualitative assessment (Galliot & Graham, 2016), as the students were with their classmates, the focus group allowed them to feel more at ease when it came to sharing their experiences and answering the interviewer's questions (Nel, Romm, & Tlale, 2015).

The 69 students who participated in the test validation stage were not considered in the experimental sample, although they satisfied the description of a potential participant. Finally, the results revealed a Cronbach's alpha of 0.76, thus allowing us to use the questionnaire in our study.

In-class behavior

An instrument was designed to record and measure the students' on-task and off-task behavior. The aim of using this instrument was to analyze the students' in-class behavior. In order to do so, a set of observation guidelines was developed. These guidelines include 11 actions that demonstrate on-task and off-task behavior (Table 2). An observer's manual provided a definition of each of these actions as well as examples and a description of how to use the guidelines.

On-task behavior (Table 2, Column 1) is divided into two groups. The first of these demonstrates the students' motivation and includes the following actions identified by Baker (2007): Commenting on achievements (Table 2, Row 1), commenting when they make progress in the game (Table 2, Row 2), and commenting on success/failure when completing an exercise (Table 2, Row 3). The second group of actions refers to pedagogical elements of the task and includes the following actions that were identified by Baker, Corbett, Koedinger, and Wagner (2004): asking for help from the teacher (Table 2, Column 1, Row 4) or from another student (Table 2, Row 5).

Off-task behavior (Table 2, Column 2) is also divided into two groups. The first of these is associated with the students' emotional state, such as boredom and frustration. This group includes the following actions identified by Baker, D'Mello, Rodrigo, and Graesser (2010): resting their head on one or both hands and looking at, or away from, the screen (Table 2, Row 1); looking around the room for something other than the game (Table 2, Row 2); and doodling on a piece of paper/the desk (Table 2, Row 3). The second group is related to the disruptive behaviors proposed by Ziemek (2006): playing with other objects (phone, book, toy, etc.; Table 2, Row 4), talking about things not related to the activity (Table 2, Row 5), and disrupting their classmates (Table 2, Row 6). The aforementioned team of experts, who were responsible for creating the motivation questionnaire, also reviewed the observation guidelines (Table 2) so as to corroborate their validity.

The observations were recorded on a Tablet, with a timestamp of each record. A single research assistant was trained to conduct the observations, in order to guarantee the reliability of the gathered data (Inan, Lowther, Ross, & Strahl, 2010) and to minimize the interference of observation in the teaching–learning process. In order to ensure that the observer was following the observation guidelines in every session, the trainer also observed every session. The aim of this was to check whether the data collected by both the observer and the trainer were different at the end of the session. To guarantee consistency, observer and trainer jointly revised the recording every time when the differences emerged.

Throughout each session, the observer watched six students for a period of 45 s. He then recorded the on-/off-task behavior that was observed (approximately 10 s) before observing another group of students. This procedure allowed each group of students to be observed at least 5 times in a session. The observer started each session by observing a different group of students. As the teacher was responsible for answering any questions regarding the content and use of the video game (with help from the person in charge of academic and technological support), the observer was able to concentrate solely on the task of observing. Therefore, the adults present in each session were the teacher, the observer, and the person in charge of academic and technological support.

Results

Learning

Learning gains (pretest and posttest)

The Pretest and Posttest scores are shown in Table 3. Cronbach's alpha for these was greater than .94 for every class. The scores on the pretest have a normal distribution. A series of analyses were conducted in order to confirm the assumption of normality. This included an analysis using QQ norm, histograms, and measurements of central tendency. The average learning gain made by the students in each class was calculated using the difference between their pre- and posttest scores. These can be seen in the column titled Learning Gain. A t test was used to measure statistical significance between the pre- and posttest scores for each class; significant results were obtained for all of the classes. Table 3 also shows the Cohen's d effect size for each class. It is worth noting that the score on the posttest was always higher than the score on the pretest. For no more than three students in each of the four groups, the difference between the pre- and posttest scores was minimal (four points).

With the data of Table 3, two analyses were carried out using the pretest results in order to statistically validate the equivalence between the two classes in each school. The first of these was an analysis of variance (ANOVA), which showed that there were no significant differences between the classes in School 1, F(1, 59) = 1.39; p = .242, or School 2, F(1, 47) = 1.64; p = .206. The second analysis was the Levene's test for equality of variances, which also failed to reveal any significant differences between the classes in School 1, F(1, 59) = 1.04; p = .312, or School 2, F(1, 47) = .001; p = .98.

By taking the results of these two analyses, we can assume that the two classes in each school were equivalent in terms of their prior knowledge. In addition to this, given that class pairs A1 + B1 and A2 + B2 used the same software and hardware, these pairs were grouped together for further analysis in order to minimize the impact of their respective teachers and to control for this variable. A1 and B1 were thus grouped together to represent School 1, with A2 and B2 grouped together to represent School 2.

In order to see whether there were any differences between the students from the two schools at pretest level, we performed an ANOVA, with data of Table 3, which showed that there were significant differences between the School 1 (A1 + B1) and School 2 (A2 + B2), F(1, 108) = 57.53; p < .001. This result is consistent with the national test of mathematics, where in a national average of 259 points, School 1 obtained 332 points, while School 2, 274 points.

A one-way analysis of covariance was carried out in order to statistically validate that there was a difference in learning between School 1 and School 2. This was done by statistically comparing the scores on the pre- and posttest for each school. The results of this analysis revealed a significant difference between School 1 and School 2. The analysis of the difference between the adjusted means (School 1 = 33.33 and School 2 = 28.32) leads us to conclude that School 1 had a greater impact than School 2 on the knowledge acquired by the students.

An ANOVA was conducted to statistically validate that there was significant learning in both schools (A1 + B1 and A2 + B2). This was done by statistically comparing the difference between the pre- and posttest scores. The results of this analysis revealed a significant difference, that is, improvement in learning, for both schools, F(1, 120) = 84.82, p < .001; F(1, 96) = 42.33, p < .001.

Pace of learning

Given that students learned as they played the video game, we studied the learning behavior using the information on student progress that was saved by the software after each session. Figure 1 shows the percentage of students (y axis) from each school that completed a corresponding level of the game (x axis). This analysis only includes students who participated in at least 12 of the 14 sessions so that their progress would be comparable. The video game was designed for teaching math from first to fourth grade, with the first 20 levels associated with the learning objectives for second-grade students in Chile. Figure 1 reveals that some of the students (from School 1) were capable of exceeding these objectives.

Figure 1.

Percentage of students who completed each level of the video game.

Figure 1 also shows that students from School 1 made noticeably more progress than students from School 2. Only approximately 50% of students from School 2 managed to complete 10 levels, while approximately 50% of students from School 1 managed to complete 15 levels. These results are consistent with the leaning gains analyzed in the previous section.

Engagement

Table 4 shows the percentage of positive answers given by students from each group when asked about their level of motivation at the end of the second semester (Columns 3 and 4). Questions 1, 4, and especially 8 reveal that students from the lower income school were more immersed and motivated than students from the higher income school. In addition, the data show that more than 70% of the students reported that their experience playing the video game was positive. We could not delve deeper since student interviews were not allowed by the participating schools. As we will see in the next section, one possible answer for this difference between schools (and possibly within schools) is the difference in access to technology.

On-/Off-task behavior

To analyze the on-/off-task behavior, the 30-minute observations were divided in 10-minute intervals, as this was the longest time span that allowed for trends to be observed in the data (Figure 2(a) and (b)). For example, in the first interval (00:00–09:59), School 1 saw an average of 7 observations of off-task behavior and 20 observations of on-task behaviors for the 14 sessions of the study (Figure 2(a)). School 2, on the other hand, saw an average of 13 observations of off-task behavior and 57 observations of on-task behavior for the same interval (Figure 2(b)).

Figure 2.

Average number of observations of on-task and off-task behavior in School 1 (a) and School 2 (b) during 30-minute sessions.

Figure 2(a) and (b) shows that off-task behavior increases as the activity goes on in both schools. Consequently, on-task behavior decreases, particularly in School 2. The graphs in Figure 2(a) and (b) also show that the increase in off-task behavior is more significant from the first interval (00:00–09:59) to the second interval (10:00–19:59), than from the second interval (10:00–19:59) to the third (20:00–30:00).

When analyzing only the second interval (10:00–19:59), there are a total of 23 observations of off-task behavior and 22 observations of on-task behavior in School 1 (Figure 2(a)). This represents a difference of just a single observation between the two types of behavior. In School 2, the difference between the two is 6 observations (28 observations of off-task behavior vs. 34 of on-task behavior). It can therefore be concluded that at some stage during the second interval, there is a balance between the number of observations of off-task and on-task behavior in both schools.

The third interval (20:00–30:00) also reveals a significant difference between the two types of behavior and reverses the trend seen in the first interval. Both behavioral patterns are similar, although the values in the lower socioeconomic status school (School 2) are higher (Figure 2). This may be explained by the fact that the students progressed at their own pace, but after some time they (possibly) got tired of the activity. The higher values for on-task behavior may (possibly) be attributed to a difference in access to technology, making the activity more engaging to students from the lower socioeconomic status school.

Figure 3(a) and (b) shows the average number of observations of off-task behavior during the first and second semester, respectively, for each school and in 10-minute intervals (x axis). Two observations can be made: (1) throughout the second semester, off-task behavior increases for every time interval in both schools, especially in School 1; (2) the increase in off-task behavior for the second semester is more considerable in School 1, reversing the trend observed in the first semester, where School 2 showed more off-task behavior for each interval. This trend in off-task behavior suggests that the video game loses its attraction more rapidly for students with higher economic status (School 1), which leads to a significant increase in disruptive behavior. The difference in access to technology for students from different socioeconomic statuses may explain the variation in off-task behavior that is observed within a session (Figure 2) and within semesters (Figure 3; Bradley & Corwyn, 2002). In fact, it has been reported that students with higher socioeconomic status in Chile have more access to technology and Internet connection (MINEDUC, 2013).

Figure 3.

Average number of observations of off-task behavior per school in the first semester (a) and the second semester (b).

Discussion and Conclusions

We analyzed on-/off-task behavior of 110 second graders (aged between 8 and 9), who used an educational drill-and-practice video game to study basic arithmetic in weekly sessions a whole academic year. When a video game leads to a state of flow, the student's attention is held (Hamari et al., 2016). However, we observed that off-task behavior increases as both the class and semester goes on, therefore reducing the game's flow. This result suggests that in both semesters, though more so in the second, off-task behavior exceeds on-task behavior after 20 minutes of playing the educational drill-and-practice video game for studying arithmetic. This leads to a decrease in the quality of the learning process for arithmetic, suggesting that the maximum duration of the activity should be (around) 20 minutes. This finding is coherent with those reported by Hamdy and Urich (1998), who compared class periods of 45 and 90 minutes and found that managing student behavior becomes more difficult in longer class sessions, and Godwin et al. (2016), who also found that on-task behavior declines as instructional duration increases from 10 to 30 minutes, and at the end of the school year. Therefore, the quality of the teaching–learning process is affected by the length of the class; correctly estimating time on task is a crucial element in fostering the students' needs in the learning process (Kovanovic, Gašević, Dawson, Joksimovic, & Baker, 2016). Further research is required in order to show that time on-task can be increased by having different games within a session and within semesters.

We also found an increase in off-task behavior from the first to the second semester, which was bigger for the school with a higher socioeconomic status (School 1). This result can be explained by considering the expectations that the students of each school have regarding technology. Still, this increased off-task behavior of School 1 did not impede a greater learning impact over School 2. The socioeconomic difference influences the students' digital skills (Davis-Kean, 2005; Román & Murillo, 2013) as well as their performance on standardized tests (Bellei, 2013), which may explain the change in the knowledge acquired by the students. This is consistent with the observed learning progress through the completion levels (Figure 1). Nevertheless, both schools had a significant increase in learning. It should also be reminded that the students at both schools only practiced arithmetic exclusively using the educational video game.

The significance of this study lies in showing that when an activity is used in class, student attention is affected not only by the length of a lecture but also by the duration of its use (i.e., several semesters). Within the context of our study domain, we build the hypothesis that in order to make the most of the students' time in the classroom, the teacher should frequently change activities. This is especially true for students who are already familiar with such technology.

Finally, it is widely known that in-classroom research requires flexibility from all involved (the school, by adopting the methods and processes, as well as the research team, by adapting to the specific needs and requirements of the school). Lonergan and Cumming (2017) highlight the fact that limitations are part of any in-classroom research and must therefore be taken into consideration. Failing to do so may lead to inconsistent results.

The first limitation of this study comes from the size and representativeness of the student sample; we only analyzed two schools of a different socioeconomic status using just one video game. A representative sample and a broader variety of games would allow generalization. The second limitation arises from the fact that the school principals did not give their permission for the research team to interview and record the students. An interview would have provided further information regarding the students' engagement with the educational video game and their relation with technology in general. Recording the students would also have allowed for subsequent analysis, rather than relying on a single observer splitting short periods of time across multiple groups of students. The third limitation is associated with the lack of a control group. Participating schools requested for all students to be involved in the intervention, considering it lasted a whole academic year. The presence of a control group would have allowed us to analyze how much the educational drill-and-practice video game contributes in terms of motivation, by seeing whether the game reduced off-task behaviors when compared to a traditional classroom.

Three lines of future research are suggested to validate our conclusions. The first would be to study the relationship between on-task and off-task behavior in different subjects, using different teaching dynamics and comparing the results against a control group (without the use of technology). The second of these is a study where one group changes the activity in a session, while the other performs the same activity. This could confirm the hypothesis that shorter activities increase student engagement and therefore learning. The third, could verify the hypothesis that changing activities after sessions increases student engagement and therefore learning. In this case, one group would receive a different activity after a couple of sessions, while the control group would do the same activity for a longer period of time.

Footnotes

Appendix

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was partially funded by FONDECYT / CONICYT 1150045 and Dirección de Investigación, Postgrado y Transferencia Tecnológica at Universidad de Tarapacá [UTA-Educación N#8762-16].

Author Biographies

Vagner Beserra is a professor at the Escuela Universitaria de Educación Virtual of Universidad de Tarapacá, Chile. He is a computer engineer, PhD, at the Pontificia Universidad Católica de Chile. He has a Mathematics degree from the Universidade Estadual Paulista (UNESP/Brasil). He studies the use of video games in the teaching process.

Miguel Nussbaum is a professor at the Computer Science Department of the School of Engineering of Pontificia Universidad Católica de Chile. In 2011, he was honored with the Chilean award for innovation in the Education category, and he has been a member of Chile’s Agency for Quality in Education, since 2012. He studies the use of educational games and multimedia in the classroom and school effectiveness.

Macarena Oteo is currently completing her Master’s degree in Sociology of Childhood and Children’s Rights at the Institute of Education UCL and working at Child to Child’s NGO in the BOND Child Rights Working Group project. She is a Preschool teacher and a Computer Engineer, MA, at Pontificia Universidad Católica in Chile.

References

Alcoholado, C., Nussbaum, M., Tagle, A., Gomez, F., Denardin, F., Susaeta, H., …, Toyama, K. (2012). One mouse per child: Interpersonal computer for individual arithmetic practice. Journal of Computer Assisted Learning, 28(4), 295–309. doi:10.1111/j.1365-2729.2011.00438.x.

Alcoholado, C., Diaz, A., Tagle, A., Nussbaum, M., & Infante, C. (2014). Comparing the use of the interpersonal computer, personal computer and pen-and-paper when solving arithmetic exercises. British Journal of Educational Technology, 47(1), 91–105. doi:10.1111/bjet.12216.

Aleven, V., Myers, E., Easterday, M., & Ogan, A. (2010). Toward a framework for the analysis and design of educational games. 2010 Third IEEE International Conference on Digital Game and Intelligent Toy Enhanced Learning (DIGITEL) (pp. 69–76). Piscataway, NJ: IEEE. doi:10.1109/DIGITEL.2010.55.

Allday

R. A.

Pakurar

(2007) Effects of teacher greetings on student on-task behavior. Journal of Applied Behavior Analysis 40(2): 317–320. doi:10.1901/jaba.2007.86-06.

Appleton

J. J.

Christenson

S. L.

Kim

Reschly

A. L.

(2006) Measuring cognitive and psychological engagement: Validation of the student engagement instrument. Journal of School Psychology 44(5): 427–445. doi:10.1016/j.jsp.2006.04.002.

Baek

Y. K.

(2008) What hinders teachers in using computer and video games in the classroom? Exploring factors inhibiting the uptake of computer and video games. CyberPsychology & Behavior 11(6): 665–671. doi:10.1089/cpb.2008.0127.

Baker, R. S. (2007). Modeling and understanding students' off-task behavior in intelligent tutoring systems. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1059–1068). New York, NY: ACM. doi:10.1145/1240624.1240785.

Baker, R. S., Corbett, A. T., Koedinger, K. R., & Wagner, A. Z. (2004). Off-task behavior in the cognitive tutor classroom: When students game the system. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 383–390). New York, NY: ACM. doi:10.1145/985692.985741.

Baker

R. S.

D'Mello

S. K.

Rodrigo

M. M. T.

Graesser

A. C.

(2010) Better to be frustrated than bored: The incidence, persistence, and impact of learners' cognitive–affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies 68(4): 223–241. doi:10.1016/j.ijhcs.2009.12.003.

10.

Bassiouni

D. H.

Hackley

(2016) Video games and young children's evolving sense of identity: A qualitative study. Young Consumers 17(2): 127–142. doi:10.1108/YC-08-2015-00551.

11.

Bellei

(2013) El estudio de la segregación socioeconómica y académica de la educación chilena [Study of socioeconomic and academic segregation in Chilean education]. Estudios Pedagógicos (Valdivia) 39(1): 325–345. doi:10.4067/S0718-07052013000100019.

12.

Beserra, V., Nussbaum, M., Oteo, M., & Martin, R. (2014). Measuring cognitive load in practicing arithmetic using educational video games on a shared display. Computers in Human Behavior, 41(0), 351–356. doi:10.1016/j.chb.2014.10.016.

13.

Beserra, V., Nussbaum, M., Zeni, R., Rodriguez, W., & Wurman, G. (2014). Practising arithmetic using educational video games with an interpersonal computer. Educational Technology & Society, 17(3), 343–358.

14.

Beserra, V., Nussbaum, M., & Grass, A. (2017). Using a fine-grained multiple-choice response format in educational drill-and-practice video games. Interactive Learning Environments, 25(6), 717–732. doi:10.1080/10494820.2016.1172244.

15.

Bland

J. M.

Altman

D. G.

(1997) Statistics notes: Cronbach's alpha. BMJ 314(7080): 572. doi:10.1136/bmj.314.7080.572.

16.

Bradley

R. H.

Corwyn

R. F.

(2002) Socioeconomic status and child development. Annual Review of Psychology 53: 371–399. doi:10.1146/annurev.psych.53.100901.135233.

17.

Brockmyer

J. H.

Fox

C. M.

Curtiss

K. A.

McBroom

Burkhart

K. M.

Pidruzny

J. N.

(2009) The development of the game engagement questionnaire: A measure of engagement in video game-playing. Journal of Experimental Social Psychology 45(4): 624–634. doi:10.1016/j.jesp.2009.02.016.

18.

Cardoso-Leite

Bavelier

(2014) Video game play, attention, and learning: How to shape the development of attention and influence learning? Current Opinion in Neurology 27(2): 185–191. doi:10.1097/WCO.0000000000000077.

19.

Castell

Jenson

(2004) Paying attention to attention: New economies for learning. Educational Theory 54(4): 381–397. doi:10.1111/j.0013-2004.2004.00026.x.

20.

ChanLin

(2007) Perceived importance and manageability of teachers toward the factors of integrating computer technology into classrooms. Innovations in Education and Teaching International 44(1): 45–55. doi:10.1080/14703290601090390.

21.

Coll

Ullastres

A. M.

Palacios

(2001) Desarrollo psicológico y educación . [Psychological Development and Education] , Madrid, Spain: Alianza Editorial.

22.

Cowan

(1988) Evolving conceptions of memory storage, selective attention, and their mutual constraints within the human information processing system. Psychological Bulletin 104(2): 163–191.

23.

Davis-Kean

P. E.

(2005) The influence of parent education and family income on child achievement: The indirect role of parental expectations and the home environment. Journal of Family Psychology 19(2): 294–304.

24.

Dede

(2008) Theoretical perspectives influencing the use of information technology in teaching and learning. In: Voogt

Knezek

(eds) International Handbook of Information Technology in Primary and Secondary Education, New York, NY: Springer US, pp. 43–62. doi:10.1007/978-0-387-73315-9_3.

25.

Duncan

G. J.

Magnuson

(2011) The nature and impact of early achievement skills, attention skills, and behavior problems. Whither Opportunity 1: 47–69.

26.

Eiser

Mohay

Morse

(2000) The measurement of quality of life in young children. Child: Care, Health and Development 26(5): 401–414. doi:10.1046/j.1365-2214.2000.00154.

27.

Festl

Scharkow

Quandt

(2013) Problematic computer game use among adolescents, younger and older adults. Addiction 108(3): 592–599. doi:10.1111/add.12016.

28.

(2009) EGameFlow: A scale to measure learners' enjoyment of e-learning games. Computers & Education 52(1): 101–112. doi:10.1016/j.compedu.2008.07.004.

29.

Galliot

N. Y.

Graham

L. J.

(2016) Focusing on what counts: using exploratory focus groups to enhance the development of an electronic survey in a mixed-methods research design. The Australian Educational Researcher 43(5): 567–585. doi:10.1007/s13384-016-0216-5.

30.

Godwin

K. E.

Almeda

M. V.

Seltman

Kai

Skerbetz

M. D.

Baker

R. S.

Fisher

A. V.

(2016) Off-task behavior in elementary school children. Learning and Instruction 44: 128–143. doi:10.1016/j.learninstruc.2016.04.003.

31.

Hamari

Shernoff

D. J.

Rowe

Coller

Asbell-Clarke

Edwards

(2016) Challenging games help students learn: An empirical study on engagement, flow and immersion in game-based learning. Computers in Human Behavior 54: 170–179. doi:10.1016/j.chb.2015.07.045.

32.

Hamdy

Urich

(1998) Perceptions of teachers in south Florida toward block scheduling. NASSP Bulletin 82(596): 79–82. doi:10.1177/019263659808259614.

33.

Inan

F. A.

Lowther

D. L.

Ross

S. M.

Strahl

(2010) Pattern of classroom activities during students' use of computers: Relations between instructional strategies and computer applications. Teaching and Teacher Education 26(3): 540–546. doi:10.1016/j.tate.2009.06.017.

34.

Kaplan

DoLenh

Bachour

Kao

G. Y.

Gault

Dillenbourg

(2009) Interpersonal computers for higher education. In: Dillenbourg

Huang

Cherubini

(eds) Interactive Artifacts and Furniture Supporting Collaborative Work and Learning, New York, NY: Springer US, pp. 1–17. doi:10.1007/978-0-387-77234-9_8.

35.

Karweit

Slavin

R. E.

(1982) Time-on-task: Issues of timing, sampling, and definition. Journal of Educational Psychology 74(6): 844–851.

36.

Kovanovic

Gašević

Dawson

Joksimovic

Baker

(2016) Does time-on-task estimation matter? Implications on validity of learning analytics findings. Journal of Learning Analytics 2(3): 81–110. doi:10.18608/jla.2015.23.6.

37.

Kuiper

de Pater-Sneep

(2014) Student perceptions of drill-and-practice mathematics software in primary education. Mathematics Education Research Journal 26(2): 215–236. doi:10.1007/s13394-013-0088-1.

38.

Lim

Tang

Kor

(2012) Drill and practice in learning (and beyond). In: Seel

(ed.) Encyclopedia of the Sciences of Learning, New York, NY: Springer US, pp. 1040–1042. doi:10.1007/978-1-4419-1428-6_706.

39.

Lim

M. H.

(2015) How Singapore teachers in a pioneer “School of the Future” context “deal with” the process of integrating information and communication technology into the school curriculum. The Australian Educational Researcher 42(1): 69–96. doi:10.1007/s13384-014-0153-0.

40.

Lonergan

Cumming

T. M.

(2017) Riding the rapids of classroom-based research. The Australian Educational Researcher 44(2): 141–160. doi:10.1007/s13384-016-0223-6.

41.

MINEDUC. (2012). Ministerio de educación – bases curriculares de educación básica 2012. Retrieved from http://www.curriculumenlineamineduc.cl/605/articles-30013_recurso_14.pdf.

42.

MINEDUC. (2013). Ministerio de educación – informe final – sistema de medición del desarrollo digital de los establecimientos educacionales. Retrieved from http://www.enlaces.cl/tp_enlaces/portales/tpe76eb4809f44/uploadImg/File/2013/doc/censo/Censo_de_Informatica_Educativa.pdf.

43.

Nel

N. M.

Romm

N. R. A.

Tlale

L. D. N.

(2015) Reflections on focus group sessions regarding inclusive education: Reconsidering focus group research possibilities. The Australian Educational Researcher 42(1): 35–53. doi:10.1007/s13384-014-0150-3.

44.

Pawar, U. S., Pal, J., & Toyama, K. (2006). Multiple mice for computers in education in developing countries. 2006 International Conference on Information and Communication Technologies and Development. ICTD ‘06 (pp. 64–71). Piscataway, NJ: IEEE. doi:10.1109/ICTD.2006.301840.

45.

Przybylski

A. K.

Rigby

C. S.

Ryan

R. M.

(2010) A motivational model of video game engagement. Review of General Psychology 14(2): 154–166. doi:10.1037/a0019440.

46.

Qin

Rau

P. P.

Salvendy

(2010) Effects of different scenarios of game difficulty on player immersion. Interacting with Computers 22(3): 230–239. doi:10.1016/j.intcom.2009.12.004.

47.

Reeve

(2013) How students create motivationally supportive learning environments for themselves: The concept of agentic engagement. Journal of Educational Psychology 105(3): 579–595. doi:10.1037/a0032690.

48.

Roberge, D., Rojas, A., & Baker, R. (2012). Does the length of time off-task matter? Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 234–237). New York, NY: ACM. doi:10.1145/2330601.2330657.

49.

Rodrigo

M. M. T.

Baker

R. S. J.

Rossi

(2013) Student off-task behavior in computer-based learning in the Philippines: Comparison to prior research in the USA. Teachers College Record 15: 1–27.

50.

Román

Murillo

F. J.

(2013) Investigation into the effect of school on digital competency: The contribution of the lyceum to the development of ICT in Chilean high school students. In: Lira

C. A.

Gallinato

C. S.

(eds) Estimación del efecto escolar para la competencia digital. Aporte del liceo en el desarrollo de las habilidades TIC en estudiantes de secundaria en Chile [Developing Digital Skills for the Twenty-First Century in Chile: What Does ICT SIMCE Say?], Santiago, Chile: LOM Ediciones, pp. 141–176.

51.

Romero

Barberà

(2011) Quality of e-learners' time and learning performance beyond quantitative time-on-task. The International Review of Research in Open and Distributed Learning 12(5): 125–137.

52.

Sabourin

Rowe

Mott

Lester

(2011) When off-task is on-task: The affective role of off-task behavior in narrative-centered learning environments. Artificial Intelligence in Education 6738: 534–536. doi:10.1007/978-3-642-21869-9_93.

53.

Sampayo-Vargas

Cope

C. J.

Byrne

G. J.

(2013) The effectiveness of adaptive difficulty adjustments on students' motivation and learning in an educational computer game. Computers & Education 69: 452–462. doi:10.1016/j.compedu.2013.07.004.

54.

Schaaf

(2012) Does digital game-based learning improve student time-on-task behavior and engagement in comparison to alternative instructional strategies? Canadian Journal of Action Research 13(1): 50–64.

55.

Sweetser

Johnson

(2004) Player-centered game environments: Assessing player opinions, experiences, and issues. In: Rauterberg

(ed.) Entertainment Computing – ICEC 2004, Eindhoven, the Netherlands: Springer Berlin Heidelberg, pp. 321–332. doi:10.1007/978-3-540-28643-1_40.

56.

Tavakol

Dennick

(2011) Making sense of Cronbach's alpha. International Journal of Medical Education 2: 53–55. doi:10.5116/ijme.4dfb.8dfd.

57.

Thissen

Steinberg

Wainer

(1993) Detection of differential item functioning using the parameters of item response models. In: Holland

P. W.

Wainer

(eds) Differential Item Functioning, Hillsdale, NJ: Lawrence Erlbaum Associates, pp. 67–113.

58.

Trucano, M. (2010). EduTech A World Bank Blog on ICT Use in Education. Retrieved from https://blogs.worldbank.org/edutech/one-mouse-per-child.

59.

Usart

Romero

Barberà

(2013) Measuring students' time perspective and time on task in GBL activities. ELC Research Paper Series 6: 40–51.

60.

Ziemek, T. R. (2006). Two-D or not two-D: Gender implications of visual cognition in electronic games. Proceedings of the 2006 Symposium on Interactive 3D Graphics and Games (pp. 183–190). New York, NY: ACM. doi:10.1145/1111411.1111444.