The Negative Impact of Information and Communication Technologies Overuse on Student Performance: Evidence From OECD Countries

Abstract

The increasing presence of technologies at school has triggered a vivid debate on the way ICT influences students’ learning process. Using PISA 2018 data for 15-year-old students and hierarchical linear models, we find an inverted U-shaped relationship between ICT use at school and students’ performance in mathematics in 22 OECD countries. In all cases, the excessive use of technology is associated with a lower academic performance, although this penalty differs across countries, which points to the importance of addressing country-specific analyses. The differentiated profile of those very intensive users, who suffer from above-average bullying exposure, draws into question whether the effect can be deemed as causal. Based on Inverse Probability Weighting techniques, the findings indicate that the very intensive use of ICT at school causes an underperformance of students equivalent to around half an academic course in Estonia, Finland and Spain. The results highlight the need for policy makers and instructors to ensure that the frequent use of ICT at school does not interfere with students’ learning process.

Keywords

ICT overuse learning performance PISA non-linear causal analysis

The rapid process of digitalisation has permeated and transformed a number of aspects of citizens’ daily live, from social relationships to labour organisation. The last decade has particularly uncovered that adoption of digital skills is paramount in two key ways. Firstly, because it contributes to enhancing citizen participation due to increasing access to information (Polizzi, 2021). Secondly, because it facilitates the process of reskilling or upskilling in a context where demand for new digital skills has risen steadily. In consolidating the process of digital transformation, education and training play a central role. In this paper, we focus on the role of digital technologies to support the learning process of the youth, the engine behind the future of work.

The integration of digital technology to facilitate students’ learning, also known as educational technology, has the potential to create a powerful and engaging environment for collaborative and creative learning (European Commission, 2020; Rubach & Lazarides, 2021). However, in absence of a well-founded pedagogical strategy, the use of digital technology at school risks that individuals lag behind (Comi et al., 2017), a matter at the forefront of the debate during the closure of educational institutions with the advent of the COVID-19 pandemic (Talib et al., 2021). The past two decades have seen firm attempts by policymakers to reduce the so-called “digital gap” (Szeles, 2018), or the unequal access to Information and Communication Technologies (ICT). Over time, the reduction of this gap has been substantial, particularly in technologically and economically advanced societies (Vassilakopoulou & Hustad, 2021).

In this context, one of the key questions now is to what extent the use of ICT—rather than solely the access to them—ultimately impacts on students’ performance. This paper explores the non-linear association between ICT usage at school and student performance in a number of OECD countries and it assesses the causal impact of ICT overuse on student performance. Results from this study contribute to expanding policymakers and educators’ earlier knowledge on the way technology, widely present in the classrooms, influences student performance.

Literature Review and Contribution

Literature on the Linear Relationship between ICT Usage and Student Performance

The existing evidence on the linear effects of ICT usage on student performance fundamentally depends on the nature of the data. Results arising from experimental (or quasi-experimental) studies are mixed, while those based on international survey data, such as PISA, generally point to a negative association between ICT use and student performance (OECD, 2015). Focusing on PISA studies assessing the effects of ICT use at school, Hu et al. (2018) find that a one-score increment in the frequency of use is negatively associated with academic performance on mathematics, science and reading in the 44 countries examined with PISA 2015 data (between 10 and 13 points in the three fields of analysis, which is roughly equivalent to a fourth of a full academic year). These findings are consistent with previous studies which make use of a number of waves of PISA (Zhang & Liu, 2016 find a negative effect of nine points on mathematics and science using 2000–2012 PISA data) or which focus on a specific wave of PISA (Petko et al., 2017 find a negative association between educational use of ICT in the classroom and the PISA results using PISA 2012 data). Other authors (Skryabin et al., 2015) question whether this issue differs by grade, and find a negative impact for secondary school students (between 13 and 15 points for the three PISA areas), but a positive impact for primary school students (between five and seven points depending on the area).

Country-specific literature using PISA data mostly points to a negative association between the educational use of ICT and student performance. For Turkey, the use of computers for educational purposes is found to negatively affect students’ reading performance (Gumus & Atalmis, 2011). This negative association is also found for Spain (Gómez-Fernández & Mediavilla, 2021): using PISA 2015 data, the authors find a negative association between the educational use of ICT at school and at home and the performance in all three areas of assessment. With regard to the school effect, the authors suggest that the lack of preparation of teachers in terms of digital competences may explain part of the result. In a recent paper, Fernández-Gutiérrez et al. (2020) find, also for Spain, that the use of ICT at school in an Autonomous Community does not positively affect performance in mathematics and reading (although it does for science). For Italy, it is found that the usage of at least one digital device has a positive impact on students’ performance in mathematics (Ferraro, 2018) compared to the absence of usage of digital devices, yet the frequency of use is not captured in the model.

Literature on the Non-Linear Relationship Between ICT Use and Student Performance

Previous literature, hence, mostly focuses on analysing the relationship between technology and academic performance in a linear fashion, disregarding the possibility of non-linearity. This may, however, be paramount as the oftentimes found negative relationship might be capturing an average effect that might be overlooking potentially positive effects related to certain degrees of use. The OECD (2015) already suggested that a limited usage of computers in school may trigger better performance than no use at all, but a high use (above the OECD average) could lead to significantly worse academic results.

Some exceptions which have explored the potential existence of non-linearity are identified below. In particular, Woessman and Fuchs (2004), using data from PISA 2000, find an inverted U-shaped relationship between Internet connectivity at school and student mathematics and reading performance. For the specific case of the Netherlands, Gubbels et al. (2020) find an inverted U-shaped relationship between ICT use and reading performance using data from PISA 2015. Focusing on Hong Kong, a recent study (Zhu & Li, 2022) finds that the use of ICT at school is negatively associated with student performance in a linear fashion, while the usage for other purposes (e.g., for leisure or for off-school learning) follows a hill-shaped relationship with student performance. Relatedly, Hu and Yu (2021) assess the relation of ICT use at school for communication (chatting online with other students and using email at school)—among other variables—and student performance on digital reading by analysing whether the effect varies depending on the frequency of use. The results indicate that over the past decade, adolescents' frequent use of ICT-based social media at school, including chatting online and using email at school, have negative effects on digital reading performance compared to the peers who seldom do so. Lastly, Borgonovi and Pokropek (2021) identify an inverted U-shaped association between different forms of ICT use—including use at school—and reading achievement by using PISA data for 2009–2018 for OECD countries.

Literature on the Causal Impact of ICT Use on Student Performance

As outlined above, the literature using large-scale surveys usually establishes a correlation relationship, rather than a cause-effect analysis, given the difficulty to address non-observable features such as student motivation (Fernández-Gutiérrez et al., 2020; Fariña et al., 2015). In broad terms, (quasi-) experimental studies allow for a deeper understanding of a potential causal effect between ICT usage and student performance when compared to the usage of large-scale surveys, while the drawback is that these results are mostly not generalisable as they focus on a very particular context (Fernández-Gutiérrez et al., 2020). In fact, depending on the type of intervention and the context, the impact of ICT on student performance may vary remarkably.

Some interventions have been undertaken through randomized controlled trials (RCT), which are generally regarded as the strongest research design for quantitatively estimating average causal effects (Angrist & Pischke, 2008). A tablet intervention addressed to primary school students in Malawi was assessed by Pitchford (2015) through an RCT. The findings point to a positive impact on the mathematical achievement if the software is carefully designed in terms of content and ability to engage students in the learning process. More recently, Araya and Diaz (2020) carried out an RCT to evaluate the impact of an online math program in Chile and found that the platform had a positive impact on students’ performance, which amounted to almost double the results achieved by using the same exercises on a paper version for a whole year. Lastly, the evaluation of a math homework program in a state of the United States through an RCT yielded a positive impact on students’ mathematics scores compared to existing homework practices (Roschelle et al., 2016).

Because it is not always possible to conduct RCTs (e.g., as they may be financially costly), additional research designs for causal inference have been developed over the past few years, such as instrumental variables, difference in differences, regression discontinuity designs and propensity score designs, known broadly as quasi-experiments (Escueta et al., 2020). As an early example, Angrist and Lavy (2002) adopted an Instrumental Variables approach to assess a policy of installing computers into Israeli primary schools on a wide-spread basis, and they did not find evidence of any relevant effect on students’ test scores. Findings from two regression discontinuity designs (RDD) on subsidized computers for households in the Netherlands and Romania (Leuven et al., 2007; Malamud & Pop-Eleches, 2011, respectively) point to negative impacts on achievement outcomes, likely in part a result of the students spending more time playing games (Escueta et al., 2020).

Focusing on large-scale surveys, the literature measuring the causal impact of the frequency of ICT usage on student performance is scarce, although some other variables, such as ICT investment, have been analysed. For example, Cabras and Tena Horrillo (2016) study the impact of ICT investment on student performance in Spain using PISA 2012 data and applying Bayesian Additive Regression Trees (BART). Results suggest a moderate positive causal effect of ICT investment on student performance. Some additional techniques to overcome the potential endogeneity bias arising from ICT usage have been employed by authors. For example, Agasisti et al. (2020) resort to propensity score matching and Instrumental Variable techniques to examine the effect of ICT use at home and find a negative causal impact on student performance in almost all EU-15 countries.

In broader terms, meta-analyses over the last decade have pointed to moderate effects of technology integration on student achievement, with these effects varying significantly by educational technology type. For instance, Cheung and Slavin (2013) suggest that educational technology applications generally offer modest positive effects compared to traditional methods. Consistent with those findings, results from a meta-analysis and research synthesis in Sung et al. (2016) indicate that the application of mobile devices to education has a moderate mean effect size. A systematic review carried out in Crompton and Burke (2018) shows that, of 23 studies analysed, the use of mobile learning in higher education has a positive impact in 16 (70%) of them. Lastly, some other meta-analyses or systematic reviews highlight the existing variability in the magnitude of the impact across different contexts. For instance, the effect of ICT on student learning outcomes is found to vary across grades and subjects (Bayraktar, 2001; Torgerson & Zhu, 2003).

Rationale for the Present Study

The widespread presence of technology has triggered a vivid debate around its usefulness as a tool to enhance student performance. This debate further gained momentum with the outbreak of the COVID-19 pandemic which followed a forced transition to more technology-based education during the closure of educational institutions and triggered learning losses in a wide range of contexts (see Engzell et al., 2021, for the Netherlands; Maldonado & De Witte, 2022, for Belgium; or Aucejo et al., 2020, for the United States).¹

In general terms, while the evidence on the impact of technology on academic achievement is not conclusive, as shown in the literature section above, this paper attempts to shed light on two fundamental points related to the impact of the frequency of ICT use at school on student performance. First, it was not until recently that the possibility of a non-linear relationship was formally considered (Zhu & Li, 2022). This is, however, paramount to policy makers, as the establishment of a linear relationship might be capturing an average effect that might not be reflective of the actual relationship. This would happen when the positive or negative effect might vary depending on the degree of usage. If this were the case, instructors and policy makers would need to aim for the optimal frequency of usage, which requires to be combined with an appropriate implementation of digital devices at school (OECD, 2015). Earlier literature in this context has been either country-specific (e.g., based on the Netherlands or Hong Kong) or has provided average results for OECD countries in an aggregate manner, and this study aims to expand the geographical scope to test for the possibility of non-linearity in a wider sphere of OECD countries. This is tested in a non-parametric way, contrary to the vast majority of previous research, where non-linearity is gauged by means of quadratic models. Relatedly, we argue that the separate, country-specific analysis allows to gauge potential geographical divergences in the effects of ICT use on student performance, as found in Agasisti et al. (2020).

Second, most of the previous literature focused on examining the correlation between ICT use and student performance. This approach, while informative, risks offering a blurred picture of the real cause of the impact on account of confounding variables (Busenbark et al., 2021). For instance, if the frequency of use of technology were found to be negatively associated with student performance, then it could well be the case that this be caused by other non-observable variables that correlate with frequency of use (e.g., if more frequent users happened to lack motivation to excel academically, and this was the cause of their underperformance, then the estimate would not be reflective of the causal impact). Addressing causality is, hence, paramount in the development of well-founded public policy recommendations (Athey & Imbens, 2017). This study aims to explore the potential existence of a cause-effect relationship between frequency of ICT use at school and student performance. To our knowledge, this is the first time that the causal impact of frequency of ICT usage at school is analysed, especially in the framework of large-scale surveys. The ultimate goal is to contribute to the guidance of educational policy choices in a context where technology is playing an increasingly central role in the learning process of students.

Method

Research Context and Sample

The present study feeds from the PISA 2018 microdata, a programme led by the OECD that measures the ability of 15-year-old students to use their mathematics, science and reading skills to meet real-life challenges. The 2018 edition includes participation of 600,000 students from 79 countries, representing about 32 million students (OECD, 2020a). The assessment comprises a number of questionnaires, addressed to a wide range of stakeholders, namely students, teachers, parents and school managers. The key questionnaire for this study is the ICT familiarity questionnaire, which includes detailed information on students’ use of ICT and their attitude towards it.

The focus of this paper is particularly placed in Estonia (N = 4862), Finland (N = 4898) and Spain (N = 28319), though Appendix D, as shown later, will extend the empirical results to a number of additional countries in order to test for the robustness of the results.² We compare Spain, a relatively low-performing country (OECD, 2019) with limited integration of ICT at school despite its ample ICT infrastructure (Gil-Flores et al., 2017), with the opposite side of the coin: two traditionally top-players in the PISA context and where the education policy has made a firm commitment to integrating ICT into their education system.³ This comparison aims to quantify whether the problems that may be identified are generalisable across countries or, conversely, whether these potential deficiencies do not apply in countries with advanced policies on ICT integration for educational purposes. In total, 22 countries are analysed—including Spain, Finland and Estonia—those for which the questions related to the key variable of interest (i.e., ICT usage in terms of frequency at school) do exist in the database. Those countries are Australia (N =10830), Belgium (N = 6891), Switzerland (N = 5164), Czech Republic (N = 6.181), Denmark (N = 5976), the United Kingdom (N = 6975), Greece (N = 5641), Hungary (N = 4717), Ireland (N = 5049), Island (N = 2675), Italy (N = 9484), Lithuania (N = 5840), Luxembourg (N = 4706), Latvia (N = 4630), Poland (N = 5087), Slovakia (N = 4997), Slovenia (N = 5447) and Sweden (N = 4617).

Variable Description

Dependent variable

As outlined above, PISA measures students’ skills to solve real-life problems in three main areas: mathematics, reading and science. In the present study, the descriptive analysis is presented for these three areas to test whether the observed patterns apply relatively homogeneously. In fact, after confirming that the functional form to relate ICT usage with student performance is comparable across all three knowledge areas, the empirical section specifically focuses on the mathematics field to simplify the analysis. The reason underpinning this choice is that mathematics fosters mental discipline, logical reasoning, mental rigor, and is a paramount element to understanding the content of other fields, such as science. Mathematics is also the engine to STEM-related careers, which are closely related to jobs that will only gain momentum in the future, such as those related to artificial intelligence, machine learning, automation or robotics (Wang & Siau, 2019).

In terms of measurement, the OECD quantifies students’ grades around an OECD average of 500 points and a standard deviation of 100 points.

Control variable of interest

Drawing on the ICT familiarity questionnaire, this study focuses on the module of the questionnaire tackling the frequency of use of ICT by students at school, rather than at home. This choice is central to the interpretation of the results, as there are arguments to consider it as more exogenous than usage in other contexts: there is an external factor (such as the teaching staff or the school’s policy concerning the use of ICT) which, in principle, determines the use of ICT made at school. This would contrast with the choice of the variable of educational use at home, which may suffer from greater selection bias because it could be determined by the student’s own initiative, their socio-economic background or the family environment. Another reason why the analysis focuses on the use of ICT at school is due to its impact on education policy, which is more straightforward to implement as compared to the use of ICT in the private domain.

In order to measure the frequency of use of ICT at school, the questionnaire includes 10 different questions. These reflect the extent to which students use a computer at school to do their schoolwork, use the school’s computers to do group work or communicate with other students, or surf the Internet in connection with class work. The remaining questions are specified in Appendix A. The possible answers that students can provide are the following: “never or hardly ever”, “once or twice a month”, “once or twice a week”, “almost every day”, or “every day”.

To synthesise the frequency of use of ICT at school, we create an index that allows to compare students’ frequency of use of ICT at school. This index is benchmarked at the country level. This intra-rather than inter-country comparison is most suitable in the context of the present study, especially since cross-country comparisons show a blurred relationship between average ICT use and the average score in mathematics, as described in Appendix B. More broadly, another argument to support this intra-country comparison relates to the fact that reported variables have intrinsic limitations that might hinder inter-country comparisons (for instance, certain cultural aspects of countries might lead students to overstate or understate some questions).

The index summarises the use of ICT at school for student $i$ ( $I C T *$ ). It is calculated by obtaining each student’s mean reported frequency ( $I C T$ )—across the 10 questions included in the questionnaire—and normalising it by subtracting country c’s mean use, $\bar{I C T}$ , and dividing it all by the country’s standard deviation $σ_{I C T}$ . This index will therefore have a mean value of zero for each country, and a standard deviation of one.

I C T_{i} * = \frac{I C T_{i} - \bar{I C T_{c}}}{σ_{I C T, c}}

(1)

It is important to note that the OECD already offers an index to synthesise the use of ICT at school by students. The index is centered around an OECD mean of zero and a standard deviation of one, and its construction is based on the Item Response Theory (IRT) (see OECD, 2017 for further methodological details). While this index is useful for inter-country comparisons, it is not fully suitable for our analysis for the abovementioned reasons. To ensure that the index created here is, however, robust to the OECD’s index, we calculated the correlation between the two. In the case of Spain, for instance, the correlation between the ICT index created here and that of the OECD is 0.9406028.

Based on the ICT index (

I C T_{i} *

), five types of users are created, ranging from the very low frequency user to the very intensive one (Table 1). The rationale for the creation of those users is to further explore the possible non-linear relationship between ICT use and academic performance. These users are defined on the basis of the country-specific quintiles of the ICT index created in this analysis. Quintiles are created with the purpose of summarising the average reported frequency of the 10 questions on ICT usage, which makes the interpretation more comprehensive than if a continuous variable (i.e., the mean of a variable that was originally categorical) were to be included. As quintiles rely on the country’s specific distribution, it is worth noting that users might not be directly comparable across countries.

Table 1.

Definition of ICT Users at School.

Country-specific quintiles of frequency of use of ICT at school	Type of ICT user at school
Quintile 1	Very low ICT user
Quintile 2	Low ICT user
Quintile 3	Medium ICT user
Quintile 4	Intensive ICT user
Quintile 5	Very intensive ICT user

Other control variables

As explanatory variables we include a set of student background variables that have been frequently identified as relevant factors in earlier literature (e.g., Gamazo & Martínez-Abad, 2020; Hu et al., 2018). These variables reflect both the student- and school-level characteristics. At the student level, we include discrete indicators of gender, repetition and immigration status, given their relevance in explaining student performance. In particular, repeaters and migrants are frequently found to be negatively correlated with performance, whereas results by gender are mixed depending on the competence under study (Gamazo et al., 2018). Additionally, a binary variable that captures students’ late start in the use of technologies (above 9 years of age) is also covered, a feature that is generally more common amongst students from lower socio-economic backgrounds (Rodrigues & Biagi, 2017). Separately, the PISA index of economic, social and cultural status (ESCS) and a PISA index on the degree of bullying suffered are also included as control variables. These are standarised variables centered around an OECD mean of zero and a standard deviation of one. The ESCS index synthesises students’ responses regarding their family background (e.g., home possession, parents’ occupations and parents’ highest educational level), while the bullying exposure index comprises other questions such as whether other students made fun of the respondent. The literature indicates that the socio-economic status index is positively related to student performance, although this relationship is far from deterministic (OECD, 2020b). The ESCS is one of the most used variables in the PISA literature, as it helps address questions about educational opportunity and inequalities in learning outcomes (Avvisati, 2020). Conversely, the inclusion of the bullying index is often overlooked in the literature, while research highlights its negative impact on student performance (Yu & Zhao, 2021).

At the school level, we include the school size (in logarithmic terms), which the literature suggests to be associated with improved student performance (Giambona & Porcu, 2018). The type of school ownership (public or not) is also included as a control variable. In this case, the literature on its effect on academic achievement offers mixed results (Gamazo & Martínez-Abad, 2020). Lastly, the inclusion of a ratio to measure the number of computers per student (as a continuous variable) serves as a proxy of the school’s available ICT resources per student.

Research Model and Procedure

The methodological framework is divided into two parts. First, it assesses—through hierarchical linear models—whether the hill-shaped relationship between ICT usage and student performance still holds after taking into account other student-specific determinants. The second part focuses on the very intensive ICT user and adopts a complementary technique to establish a causal relationship between the very intensive ICT usage at school and mathematical performance. This is done through a widely applied technique in the causality literature: Inverse Probability Weighting.

Hierarchical linear models

This first part outlines the empirical strategy to assess the relationship between ICT usage and student performance taking into consideration the nested nature of the data. The fact that students are nested within schools implies that multiple regression analysis is not suitable. Instead, the relation is estimated by means of multilevel models, also known as hierarchical linear models (Bryk & Raudenbush, 1992). This is a form of Ordinary Least Squares that analyses the variance in the dependent variable when the predictor variables are at different levels (Woltman et al., 2012).

The rationale for this estimation procedure is described below and is formally specified in equations (2) and (3). The first-level specification gauges the relationship between student performance (student i attending school j) and the p different explanatory variables considered (i.e., the set of independent variables outlined in the “Variable Description” subsection). More specifically, the variables ranging from X1 to X4 are binary variables that denote the type of ICT user each student can be deemed as depending on the level of usage of ICT (based on the country-specific quintiles of ICT usage): low, medium, intensive and very intensive, respectively, and the very low user is taken the as the reference variable. This allows to estimate the relationship between the frequency of ICT usage and mathematics performance when compared to those students who barely ever (or never) make use of it. This is in contrast with most of previous studies that attempt to gauge the non-linear association between ICT use and academic performance, which usually resort to quadratic models, whereas the specification herein used is more flexible by being non-parametric.

The remaining variables entail other features such as the student’s gender or socio-economic status, among others (see “Variable Description” subsection). Lastly, e_ij refers to the residuals. The second-level specification shows that the intercept varies across schools; that is, the overall mean intercept includes a school-specific random-effect term. The reminder $β$ s are constant across schools. The combination of the equations at level 1 and level 2 gives rise to the final specification as shown in equation (3).

Level 1 specification

Y_{i j} = β_{0 j} + β_{1 j} X 1_{i j} + β_{2 j} X 2_{i j} + \dots + β_{p j} * X p_{i j} + e_{i j}

(2)

Level 2 specification

β_{0 j} = γ_{00} + u_{0 j}

β_{1 j} = γ_{10}

β_{2 j} = γ_{20}

\dots

β_{p j} = γ_{p 0}

Mixed model specification

Y_{i j} = (γ_{00} + u_{0 j}) + γ_{10} * X 1_{i j} + γ_{20} * X 2_{i j} + \dots + γ_{p 0} * X p_{i j} + e_{i j}

(3)

In all cases, the 10 plausible values for each student are considered simultaneously, and the 80 weights assigned to each student are taken into account to avoid potential bias in the estimated coefficients (OECD, 2017).

In sum, while this methodology allows to isolate the correlation between ICT usage at school and the academic performance, causality cannot be inferred. To address this, the following subsection outlines the methodology underpinning the causality analysis.

Inverse probability weighting

The second part of the empirical framework focuses on the very intensive ICT user, who is of particular interest on account of the results, which evidence their differentiated socio-demographic profile and their notorious underperformance in mathematics compared to the rest of users. Those results, both at the descriptive and at the empirical levels (through hierarchical linear models), cannot be deemed as causal, which is to be analysed in this second part of the analysis.

The causality analysis attempts to identify whether the variable of interest (very intensive ICT usage in this case) is actually causing the outcome variable (student performance) to decrease. For example, if some non-observable variables shared across very intensive ICT users were determining the low mathematics performance, then these variables—rather than very intensive ICT use—would be the cause of a low performance. The fundamental rationale of the causality analysis is to ideally compare a situation where an individual uses technology very intensively with a situation where that same individual hardly uses it at all. If the comparison were to lead to a significant gap in mathematical performance in favour of the non-intensive user, it could be concluded that the very intensive use of ICT is the cause of poor mathematical skills acquisition. However, since in reality this comparison is not feasible for the same individual, there are a number of econometric techniques that offer an approach to address this issue.

In this paper, the Inverse Probability Weighting (IPW) method is applied. This methodology is based on the idea that random assignment ensures that the distribution of variables among treated and control individuals is probabilistically equivalent. Nevertheless, when the assignment is not random (and this is the case for being a very intensive ICT user), some students have higher probability of being treated, depending on their characteristics. In order to obtain a pseudo-random sample that guarantees that the distribution of covariates would be probabilistically equivalent, we weight students by the inverse probability of being very intensive users (De la Rica and Gorjón, 2019). The aim of this estimation method is, in turn, to approximate the distribution of the observable variables of the treatment group (very intensive users) and of the control group (the rest of the students), assuming that in this way the distribution of the non-observable variables would also be assimilated (see Wooldridge, 2002 and 2010 for a detailed explanation of this methodology).

The estimation method is based on the following procedure. Firstly, a logit model is defined to estimate the probability that student i is a very intensive ICT user ( $\Pr (V e r y I n t e n s i v e_{i})$ ) based on a number of explanatory variables reflected in vector X in equation (4), and include gender, socio-economic level, repetition and immigration status, late introduction to ICT, school size, computer/student ratio, bullying rate, and school ownership (public or not). In addition, the final student weights ( $s w$ ) are also included as established in the framework proposed by DuGoff et al. (2014), who state that the logit model should not be weighted by sample weights, but that these should be included as an additional variable in the model.

\Pr (V e r y I n t e n s i v e_{i}) = f (X, s w_{i})

(4)

Once the model is estimated, the probability of being a very intensive user is predicted ( $p_{i}$ ). These predictions are used to create inverse probability weights ( $w_{i}$ ) in the following way

w_{i} = 1 / p_{i}, i f V e r y I n t e n s i v e_{i} = 1

(5)

w_{i} = 1 / (1 - p_{i}), i f V e r y I n t e n s i v e_{i} = 0

(6)

These weights enable the over-representation of those individuals who, given their characteristics, are likely to be very intensive users but do not report being so on the basis of the ICT questionnaire. On the contrary, if the student’s characteristics lead to a prediction of low probability of being a very intensive user and the student does not report to be one, the weight to be applied to that student will be close to one. Similarly, if the model predicts a high probability of being a very intensive user and this is indeed the case, the weights assigned will also be close to one. Finally, when the user is indeed very intensive but her/his characteristics predict a low probability of being so, this person will also be over-represented. Through the approximation of observable variables between the control and treatment groups, it is assumed that this approximation is also assimilated in the unobservable variables.

The estimation of the model through IPW allows to obtain the Average Treatment Effects (ATE), which measures the potential causal impact of the very intensive usage of ICT on student performance. The ATE requires that the whole population under study is eligible to be treated, given that it compares the whole population were it treated versus were it not treated. To ensure that this is the case in the present paper, we will analyse the distribution of the propensity score (i.e., the predicted probability of being a very intensive ICT user) between the treatment and control groups (“Results” section). If the distribution is comparable, then the estimation of ATE is well founded, as long as extreme values are not present in the distribution (Cunningham, 2021). In fact, the presence of extreme values could bias the estimator and induce excessive variance, given that the weights attained through the IPW methodology (see equations (5) and (6)) could become overly large and could hence give rise to unstable estimates (Avagyan & Vansteelandt, 2018).

Following the approach proposed by DuGoff et al. (2014), the final weights applied to the model are the product of the sample weights and the IPW weights, calculated as detailed in (4), (5) and (6) above. With these final weights, the average impact in mathematics between the very intensive user and the remainder of the users is estimated, in order to capture whether the existing mathematical gap changes when these weights are applied.

Data Analysis

The descriptive results show, in first place, the mean use of ICT at school for each of the five types of users herein defined and for the three countries. This allows to infer to what extent a specific ICT user is indeed comparable across the three countries.

Figure 1 shows how the average frequency of ICT use at school varies by type of ICT user in all three countries. For each type of user, the average frequency of ICT use at school is similar in Spain and Estonia, and lower than in Finland. Nevertheless, these differences are relatively small. A clear pattern that emerges is the jump in terms of the frequency reported by the very intensive user in the three countries. While the difference between the four reminder users is relatively stable, the very intensive user reports significantly higher frequency, with the use being close “almost every day”, especially for Estonia and Spain. The average frequency reported for each of the 10 questions, by country and ICT user type, is provided in Appendix C.

Figure 1.

Average frequency of use of ICT at school in Spain, Estonia and Finland.

After identifying the actual average use of ICT at school for each type of user, Figure 2 shows the average score of each user in the three main areas of PISA. The results confirm that, in the three countries, the relationship between frequency of ICT use at school and the performance in mathematics, science and reading follows an inverted U shape, where the highest frequency user group (i.e., very intensive users) obtain a significantly lower average grade than the remainder of users. In Spain and Estonia, the maximum peak score in the three knowledge areas is obtained by the medium user (quintile 3, i.e., those who use ICT at school more than 1–2 times per month). In Finland, low users (those that use the ICT 1–2 times per month) and medium users (less than once a week, approximately)—and intensive users, although slightly less so—are the ones who show the strongest mathematical competences, in contrast to the scientific and reading competences, where it is the low intensity user (quintile 2) who obtains the best average competences. Given the fact that the inverted U-shaped relationship holds for the three knowledge areas (mathematics, science and reading), the empirical analysis will focus on the particular case of mathematics.

Figure 2.

Average score in Spain, Finland and Estonia by frequency of use of ICT at school.

Lastly, this section presents a descriptive analysis of the characteristics of the students depending on their frequency of use. Behind each ICT user type, there might be certain socio-demographic profiles that make users perform differently. Table 2 aims to compare these features to examine whether patterns emerge depending on the frequency of ICT use.

Table 2.

Descriptive Statistics by Type of ICT User.

	Very low	Low	Medium	Intensive	Very intensive
% Female
Estonia	50.5%	58.3%	57.3%	50.2%	40.2%
Finland	47.3%	60.2%	59.5%	52.0%	33.5%
Spain	47.9%	56.1%	56.4%	50.4%	38.0%
% Immigrant
Estonia	1.3%	0.9%	0.7%	1.1%	3.0%
Finland	3.1%	2.5%	2.9%	4.1%	5.0%
Spain	8.7%	7.6%	7.6%	8.3%	10.3%
% Repeater
Estonia	3.0%	1.8%	2.2%	2.1%	2.7%
Finland	2.4%	2.4%	2.5%	2.7%	3.7%
Spain	26.7%	21.8%	18.9%	20.8%	28.2%
% Public school
Estonia	95.9%	95.8%	97.0%	96.9%	95.9%
Finland	96.9%	96.5%	94.4%	94.7%	94.9%
Spain	66.4%	65.5%	63.0%	58.6%	54.5%
ESCS
Estonia	−0.0031	0.0493	0.1397	0.1439	0.0878
Finland	0.2076	0.2647	0.3771	0.4263	0.3678
Spain	−0.1819	−0.0904	−0.0899	−0.0455	−0.0597
Bullying index
Estonia	−0.0681	0.0203	0.0301	0.0947	0.3262
Finland	−0.0899	−0.1258	0.0033	−0.0118	0.0536
Spain	−0.2990	−0.2767	−0.2974	−0.1691	0.0315

By gender, male students are more numerous in the extremes of ICT usage: they dominate the groups of analogous users and, more notably, they conform a majority within the group of very intensive ICT users. Immigrants are overrepresented in the group of very intensive users, while they are relatively evenly distributed for the reminder of users. Some other differences can also be observed by country: in Spain, there is a substantial overrepresentation of repeaters within the group of very intensive ICT users. This is also the case in Finland, although the share of repeaters in the country is far lower than in Spain. Additionally, the more frequently students use ICT at school, the higher the proportion of those attending non-public schools is in the Spanish case. This is despite the fact that performance of Spanish students is typically better in private schools than in public ones (Vega-Bayo & Mariel, 2018), as also found later in the empirical results section (Table 3). In Estonia and Spain, there is a positive correlation between ICT usage and the socio-economic profile of students. In those two countries, very intensive users have the largest average ESCS of all the users herein defined. Finally, there is a clear pattern between ICT usage and exposure to bullying. In Spain and Estonia, the average exposure to bullying increases as the frequency of use of ICT at school increases. In fact, analogous users in both countries report, on average, lower exposure to bullying than the average of the OECD, whereas the exposure to bullying for very intensive users is far higher than the country’s averages in both cases. Very intensive users, in turn, appear as being much more prone to bullying exposure than the rest of users in those countries, and this applies to the three countries under study.

Table 3.

Estimated Association Between the Mathematics Achievement and the ICT Usage and Other Covariates.

	Spain	Estonia	Finland
Low ICT user	10.21***	6.030	8.759***
Low ICT user	(2.865)	(3.749)	(3.147)
Medium ICT user	10.03***	11.41***	4.503
Medium ICT user	(3.050)	(4.061)	(3.639)
Intensive ICT user	−2.963	−4.206	0.245
Intensive ICT user	(2.965)	(4.537)	(4.009)
Very intensive ICT user	−22.45***	−24.65***	−32.37***
Very intensive ICT user	(3.447)	(4.317)	(3.642)
ESCS (socio-ec. Index)	10.36***	19.01***	29.76***
ESCS (socio-ec. Index)	(0.944)	(1.995)	(1.818)
Immigrant	−19.31***	−25.43**	−25.96***
Immigrant	(3.663)	(12.95)	(7.091)
Repeater	−86.20***	−48.51***	−59.99***
Repeater	(2.853)	(10.47)	(8.595)
Female	−17.57***	−12.30***	−3.310
Female	(2.313)	(2.832)	(2.817)
Public school	−7.017***	−17.66***	−17.44***
Public school	(1.698)	(4.439)	(2.750)
Number of students at school (log)	3.661***	4.349***	2.358
Number of students at school (log)	(0.774)	(1.552)	(1.515)
Computer-student ratio	−0.708	4.936***	2.263**
Computer-student ratio	(0.683)	(1.742)	(1.085)
Late ICT users (>9 years old)	−21.19***	−26.93***	−28.40***
Late ICT users (>9 years old)	(1.952)	(4.291)	(5.080)
Bullying (index)	−4.015***	0.264	1.164
Bullying (index)	(1.322)	(1.679)	(1.324)
Intercept	512.7***	521.3***	512.9***
Intercept	(6.138)	(11.72)	(10.98)

Note. Standard errors in parentheses: ***p < 0.01, **p < 0.05, *p < 0.1. The reference for the different types of ICT users refers to the category of the very low ICT user.

It is important to note that the results presented in this section are merely descriptive. They do not imply that the hill-shaped relationship is necessarily attributed to the frequency of ICT usage, as there might be other variables beyond the ICT usage that are driving the effect. This might be particularly the case for very intensive ICT users, who have a very differentiated socio-economic profile (Table 2). The following section will attempt to infer whether this relationship remains once students’ personal characteristics are taken into account.

Results

This section presents the empirical results and is divided in two parts. The first one shows the estimated relationship between each type of ICT user and their mathematical performance to identify whether the hill-shaped relationship found in the descriptive analysis holds when considering students’ characteristics. Results are then disaggregated by gender, ESCS and ICT-related activities. The methodology underpinning these results is based on Hierarchical Linear, or Multi-Level, Models. The second part of this section presents the causal estimates for the very intensive ICT user by applying the IPW framework.

Results of the Multi-Level Analysis

The results derived from the hierarchical linear model allow to compare the over- or under-performance of low, medium, intensive or very intensive ICT users when compared to very low users, taking into account their socio-demographic features.

Main Results

The estimated coefficients of the hierarchical linear model are summarised in Table 3 for the three selected countries and expanded to the 22 OECD countries herein considered in Appendix D. On the one hand, the results show that, for practically all the countries under study (including those in the appendix), the low ICT user status (quintile 2, i.e., average ICT usage slightly below once a month for Spain and Finland; and once or twice a month in Estonia) is associated with better results than the very low user status (quintile 1). The medium user (quintile 3, i.e., those with an average use between 1 and 2 times per month and 1–2 times per week in Spain and Estonia, and 1–2 times per month in Finland) also tends to be related with more positive results than the very low user, although this variable is not significant for an important part of the countries. For Spain and Estonia, on the other hand, positive and significant effects of 10 and 11 points, respectively, are found in relation to the less frequent user.

On the other hand, for the intensive user of ICT at school, a clearly negative trend is observed in most of the countries analysed. However, the coefficient associated with this variable is not significant in Spain, Finland and Estonia, among others. The strong and very significant impact in all the countries lies on the very intensive users, that is, those who use ICT almost every day. In this group of very intensive users of ICT at school (last quintile), a unanimous pattern is observed in all the countries examined: compared to very low frequency users and, broadly, to the reminder of users, very intensive users score significantly lower in mathematics. In order to interpret such results, it is important to recall that a difference of 40 points is roughly equivalent to a full academic year. This means that very intensive users in Spain or Estonia underperform by more than half year compared to non-ICT users, and by three-quarters of year when compared to low or medium ICT users. In the case of Finland, very intensive ICT users perform a full academic year worse than their low ICT user counterparts.

To expand the analysis, equation (3) is estimated separately by gender and socio-economic status (higher or lower than the median). The results, shown in Table E1 of Appendix E, confirm that the existence of the inverted U-shaped relationship between ICT usage at school and students’ performance in mathematics still holds for all the four groups herein considered.

As a robustness exercise, Appendix F estimates the results for the 22 OECD countries by using the ICT frequency index as a continuous variable—as opposed to the user-specific dummies—and compares it to the results found in Hu et al. (2018), undertaken with the PISA 2015 wave. Results are similar in magnitude when using PISA 2018 and PISA 2015 data, and they point to a negative—and highly significant—relationship between ICT usage and mathematical performance in all the countries analysed.

Results by ICT Activity

The analysis so far has focused on the overall use of ICT at school. However, a relevant question to policy makers and instructors is to disentangle which specific ICT-related activities might be associated with improved or poorer performance of students (see Agasisti et al., 2020 for a preliminary analysis on the effects of ICT activities at home). To this end, equation (3) is now estimated by including the 10 specific ICT-related activities listed in Appendix A, which replace the ICT index as the control variable of interest.⁴ Each of these activities is introduced in the model as a categorical variable that reflects the reported frequency by students, with five possible answers that range from “never or hardly ever” to “every day” (see the “Variable Description” section), taking the earlier as the reference category.

The results (shown in Appendix G) suggest that for most of the activities, there is a negative association between excessive use and student achievement—particularly the instruction-related ones, to a larger extent than those related with homework—with very few exceptions. The most noteworthy exception relates to the activity of browsing the internet for schoolwork: spending time in this activity is associated with improved performance in mathematics, compared to never or hardly ever doing so, for the three main countries under study. For Spain and Estonia, this association is hill-shaped, whereas it is linear for Finland. Conversely, some other activities such as playing simulations at school, posting work on the school’s website or practicing and drilling yield either negative effects (in the first case) or non-significant effects (in the latter activity). These results support the findings of Agasisti et al. (2020), Hori and Fujii (2021), Luu and Freeman (2011), Odell et al. (2020) and OECD (2021) when studying the effects of ICT-related activities on different outcomes. Lastly, a hill-shaped relationship is found between the use of school computers for group work and communication with other students—among other activities—in Spain and Finland, where the monthly or weekly usage has positive effects on academic performance.⁵ In sum, this analysis suggests that the use of ICT at school is more helpful in some school activities than in others, which may hamper student achievement, and that the use of digital devices might be displacing other instructional activities (Falck et al., 2018; OECD, 2021).

As the overall conclusions on the negative effects of ICT overuse hold when analysing the activities separately, the reminder of the paper turns to the usage of the aggregate index. The following section will delve deeper into whether causality can be inferred regarding the negative impact of a very intensive use of ICT on students’ mathematical performance.

Results of the Inverse Probability Weighting Analysis

This subsection focuses on the causal impact of the very intensive user to assess whether the underperformance related to the very intensive ICT user seen in the previous subsection can actually be attributed to the very frequent use of ICT.

Before presenting the causal estimates, it is important to first ensure that the distribution of the propensity score (i.e., the predicted probability of being a very intensive ICT user) between the treatment and control groups are comparable such that the ATE is well founded, as long as extreme values are not present in the distribution (Cunningham, 2021). Figure 3 shows the distribution of the propensity scores to ensure that the application of IPW would not lead to biased estimates. In the three countries under study, the propensity scores are comparably distributed across treatment and control groups. In addition, extreme cases are uncommon, ensuring that the ATE estimation through IPW is justified.

Figure 3.

Propensity score distribution of treatment and control groups.

The Inverse Probability Weighting estimates are presented in Table 4. In particular, the table shows the Predicted-outcome means (Pomeans) and the ATE, that is, the estimated mean difference in performance of the very intensive user compared to the performance if she/he were to use ICT at school less frequently. For context, the observed gap in the math mean score is also included in the table.

Table 4.

Inverse Probability Weighting Estimates of Very Intensive ICT Usage.

	Spain	Estonia	Finland
Inverse probability weighting estimates
Pomeans	491.8***	533.5***	516.7***
Pomeans	(0.873)	(1.332)	(1.158)
ATE	−26.28***	−32.38***	−32.83***
ATE	(2.015)	(2.821)	(2.858)

Observed statistics
Observed math mean score not very intensive ICT user	485.3	529.0	512.0
Observed gap math mean score very intensive user versus not	−25.9	−30.4	−27.9

Note. Standard errors in parentheses: ***p < 0.01, **p < 0.05, *p < 0.1. Pomeans refers to the predicted-outcome means; and ATE, to the average treatment effect.

The results shown in Table 4 confirm that very intensive ICT usage causes significant underperformance in mathematics. That is, after approximating the observed variables between the treatment and control group (and assuming that the unobserved features are also assimilated), there is evidence that a very intensive usage of ICT causes substantial underperformance in mathematics. The usage of ICT at school more than 1–2 times per week reduces very significantly students score in mathematics. This penalty is equivalent to more than half an academic year for very intensive users in Spain, and ¾ of an academic year for the homologous users in Finland and Estonia.

As done earlier, the analysis is extended by socio-economic status and gender, with the aim of assessing whether the causal impact of the very intensive ICT usage on the mathematical performance holds across these groups.⁶ These results, shown in Appendix E (Table E2), confirm this fact and show that the relative impact of very intensive ICT usage is more negative for female students in Estonia and Finland, and for male students in Spain. In Spain and Estonia, the relative impact is more negative for students from high socio-economic profiles, while the opposite is found for Finland.

Discussion

In sum, after confirming the existence of an inverted U-shaped relationship between frequency of use of ICT at school and academic performance, which is significantly negative for very intensive users, the present study has confirmed that the penalty associated with the very intensive ICT usage is causal, rather than explained by the particular socio-demographic features of this student subgroup. Below, we discuss the results arising from this study and put them into context based on related literature.

The Hill-Shaped Relationship Between ICT Usage and Student Achievement in OECD Countries

The results on the inverted U-shaped relationship between ICT usage and student performance are aligned with Gubbels et al. (2020), who focused on the specific case of the Netherlands. While the setting of the study is slightly different—particularly regarding the field of study (reading), the measurement of the frequency of use of ICT at school (a continuous variable based on the OECD index) and the quadratic functional form of the model—results are still comparable. After controlling for similar covariates as in the present study, a hill-shaped relationship is found, and the difference in the mean predicted performance between the least and the most intensive user amounts to the equivalent of over an academic course. This is similar to the difference in the predicted mean in the reading performance found by the OECD (2015). Again, these magnitudes are not directly comparable to the present study, but the overall conclusions do concur. Similarly, the broad conclusions are in line with Borgonovi and Pokropek (2021) and Hu and Yu (2021), while they cover a number of countries in an aggregate manner, as in OECD (2015), and assess the relation with regard to students’ reading performance. In contrast, recent findings by Zhu and Li (2022) for Hong Kong are slightly different. While accounting for non-linearity, the authors assess a linear and negative relationship between ICT use at school and student performance in reading. However, the relatively less time available to efficiently use ICT tools compared to OECD peers might partly explain this divergence (Zhu & Li, 2022).

The Negative Impact of a Very Intensive ICT Usage on Student Performance

Concerning the causal negative impact of very intensive use of ICT at school on the mathematical performance found in this paper, to our knowledge there is no directly comparable paper to contrast the results with. However, the study by Agasisti et al. (2020) would constitute a close example assessing causality through analogous econometric techniques. The findings reveal that the intensive use of ICT at home has a negative causal impact on all subjects in most EU-15 countries. The present study focuses on the use of ICT at school, and hence further studies are needed to further delve into the direction of the impact, as well as the factor that might be driving the results.

Overall, the reasons underpinning the negative impact between the very intensive ICT usage on the mathematical performance are beyond the scope of the paper, but some potential hypotheses are explored here. On the one hand, students could possibly get distracted by using ICT at school for activities unrelated with the educational purpose of the usage of these devices. This might lead them to over-report the amount of time spent using technology at school (Agasisti et al., 2020). The possibilities that ICT offers students for “multitasking”, that is, performing a large number of tasks at the same time, can prove detrimental to students’ ability to capture information (Borgonovi & Pokropek, 2021; OECD, 2018; Vedechkina & Borgonovi, 2021). On the other hand, deficiencies in training teachers towards digitalisation have also been identified by the OECD (2018) and other authors (e.g., Hu et al., 2018) as an obstacle to successfully foster student learning through digital devices. This might be the case when teachers’ ICT knowledge is not regularly updated, although since the outbreak of the COVID-19 pandemic—which is not gauged in this study—many teachers and educators were forced to rapidly develop and learn ICT skills to optimise their instruction (Vedechkina & Borgonovi, 2021).

In view of the results, some policy implications can be drawn. First, stakeholders should exercise caution when integrating ICT at school to ensure that educational technology does not interfere with students’ learning processes. To this end, it is important to particularly monitor the very frequent users of technology at school—as the study has shown that this overuse has a strong negative impact on students’ achievement in mathematics—together with other key stakeholders in the integration of ICT at school, notably, teachers. If factors largely beyond the reach of instructors were found to explain students’ underperformance (e.g., distraction as a result of the large amount of time spent using ICT), as the study suggests, then policies should be addressed in order to limit what could be deemed as an excessive frequency of use. The study also highlights the need for teachers, educators, school principals and policy makers to carefully identify their context-specific deficiencies, which the paper has shown to differ both geographically and across student subgroups. Preliminary results suggest that the negative effects of ICT overuse at school be primarily associated with instruction-related activities, in contrast with other activities such as browsing the internet for schoolwork. Investigating the specific activities that contribute to improved achievement through rigorous studies is paramount for a well-founded implementation of ICT at school. The quality of integration of ICT at school could be improved through channels claimed in the literature, such as computer-assisted instruction or technology-based curricula (Hu et al., 2018).

Conclusions, Limitations and Future Research

The present study contributes to the field in two key ways. First, this study captures the varying effects of ICT at school in the performance in mathematics depending on the intensity of use for a number of OECD countries. Second, it applies the Inverse Probability Weighting technique to gauge the potential causal impact of ICT overuse on student performance, while most of previous studies using large-scale surveys limit the results to the correlation sphere.

The results from this study confirm the existence of a hill-shaped relationship in explaining the frequency of ICT usage at school and students’ performance in mathematics in 22 OECD countries, with varying magnitudes across countries and types of ICT activities. The study reveals that even in the most advanced countries in terms of ICT integration at school—such as Finland or Estonia—the group of very intensive frequency users experiences a significant penalty in terms of their performance in mathematics, while the low and medium ICT user status is related to better results than the very low user status.

However, these very intensive have a very differentiated socio-economic profile compared to the rest of users: they report above-average levels of bullying, and they are over-represented by male, repeaters and immigrant students. Given this, the study further explores whether the observed underperformance is attributed to such differentiated profiles or, conversely, whether the penalty can be attributed to the excessive use of ICT at school. Results indicate that the overuse of ICT causes an underperformance in mathematics, which is of the order of more than half academic year in Estonia, Finland and Spain.

The present study is not without limitations, which could be addressed in future research. The first relates to the measurement and definition of the main variable of interest, use of ICT at school, which is made on the basis of quantity of time spent, as opposed to quality of usage (Petko et al., 2017). In fact, the definition of the variable of interest is broad, which limits the interpretation of the results. Although we have attempted to disentangle the role of each ICT-related activities on student performance, the rationale for the differing effects—as well as the causal impact of the separate activities—remains a question to be addressed. In addition, future research could further explore whether the impact differs when looking at other computer-related activities that might be more specifically addressed to improving student performance, such as computer thinking applied to digital devices. The PISA data to date did not cover information of this nature; however, future editions intend to include activities such as the use of digital resources to solve equations or for coding purposes (Lorenceau et al., 2019). The second limitation refers to the absence of other covariates that might be of relevance to the model. One question that arises in view of the results is how the performance of students clustered in the same classroom where ICT is very intensively used might vary. This is currently unattainable with the PISA database due to the lack of an identifier that links students with classrooms. The availability of this data would also allow to identify whether specific ICT methodologies implemented by teachers, as well as ICT training received, entail a differential impact on student performance, which has been noted to be paramount in this research context (Pérez-Sanagustín et al., 2017). The third limitation relates to the cross-sectional nature of the database: the usage of panel data (or quasi-experimental studies) would further enrich the analysis and, notably, the causality analysis. A fourth limitation related to the causality analysis lies on the assumption that the unobserved features between treatment and control groups are assimilated, which might not always be the case if unobserved variables proved relevant in either of the two groups (whether treatment or control). Lastly, although this paper covers a wide range of countries, ICT activities and population subgroups, results cannot necessarily be generalised to other contexts, whether geographic or temporal. Despite the gaps that are yet to be overcome, the present paper has intended to further contribute to the exploration of the way ICT—which is increasingly present in schools—affects student performance, a paramount topic for instructors and policy makers in their search for an optimal use of technology that enhances students’ learning processes.

Footnotes

Acknowledgments

We would like to thank Sara de la Rica for her valuable advice and continuous support throughout the study.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: We gratefully acknowledge funding from Fundación COTEC para la innovación (Programa de Innovación Abierta 2019).

ORCID iD

Ainhoa Osés

Notes

Appendix A

Appendix B

Appendix C

Appendix D

Appendix E

Appendix F

Appendix G

Author Biographies

Lucía Gorjón is a researcher in the research centre Fundación ISEAK and holds a PhD in Economics (Cum Laude) from the University of the Basque Country under the supervision of Sara de la Rica. Her fields of research are labour market, gender and poverty and her research has been published in prestigious journals such as Social Indicators Research. She has evaluated the impact of several public policies aimed at groups at risk, such as Minimum Income Schemes, Active Labour Market Policies, training programs for young people and gender gaps. Lucía was named in 2019 one of the 29 Spanish under 35 called to revolutionise technology, finance and politics by the Business Insider. She started her career in FEDEA and was also Guest Researcher at the Deutsches Institut für Entwicklungspolitik (DIE) in collaboration with the World Bank.

Ainhoa Osés is a researcher in the research centre Fundación ISEAK and a PhD candidate in Labour Economics at the University of the Basque Country under the supervision of Sara de la Rica. Her role in ISEAK focuses on evaluating the impact of economic and social policies related to the labour market. In 2022, she undertook a PhD research stay at the OECD, in the Employment Labour, and Social Affairs division, examining which digital jobs and skills are most in demand in labour markets by analysing data on millions of online job postings. Her PhD focuses on topics such as labour precariousness or the effects of educational technology on skill acquisition of the youth. Prior to joining ISEAK, she spent over 3 years as an Economics researcher at the Spanish and Irish Fiscal Advisory Councils, where she combined academic research with research addressed to policy makers, such as governments or international organisations. Ainhoa graduated in Economics with honours and earned her Master’s degree at Universidad Carlos III de Madrid.

References

Agasisti

Gil-Izquierdo

Won Han

(2020). ICT use at home for school-related tasks: What is the effect on a student’s achievement? Empirical evidence from OECD PISA data. Education Economics, 28(6), 601–620. https://doi.org/10.1080/09645292.2020.1822787

Angrist

Lavy

(2002). New evidence on classroom computers and pupil learning. The Economic Journal, 112(482), 735–765. https://doi.org/10.1111/1468-0297.00068

Angrist

Pischke

J. S.

(2008). Mostly harmless econometrics: An empiricist’s companion. Princeton University Press.

Araya

Diaz

(2020). Implementing government elementary math exercises online: Positive effects found in RCT under social turmoil in Chile. Education Sciences, 10(9), 244. https://doi.org/10.3390/educsci10090244

Athey

Imbens

G. W.

(2017). The state of applied econometrics: Causality and policy evaluation. Journal of Economic Perspectives, 31(2), 3–32. https://doi.org/10.1257/jep.31.2.3

Aucejo

E. M.

French

Araya

M. P. U.

Zafar

(2020). The impact of COVID-19 on student experiences and expectations: Evidence from a survey. Journal of Public Economics, 191(3), 104271. https://doi.org/10.1016/j.jpubeco.2020.104271

Avagyan

Vansteelandt

(2018). Stable inverse probability weighting estimation for longitudinal series. Scandinavian Journal of Statistics, 48(3), 1046–1067. https://doi.org/10.1111/sjos.12542

Avvisati

(2020). The measure of socio-economic status in PISA: A review and some suggested improvements. Large-Scale Assessments in Education, 8(1), 8. https://doi.org/10.1186/s40536-020-00086-x

Bayraktar

(2001). A meta-analysis of the effectiveness of computer-assisted instruction in science education. Journal of Research on Technology in Education, 34(2), 173–188. https://doi.org/10.1080/15391523.2001.10782344

10.

Borgonovi

Pokropek

(2021). The evolution of the association between ICT use and reading achievement in 28 countries. Computers & Education Open, 2, 100047. https://doi.org/10.1016/j.caeo.2021.100047

11.

Bryk

A. S.

Raudenbush

S. W.

(1992). Hierarchical linear models: Applications and data analysis methods. Sage Publications, Inc.

12.

Busenbark

J. R

Yoon

H. E.

Gamache

D. L

Withers

M. C.

(2021). Omitted variable bias: Examining management research with the impact threshold of a confounding variable (ITCV). Journal of Management, 48(1), 17–48. https://doi.org/10.1177/01492063211006458

13.

Cabras

Tena Horrillo

J. D.

(2016). A Bayesian non-parametric modeling to estimate student response to ICT investment. Journal of Applied Statistics, 43(14), 2627–2642. https://doi.org/10.1080/02664763.2016.1142946

14.

Cheung

A. C. K

Slavin

R. E.

(2013). The effectiveness of educational technology applications for enhancing mathematics achievement in K-12 classrooms: A meta-analysis. Educational Research Review, 9, 88–113. https://doi.org/10.1016/j.edurev.2013.01.001

15.

Clark

A. E.

Nong

Zhu

(2021). Compensating for academic loss: Online learning and student performance during the COVID-19 pandemic. China Economic Review, 68, 101629. https://doi.org/10.1016/j.chieco.2021.101629

16.

Comi

S.L.

Argentin

Gui

Origo

Pagani

(2017). Is it the way they use it? Teachers, ICT and student achievement. Economics of Education Review, 56, 24–39. https://doi.org/10.1016/j.econedurev.2016.11.007

17.

Crompton

Burke

(2018). The use of mobile learning in higher education: A systematic review. Computers & Education, 123, 53–64. https://doi.org/10.1016/j.compedu.2018.04.007

18.

Cunningham

(2021). Causal inference: The mixtape. Yale University Press. https://doi.org/10.2307/j.ctv1c29t27

19.

De la Rica

Gorjón

(2019). Assessing the impact of a minimum income scheme: the Basque Country case. SERIEs, 10(3), 251–280.

20.

DuGoff

E. H.

Schuler

Stuart

E. A.

(2014). Generalizing observational study results: Applying propensity score methods to complex surveys. Health Services Research, 49(1), 284–303. https://doi.org/10.1111/1475-6773.12090

21.

Engzell

Frey

Verhagen

M. D.

(2021). Learning loss due to school closures during the COVID-19 pandemic. Proceedings of the National Academy of Sciences, 118(17), e2022376118. https://doi.org/10.1073/pnas.2022376118

22.

Escueta

Nickow

A. J.

Oreopoulos

Quan

(2020). Upgrading education with technology: Insights from experimental research. Journal of Economic Literature, 58(4), 897–996. https://doi.org/10.1257/jel.20191507

23.

European Commission (2020). Digital education action plan 2021–2027. Resetting education and training for the digital age. https://ec.europa.eu/education/sites/default/files/document-library-docs/deap-communication-sept2020_en.pdf

24.

Falck

Mang

Woessmann

(2018). Virtually no effect? Different uses of classroom computers and their effect on student achievement. Oxford Bulletin of Economics and Statistics, 80(1), 1–38. https://doi.org/10.1111/obes.12192

25.

Fariña

San Martín

Preiss

D. D

Claro

Jara

(2015). Measuring the relation between computer use and reading literacy in the presence of endogeneity. Computers & Education, 80, 176–186. https://doi.org/10.1016/j.compedu.2014.08.010

26.

Fernández-Gutiérrez

Gimenez

Calero

(2020). Is the use of ICT in education leading to higher student outcomes? Analysis from the Spanish autonomous communities. Computers & Education, 157(1), 103969. https://doi.org/10.1016/j.compedu.2020.103969

27.

Ferraro

(2018). Is information and communication technology satisfying educational needs at school? Computers & Education, 122, 194–204. https://doi.org/10.1016/j.compedu.2018.04.002

28.

Gabriel

Marrone

Van Sebille

Kovanovic

de Laat

(2021). Digital education strategies around the world: Practices and policies. Irish Educational Studies, 41(1), 85–106. https://doi.org/10.1080/03323315.2021.2022513

29.

Gamazo

Martínez-Abad

(2020). An exploration of factors linked to academic performance in PISA 2018 through data mining techniques. Frontiers in Psychology, 11, 575167. https://doi.org/10.3389/fpsyg.2020.575167

30.

Gamazo

Martínez-Abad

Olmos-Migueláñez

Rodríguez-Conde

M. J.

(2018). Assessment of factors related to school effectiveness in PISA 2015. A multilevel analysis. Revista de Educación, 2017(379), 56–84. https://doi.org/10.4438/1988-592X-RE-2017-379-369

31.

Giambona

Porcu

(2018). School size and students’ achievement. Empirical evidences from PISA survey data. Socio-Economic Planning Sciences, 64, 66–77. https://doi.org/10.1016/j.seps.2017.12.007

32.

Gil-Flores

Rodríguez-Santero

Torres-Gordillo

J. J.

(2017). Factors that explain the use of ICT in secondary-education classrooms: The role of teacher characteristics and school infrastructure. Computers in Human Behavior, 68, 441–449. https://doi.org/10.1016/j.chb.2016.11.057

33.

Gómez-Fernández

Mediavilla

(2021). Exploring the relationship between information and communication technologies (ICT) and academic performance: A multilevel analysis for Spain. Socio-Economic Planning Sciences, 77(1), 101009. https://doi.org/10.1016/j.seps.2021.101009

34.

Gubbels

Swart

N. M.

Groen

M. A.

(2020). Everything in moderation: ICT and reading performance of Dutch 15-year-olds. Large-scale Assessments in Education, 8(1), 1–17. https://doi.org/10.1186/s40536-020-0079-0

35.

Gumus

Atalmis

E. H.

(2011). Exploring the relationship between purpose of computer usage and reading skills of Turkish students: Evidence from PISA 2006. Turkish Online Journal of Educational Technology, 10(3), 129–140.

36.

Hori

Fujii

(2021). Impact of using ICT for learning purposes on self-efficacy and persistence: Evidence from Pisa 2018. Sustainability, 13(11), 6463. https://doi.org/10.3390/su13116463

37.

(2021). The effects of ICT-based social media on adolescents’ digital reading performance: A longitudinal study of PISA 2009, PISA 2012, PISA 2015 and PISA 2018. Computers & Education, 175, 104342. https://doi.org/10.1016/j.compedu.2021.104342

38.

Gong

Lai

Leung

F. K. S.

(2018). The relationship between ICT and student literacy in mathematics, reading, and science across 44 countries: A multilevel analysis. Computers & Education, 125, 1–13. https://doi:10.1016/j.compedu.2018.05.021

39.

Leuven

Lindahl

Oosterbeek

Webbink

(2007). The effect of extra funding for disadvantaged pupils on achievement. The Review of Economics and Statistics, 89(4), 721–736. https://doi.org/10.1162/rest.89.4.721

40.

Lorenceau

Marec

Mostafa

(2019). Upgrading the ICT questionnaire items in PISA 2021 (OECD education working papers No. 202). Organisation for Economic Co-operation and Development. https://doi.org/10.1787/19939019

41.

Luu

Freeman

J. G.

(2011). An analysis of the relationship between information and communication technology (ICT) and scientific literacy in Canada and Australia. Computers & Education, 56(4), 1072–1082. https://doi.org/10.1016/j.compedu.2010.11.008

42.

Malamud

Pop-Eleches

(2011). Home computer use and the development of human capital. The Quarterly Journal of Economics, 126(2), 987–1027. https://doi.org/10.1093/qje/qjr008

43.

Maldonado

De Witte

(2022). The effect of school closures on standardised student test outcomes. British Educational Research Journal, 48(1), 49–94. https://doi.org/10.1002/berj.3754

44.

Odell

Cutumisu

Gierl

(2020). A scoping review of the relationship between students’ ICT and performance in mathematics and science in the PISA data. Social Psychology of Education, 23(6), 1449–1481. https://doi.org/10.1007/s11218-020-09591-x

45.

OECD (2015). Students, computers and learning: Making the connection. OECD Publishing. https://doi.org/10.1787/9789264239555-en

46.

OECD (2017). PISA 2015 technical report. OECD Publishing. https://www.oecd.org/pisa/sitedocument/PISA-2015-technical-report-final.pdf

47.

OECD (2018). How has the internet use changed between 2012 and 2015? OECD Publishing. https://doi.org/10.1787/1e912a10-en

48.

OECD (2019). Pisa 2018: Insights and interpretations. OECD Publishing. https://www.oecd.org/pisa/PISA_2018_Insights_and_Interpretations_FINAL_PDF.pdf

49.

OECD (2020a). PISA 2018 results (volume VI): Are students ready to thrive in an interconnected world? OECD Publishing. https://doi.org/10.1787/d5f68679-en

50.

OECD (2020b). PISA 2018 results (volume II): Where all students ready can succeed. OECD Publishing. https://doi.org/10.1787/b5fd1b8f-en

51.

OECD (2021). 21st-century readers: Developing literacy skills in a digital world. OECD Publishing. https://doi.org/10.1787/a83d84cb-en

52.

Pérez-Sanagustín

Nussbaum

Hilliger

Alario-Hoyos

Heller

R. S.

Twining

Tsai

C. C.

(2017). Research on ICT in K-12 schools-A review of experimental and survey-based studies in computers & education 2011 to 2015. Computers & Education, 104, A1–A15. https://doi.org/10.1016/j.compedu.2016.09.006

53.

Petko

Cantieni

Prasse

(2017). Perceived quality of educational technology matters: A secondary analysis of students’ ICT use, ICT-related attitudes, and PISA 2012 test scores. Journal of Educational Computing Research, 54(8), 1070–1091. https://doi.org/10.1177/0735633116649373

54.

Pitchford

(2015). Development of early mathematical skills with a tablet intervention: A randomized control trial in Malawi. Frontiers in Psychology, 6(485), 485. https://doi.org/10.3389/fpsyg.2015.00485

55.

Polizzi

(2021). Internet users’ utopian/dystopian imaginaries of society in the digital age: Theorizing critical digital literacy and civic engagement. New Media & Society. https://doi.org/10.1177/14614448211018609

56.

Rodrigues

Biagi

(2017). Digital technologies and learning outcomes of students from low socio-economic background: An Analysis of PISA 2015. Publications Office of the European Union. https://doi.org/10.2760/415251

57.

Roschelle

Feng

Murphy

R. F.

Mason

C. A.

(2016). Online mathematics homework increases student achievement. AERA Open, 2(4), 233285841667396. https://doi.org/10.1177/2332858416673968

58.

Rubach

Lazarides

(2021). Addressing 21st-century digital skills in schools – development and validation of an instrument to measure teachers’ basic ICT competence beliefs. Computers in Human Behavior, 118(1), 106636. https://doi.org/10.1016/j.chb.2020.106636

59.

Skryabin

Zhang

Liu

Zhang

(2015). How the ICT development level and usage influence student achievement in reading, mathematics, and science. Computers & Education, 85, 49–58. https://doi.org/10.1016/j.compedu.2015.02.004

60.

Sung

Y. T.

Chang

K. E.

Liu

T. C.

(2016). The effects of integrating mobile devices with teaching and learning on students’ learning performance: A meta-analysis and research synthesis. Computers & Education, 94, 252–275. https://doi.org/10.1016/j.compedu.2015.11.008

61.

Szeles

M. R.

(2018). New insights from a multilevel approach to the regional digital divide in the European Union. Telecommunications Policy, 42(6), 452–463. https://doi.org/10.1016/j.telpol.2018.03.007

62.

Talib

A. M.

Bettayeb

A. M.

Omer

R. I.

(2021). Analytical study on the impact of technology in higher education during the age of COVID-19: Systematic literature review. Education and Information Technologies, 26(6), 6719–6746. https://doi.org/10.1007/s10639-021-10507-1

63.

Torgerson

Zhu

(2003). A systematic review and meta-analysis of the effectiveness of ICT on literacy learning in English. EPPI-Centre, Social Science Research Unit, Institute of Education, University of London.

64.

Vassilakopoulou

Hustad

(2021). Bridging digital divides: A literature review and research agenda for information systems research. Information Systems Frontiers. https://doi.org/10.1007/s10796-020-10096-3

65.

Vedechkina

Borgonovi

(2021). A review of evidence on the role of digital technology in shaping attention and cognitive control in children. Frontiers in Psychology, 12, 611155. https://doi.org/10.3389/fpsyg.2021.611155

66.

Vega-Bayo

Mariel

(2018). School choice across different regions of Spain. Hacienda Pública Española/Review of Public Economics, 227(4), 11–36. https://doi.org/10.7866/HPE-RPE.18.4.1

67.

Wang

Siau

(2019). Artificial intelligence, machine learning, automation, robotics, future of work and future of humanity: A review and research agenda. Journal of Database Management, 30(1), 61–79. https://doi.org/10.4018/jdm.2019010104

68.

Woessman

Fuchs

(2004). Computers and student learning: Bivariate and multivariate evidence on the availability and use of computers at home and at school. Center for Economic Studies and Ifo Institute. (CESifo) Working Paper 1321. https://doi.org/10.2139/ssrn.619101

69.

Woltman

Feldstain

MacKay

J. C.

Rocchi

(2012). An introduction to hierarchical linear modeling. Tutorials in Quantitative Methods for Psychology, 8(1), 52–69. https://doi.org/10.20982/tqmp.08.1.p052

70.

Wooldridge

(2002). Inverse probability weighted M-estimators for sample selection, attrition, and stratification. Portuguese Economic Journal, 1(2), 117–119. https://doi.org/10.1007/s10258-002-0008-x

71.

Wooldridge

(2010). Econometric analysis of cross section and panel data (2nd ed.). MIT Press.

72.

Zhao

(2021). The negative impact of bullying victimization on academic literacy and social integration: Evidence from 51 countries in PISA. Social Sciences & Humanities, 4(1), 100151. https://doi.org/10.1016/j.ssaho.2021.100151

73.

Zhang

Liu

(2016). How does ICT use influence students’ achievements in math and science over time? Evidence from PISA 2000 to 2012. Eurasia Journal of Mathematics, Science & Technology Education, 12(9), 2431–2449. https://doi.org/10.12973/eurasia.2016.1297a

74.

Zhu

S. S. C.

(2022). The non-linear relationship between ICT use and academic achievement of secondary students in Hong Kong. Computers & Education, 187(C). https://doi.org/10.1016/j.compedu.2022.104546