Abstract
The advances in information and communications technology and the digitization of services offer new ways to reach, engage with, and provide services to consumers. Recent advances in technology have fueled the rapid growth of digitization in education, and the education industry has witnessed radical changes in the provision and delivery of its products and services. Digital textbooks, which are equipped with various learning resources including multimedia aids, assessment questions, and hyperlinks to external resources, can be an important channel for harnessing technologies in classrooms. Korea's digital textbook experiment provides a unique empirical setting to examine the effects of digital textbooks on students’ academic outcomes. The authors employ a panel regression model with teacher fixed effects, propensity score weighting, and an instrumental variable strategy to find that greater usage of digital textbooks in class improves students’ academic performance, academic interest, and learning skills. The authors explore the heterogeneity in the utilization effect across student levels and find greater improvements in academic performance for low-achieving students. The findings have important managerial and policy implications for major stakeholders in the education sector, including teachers, school administrators, students, and policy makers.
The advances in information and communications technology (ICT) and the digitization of services offer new ways to reach, engage with, and provide services to consumers (e.g., see Rai and Sambamurthy [2006] and Yadav and Pavlou [2014] for an overview). The digitization of services, which expands the set of technology-enabled activities of consumers and firms, has significant effects on consumers’ decision making (e.g., Ariely 2000; Häubl and Trifts 2000), firms’ product and service offerings (e.g., Ansari, Essegaier, and Kohli 2000; Ying, Feinberg, and Wedel 2006), and marketplace efficiency (e.g., Brynjolfsson and Smith 2000; Brynjolfsson, Hu, and Smith 2003). The digitization of services has reshaped not only the industries that have relied on ICT for their service delivery for some time, such as retail, finance, and travel, but also those that have relatively more recently adopted technologies for their service delivery, such as government, health care, and education. In particular, the education sector has recently witnessed radical changes in the provision and delivery of its products and services.
Education has become a critical and resource-intensive activity for both providers and consumers. According to the National Center for Education Statistics (2022), the United States spent over $800 billion on public elementary and secondary school education, or over $13,000 per public school student, in 2018–2019. Along with the increase in the demand for and consumption of education, recent advances in technology have fueled the rapid growth of digitization in education. With the significant growth of educational innovations, such as interactive media, online learning tools, and massive open online courses (MOOCs), the global market for educational technology is projected to reach $350 billion by 2025 (Renub Research 2019). 1 Furthermore, the COVID-19 pandemic, which disrupted the education of 1.2 billion students around the world, has accelerated the adoption of digital educational technologies, and digital educational services are in higher demand than ever before. Therefore, it is important for researchers, policy makers, educators, and students to understand the effects of digitization on academic outcomes. In this research, we examine how digital textbooks, which manifest the digitization of education services, affect students’ academic outcomes.
Although the literature in marketing, information systems, and economics examines service digitization in multiple industries, including retail (e.g., Dickson 2000; Diehl, Kornish, and Lynch 2003), finance (e.g., Köhler et al. 2011; Zhu et al. 2012), and health care (e.g., Anderson and Agarwal 2011; Rajan, Tezcan, and Seidmann 2019), the extant literature shows mixed empirical evidence on the effectiveness of digitization. For example, in the context of the retail industry, the digitization of services such as decision-making tools can improve consumers’ decision quality and reduce search efforts (e.g., Ariely 2000; Diehl, Kornish, and Lynch 2003). However, an excessive amount of digital information can generate an information overload problem and hamper consumers’ decision-making process (e.g., Grover, Lim, and Ayyagari 2006; Jones, Ravid, and Rafaeli 2004). In addition, the physical intangibility inherent in digital marketplaces can increase the frustration level of consumers (Peck and Childers 2003) or reduce the perceived value of services (Abaidi and Vernette 2018). It is also empirically unclear whether the prior findings are applicable to the education sector. Our literature review, as summarized in Table 1, identifies three main gaps in the literature on the digitization of services, which this research aims to address.
Literature on the Digitization of Services.
aSee Yadav and Pavlou (2014) for an overview of literature pertaining to the digitization of services in the retail industry.
See Agarwal et al. (2010) for an overview of other issues pertaining to health care digitization.
cSee Hendershott et al. (2021) for an overview of other issues pertaining to fintech. See Barrett et al. (2015) and Lamberton and Stephen (2016) for additional discussion.
Notes: This table includes selected literature, and we provide additional literature review in Web Appendix A.
First, the existing marketing literature on the digitization of services focuses mainly on the retail or private sectors to examine how digitization affects consumer behavior and firms’ strategies and tactics (see Gap 1 in Table 1). Distinct from private retail consumption, students’ education consumption has potential social effects such as a reduction in inequality, an increase in social mobility, and stimulation of economic growth (Denison 1962; Nelson and Phelps 1966). Education system decisions are often made by policy makers and school administrators, and public investment in education generates social benefits that extend beyond the private benefits realized by individual decision makers (Becker 2009; Currie 2001). Thus, the digitization of education has important policy and social implications, which warrant further investigation.
Second, the digitization of services can have differential effects on different groups; that is, some groups may benefit more from digitization than others. Yet, the heterogeneity in the effects has been understudied (see Gap 2 in Table 1). As many education policies and interventions aim to reduce the education gap, policy makers and school administrators would benefit from understanding the heterogeneity in the effects of digitization across student groups. There is significant heterogeneity in the learning levels of students, and the mismatch between the level of classroom instruction and student learning levels can considerably lower the effectiveness of education (Glewwe and Muralidharan 2016). Digital textbooks can help accommodate the heterogeneity in students’ learning levels by providing content at the appropriate level for all students and allowing lagging students to catch up to course materials. Therefore, it is imperative to understand potential heterogeneity in the effect of digitization across student levels.
Third, the literature emphasizes that the actual usage of the technology, rather than the adoption of the technology, is the main driver of the IT impact (Devaraj and Kohli 2003; Goldfarb and Prince 2008). However, many prior studies examine the service adoption decisions or do not directly observe the actual utilization intensity (see Gap 3 in Table 1). In this research, we observe the actual utilization levels to examine the effects of digital textbook utilization on students’ academic outcomes. In summary, the digitization of textbooks provides a unique research setting to examine (1) how digitization reshapes the education sector, (2) how digitization differentially affects different students, and (3) how the actual utilization of services affects students’ outcomes.
Textbooks are important educational tools that provide standardized materials to teachers and students, ensure that course material is aligned with mandated curricula, support pedagogical approaches, and provide a focal point for instructional activities (Orey, Jones, and Branch 2012). Digital textbooks have recently emerged as an important component of technology-based education. Equipped with various learning resources including multimedia aids, learning management systems, assessment questions, and hyperlinks to external resources, digital textbooks can help foster a self-directed and collaborative learning environment, improve the quality of education, democratize education, and reduce costs. In 2018, the U.S. federal government allocated $5 million to develop and expand access to digital textbooks (Douglas-Gabriel 2018).
Despite the wide interest of policy makers, school districts, and publishers in implementing digital textbooks, the effectiveness of digital textbooks remains controversial and uncertain. The Association of American Publishers, for example, has expressed concern regarding the cost of hardware, software, and technology infrastructure, and some schools have conveyed their plan to delay digital textbook adoption decisions until the academic benefits of digital textbooks can be empirically corroborated (Mardis et al. 2010). As the COVID-19 pandemic compelled sudden and massive transitions to digital education, it became even more critical for key decision makers in the education sector to understand the effects of digital textbooks on students’ academic outcomes (Chesto 2020). However, there have been limited empirical attempts to estimate the effects of digital textbooks on students’ academic outcomes.
To examine the effects of digital textbooks on students’ academic performance, academic interest, and learning skills, we study Korea's educational policy on digital textbooks. As part of its digital textbook development initiatives and digital textbook experiments, Korea's Ministry of Education introduced digital textbooks to pilot schools and collected detailed panel data on students’ academic outcomes. Starting in 2014, digital textbooks for social studies and science classes were supplied to students attending pilot schools. Students can utilize digital textbooks in class by using school-provided personal computers or tablets, and they can access the digital textbooks outside the classroom using their personal digital devices. 2 The distributed textbooks have various interactive features, including multimedia resources, hyperlinks, evaluation tools, and online communities.
To effectively implement digital textbooks in classrooms, it is important for policy makers, educators, and schools to understand whether using digital textbooks in class has positive effects on students’ academic outcomes (i.e., utilization effect). To empirically examine whether greater utilization of digital textbooks in class has positive effects on students’ academic performance, academic interest, and learning skills, we exploit variations in in-class digital textbook utilization across students and compare the academic outcomes of students who are differentially exposed to digital textbooks. To address the potential endogeneity concerns, we employ various empirical approaches, including a panel regression model with teacher fixed effects, propensity score weighting, and an instrumental variable (IV) approach.
We measure students’ academic performance by their grades in social studies and science classes. At the same time, there are other aspects of learning that cannot be directly measured by examination grades. For example, the educational literature shows that students’ academic interest and attitudes have positive effects on students’ educational career aspirations (Singh, Granville, and Dika 2002). Therefore, we assess students’ academic interest levels from questionnaires and examine whether digital textbooks have positive effects on students’ academic interest. The literature on educational theory defines and categorizes students’ learning skills that are essential for students’ academic success. Bloom's (1956) learning taxonomy categorizes learning skills into cognitive, affective, and psychomotor domains, where the cognitive domain measures students’ intellectual skills, the affective domain measures students’ attitudes and appreciation, and the psychomotor domain measures students’ psychological and physical abilities. We measure Bloom's learning skill domains, along with other learning skill domains, by utilizing detailed questionnaires.
We find that greater in-class utilization of digital textbooks has positive and statistically significant effects on students’ academic performance, academic interest, and learning skills. We additionally find that greater utilization of digital textbooks improves students’ academic interest and learning skills. We explore the heterogeneity in the utilization effect across student levels and find that students who initially had low academic performance show greater improvements, compared with students who initially had high academic performance. When we explore potential mechanisms through which digital textbooks improve students’ academic outcomes, we find that using digital textbooks makes learning easier and more fun than traditional pedagogy. Furthermore, digital textbooks encourage students to interact more frequently with teachers and other students. We also conduct a back-of-the-envelope cost–benefit analysis to find substantial potential earnings benefits that exceed the costs of digital textbooks.
Related Literature
Effects of Digital Technology on Educational Outcomes
As highlighted in the introduction and summarized in Table 1, this research contributes to the literature on the effects of digitization of services. In addition, this research contributes to the literature on the effects of digital technology on educational outcomes. An overview of related literature on the effects of digital technology on educational outcomes, along with each study's research context and findings, is also summarized in Web Appendix A. First, this research relates to the literature that examines the effects of providing students access to new information technologies. Some notable examples include Malamud and Pop-Eleches (2011), Fairlie and Robinson (2013), and Beuermann et al. (2015) on computers; Goolsbee and Guryan (2006), Belo, Ferreira, and Telang (2013), and Dettling, Goodman, and Smith (2018) on access to the internet; and Castleman and Page (2015) and Fryer (2016) on mobile devices. The literature shows mixed empirical evidence regarding the effects of access to new technology on student outcomes. For example, Fairlie and Robinson (2013) find that providing students access to home computers does not have significant effects on students’ academic outcomes, and Vigdor, Ladd, and Martinez (2014) find that it has negative effects on students’ test scores. Our research differs from previous studies in that digital textbooks not only provide students with access to the internet and other devices but also have direct effects on pedagogy and the learning environment.
Our research also contributes to the literature that examines factors that contribute to effective pedagogy and learning environments. Previous studies show that education technology that encourages a self-directed, interactive, and collaborative learning environment has positive effects on students’ academic outcomes (e.g., Barrow, Markman, and Rouse 2009; Walkington 2013). Furthermore, Jackson and Makarin (2018) find that providing online learning materials to teachers helps improve students’ academic outcomes. However, there has been limited research on government-initiated educational technology interventions that directly affect in-class pedagogy and the student learning experience, as it may be technically, politically, and financially challenging to implement such interventions (Snilstveit et al. 2017).
Effects of Digital Textbooks
As digital textbooks are innovations that can foster and harness many factors that contribute to students’ learning experience, many policy makers and schools make substantial investments to introduce digital textbooks to classrooms. Despite the wide interest in adopting digital textbooks in classrooms, the literature on digital textbooks has been limited. Previousdigital textbook studies, including Weisberg (2011) and Junco and Clem (2015), examine electronic textbooks, which differ from digital textbooks in that electronic textbooks consist of the PDF files of printed textbooks and do not include multimedia or interactive resources. Korea has been a pioneer in designing and implementing digital textbooks, and this research is the first to empirically examine the first nationally approved digital textbooks by using rich panel data that are systematically collected from students, teachers, and schools.
Settings and Data
Digital Textbooks
Traditionally, education has depended significantly on printed materials and printed textbooks. However, with the rapid growth in educational technology, the use of digital technologies in the classroom now takes various forms, including complementing class materials with relevant images and videos, facilitating class discussions using online forums, and posting assignments online. Policy makers, educators, schools, and publishers have made various efforts to introduce and expand the usage of digital textbooks in classrooms. For example, legislatures in some states, including Virginia, California, Texas, and Florida, have passed laws that support and prompt the transition to digital textbooks. 3 Compared with traditional printed textbooks, digital textbooks are more cost-efficient because they significantly reduce shipping and distribution costs. Digital textbooks can also be easily updated to new editions and can be conveniently accessed at various locations and times.
When this technology first emerged, digital textbooks mostly referred to providing PDF files of printed textbooks (i.e., electronic textbooks). However, digital textbooks have evolved to take on more sophisticated forms. Digital textbooks today include interactive components and provide students with reference tools, quizzes, assignments, dictionaries, and multimedia content such as audio clips, video clips, and animations. Printed textbooks cannot physically contain the total volume of information required for a course, but digital textbooks can carry more information than multiple textbooks. We provide more specific details regarding digital textbooks in Korea in the following subsection.
Digital Textbooks in Korea
Korea's Ministry of Education, which oversees the nation's education system, has developed many policies and initiatives that encourage the adoption of ICT in education. The Ministry of Education is responsible for developing policies for primary, secondary, and higher education institutions, publishing textbooks and curricula, providing administrative and financial support to the school system, and supervising teacher-training colleges. Korean schools follow a standardized national curriculum developed by the Ministry of Education. In addition, all elementary students of the same grade use the same textbooks that are published by the Ministry of Education in their core subjects, including social studies, science, math, and Korean classes.
Web Appendix A1 provides institutional and policy background regarding Korea's ICT adoption in education. Central to the government's master plan on ICT adoption in education was the development and implementation of digital textbooks. The Ministry of Education's initiatives on digital textbooks consist of four distinct consecutive phases: electronic textbook development (2002–2006), digital textbook prototype development (2007–2011), integration of digital textbooks into the national curriculum (2012–2016), and harnessing artificial intelligence, augmented reality, and virtual reality in digital textbooks (2017–present). In Web Appendix A2, we discuss each phase in detail.
During the digital textbook prototype development (2007–2011) phase, the first prototypes of digital textbooks were developed and tested. In 2012, the Ministry of Education established the development procedures, compilation and review standards, legal regulations, technical specifications, and production guidelines for digital textbooks. In 2013, the governmental textbook agency completed the development of the first government-authorized national digital textbooks for third-grade, fourth-grade, and middle school social studies and science classes. In 2014, for the first time, digital textbooks were incorporated into the national curriculum as government-authorized textbooks. There was one digital textbook version for each grade level and subject, and digital textbooks remained unchanged throughout our observation period.
In 2014, digital textbooks were first supplied to 163 pilot schools (81 elementary schools and 82 middle schools), and students attending pilot schools could use digital textbooks in their social studies and science classes. We utilize data collected from pilot elementary schools to examine the effects of digital textbooks on students’ academic outcomes. We provide more details regarding pilot schools in the following subsection and Web Appendix A. The digital textbooks were easily accessible by all major operating systems including Microsoft Windows, Android, and iOS. Students can access the digital textbooks in class using school-provided personal computers or tablets, and they can additionally access digital textbooks outside the classroom using their personal mobile devices, tablets, and PCs.
The distributed digital textbooks contain numerous interactive features, and Table 2 summarizes the key features of the distributed digital textbooks and their key benefits. Students can electronically take notes, highlight, bookmark pages, and zoom in and out of their notes. Compared with note-taking on printed textbooks, note-taking on digital textbooks can be easier as students can electronically choose different colored and sized pens and highlighters for note-taking. Digital textbooks provide extra learning resources and allow students to access multimedia resources, such as images, video clips, audio, and animations embedded within the textbooks or presented via hyperlinks; access dictionaries; and access external resources using the web. Such extra learning resources can promote a self-directed learning environment, and students can self-evaluate their learning using interactive quizzes and other evaluation tools. Furthermore, students have access to an online community for digital textbook users, through which they can have group discussions, collaborate on projects, share their notes, submit assignments, and receive feedback from teachers and other students. Teachers were trained on how to use digital textbooks in both online and face-to-face sessions, and a customer service center operated with a toll-free number to address any digital textbook-related inconveniences that students, teachers, and parents face.
Features of Korea's Digital Textbooks.
Korea’s Digital Textbook Experiment Project (2014–2016)
As a part of the Korean Ministry of Education's digital textbook experiment project (2014–2016), government-authorized digital textbooks for social studies and science classes were first provided to 81 pilot elementary schools starting in 2014. In the 2014 pilot experiment, third-grade and fourth-grade students attending a pilot school received digital textbooks, along with printed textbooks. In the 2015 pilot experiments, the same students were included as they became fourth-grade and fifth-grade students in 2015. The 2016 pilot experiment included the same students, and they became fifth-grade students in 2016. At the beginning of each academic year, an elementary student in Korea is assigned to a class, and the student remains with the same classmates and teacher throughout the school day, which is similar to the elementary school structure employed in the United States.
The Ministry of Education and Gwangju National University of Education conducted panel surveys on students and teachers who attended pilot schools. From the initial pool of 81 pilot schools, 35 schools were selected, and from those chosen schools, students were randomly sampled to be included in the panel. To select pilot schools, the Ministry of Education distributed official statements regarding the digital textbook experiment to all elementary schools and municipal ministries of education. Then, an external education committee was formed to carefully and impartially select 81 pilot schools while taking into account the representativeness of schools, school locations, and school sizes. Among the 81 pilot schools, random cluster sampling was utilized to randomly sample 35 nationally representative pilot schools. Then students were randomly sampled from these 35 pilot schools to be included in the survey panel. During our research period, there were no other policy changes regarding digital textbooks or printed textbooks.
The first survey was conducted in 2014, and the same students were surveyed in 2015 and 2016. The survey data cover 385 students who were in the third grade in 2014 (and became fourth-grade students in 2015 and fifth-grade students in 2016) and 386 students who were in the fourth grade in 2014 (and became fifth-grade students in 2015). The survey data include detailed information on students’ academic performance, students’ academic interest, students’ learning skills, in-class digital textbook utilization intensity, students’ demographic characteristics, and teachers’ characteristics. Table 3 presents the descriptive statistics of our main variables.
Variable Description and Summary Statistics.
Notes: DT = digital textbooks, PT = printed textbooks. The mean column shows the proportion of responses that fall into each category.
This research is the first to employ detailed panel data to examine the first nationally approved digital textbooks incorporated into the national curriculum. Prior to our research, some pilot studies were conducted during the digital textbook prototype development (2007–2011) phase to examine the feasibility and effectiveness of digital textbooks. However, prior studies often suffered from a small number of observations because the data were collected from a small number of test schools. Furthermore, as digital textbook prototypes were not standardized or nationally approved, a systematic and consistent evaluation of digital textbooks was difficult. Web Appendix A summarizes the prior studies on digital textbook prototype development (2007–2011). As a highly educated nation, Korea is one of the OECD's top-performing countries in mathematics, reading, and science in the Programme for International Student Assessment and has the highest enrollment rate of tertiary education among the OECD countries (OECD 2016). 4 In addition, Korea has a strong presence in the ICT market, with the world's fastest internet speeds and top-ranking broadband penetration rates. Korea's digital textbook experiment provides a unique opportunity to examine the effect of digital textbooks in multiple dimensions.
Outcome Measures
The data include information on students’ academic performance, academic interest, and learning skills. We assessed students’ learning skills through survey questions that measure various aspects of students’ learning abilities and behaviors. Bloom's learning taxonomy distinguishes learning into three separate dimensions comprising the cognitive, affective, and psychomotor domains (Bloom 1956). In addition to the three domains from Bloom's learning taxonomy (i.e., cognitive, affective, and psychomotor domains; Bloom 1956), we assessed students’ academic domain and social domain. We summarize our main outcome variables in this section.
Academic performance
We measured students’ academic performance from their first-semester examination grades in social studies and science classes.
Academic interest
To measure students’ academic interest in social studies and science classes, students were asked to express how much they agree or disagree with the statements: “I like social studies” and “I like science” on a five-point Likert scale (1 = “Not at all,” and 5 = “Very much so”).
Learning skills
To assess different aspects of students’ learning skills, students were asked to express how much they agree or disagree with a particular statement on a five-point Likert scale. To generate questionnaires that are valid and reliable, an expert committee, overseen by the Ministry of Education, was formed. In Web Appendix B, we describe the process through which questionnaires were developed, verified, tested, and finalized. The domains were measured as follows:
For each domain, we calculated the averages of students’ responses across all questions to measure students’ learning skills for a given domain.
Theoretical Background
Utilization Effects of Digital Textbooks
In implementing digital textbooks, policy makers and schools need to consider whether the greater usage of digital textbooks in class has positive effects on students’ learning outcomes (i.e., utilization effect). We rely on the literature on educational theory to explore mechanisms through which digital textbooks can enhance students’ learning. Learning involves the transfer of information from short-term working memory to long-term memory. Mayer's cognitive theory of multimedia learning shows that learning is more effective when information is delivered through multiple separate channels rather than a single channel; individuals learn more deeply from words and pictures together than from just words alone (Mayer 1997, 1999). Digital textbooks include various multimedia content, including images, video clips, audio, and interactive games, which can facilitate students’ learning. Furthermore, digital textbooks can encourage nonlinear learning. A nonlinear learning model refers to a learning environment where students are presented with rich information, and students can freely explore the content in a sequence of their choice rather than passively following a predefined linear learning path (Chen 2002; Farrell and Moore 2000). Nonlinear learning helps accommodate students with different levels of abilities, prior knowledge, and learning styles.
Digital textbooks encourage self-directed learning because students make active learning decisions about which content to study, how much time to spend studying, and whether they understand the materials (Azevedo 2005). Digital textbooks can help create an interactive and collaborative learning environment and improve students’ academic performance, academic interest, and learning skills. Therefore, we first examine whether greater usage of digital textbooks in class improves students’ academic performance, academic interest, and learning skills.
Heterogeneity in Utilization Effects Across Student Levels
Digital textbooks can have differential effects on students with different levels of initial abilities. The literature shows that students with low academic achievement are characterized by low self-motivation, a low attention level, and a lack of self-confidence (Al-Zoubi and Younes 2015). Furthermore, low-achieving students tend to lack prior knowledge and background information at both general and specific domain levels. Therefore, students who are behind in the curriculum may learn little in school if the classroom instruction is heavily based on standardized textbooks that are considerably beyond their learning levels (Muralidharan, Singh, and Ganimian 2018; Pritchett and Beatty 2015).
Digital textbooks can help improve such traits of low-achieving students by providing an interactive, collaborative, and self-regulated learning environment. The attention span of students can be improved and better maintained by utilizing the interactive features of digital textbooks (Maynard and Cheyne 2005). The collaborative aspect of digital textbooks can be especially helpful for low-achieving students because they can benefit from class discussions with high-achieving students and learn from their peers (Uribe, Klein, and Sullivan 2003). Low-achieving students can self-pace and self-monitor their learning process, and review materials for which they do not have sufficient knowledge. Therefore, using digital textbooks can help accelerate the learning of low-achieving students and help them develop higher-order cognitive learning skills. We examine whether students who initially had low academic performance show greater improvement in their academic performance, academic interest, and learning skills, compared with students who initially had high academic performance.
Empirical Methodology
Empirical Model
To examine whether the usage of digital textbooks in class improves students’ academic performance, academic interest, and learning skills, we exploit variations in in-class digital textbook usage across students. Students’ utilization of digital textbooks in class depends heavily on teachers’ pedagogy and practices. With students spending an average of 6.6 hours per school day in classrooms (Roth et al. 2003), teachers’ practices have significant effects on students’ learning (Ashton and Webb 1986; Rosenholtz 1985). Students who are taught by teachers who use digital textbooks more frequently in class are exposed to digital textbooks more than students who are taught by teachers who use digital textbooks less frequently. In pilot schools, as in most public schools in Korea, students are randomly assigned to teachers, which generates variations in students’ in-class exposure to digital textbooks. To empirically examine whether students who use digital textbooks more in class show greater improvements in academic performance, academic interest, and learning skills, we compare the learning outcomes of students who utilized digital textbooks frequently in class (i.e., treatment group) with those of students who did not utilize digital textbooks frequently in class (i.e., control group). We provide additional institutional details on the Korean education system and discuss potential sources of the variations in digital textbook usage in Web Appendix A5.
We empirically examine whether utilizing digital textbooks more frequently in class leads to greater improvements in learning outcomes by estimating the following baseline model:
There are potential sources of endogeneity concerns in the model, and we employ various empirical approaches to address them. Specifically, (1) teachers’ decisions on in-class digital textbook utilization may be endogenous, and (2) measurement errors may be present in the measurement of digital textbook utilization intensity. First, teachers’ decisions on in-class digital textbook utilization can be endogenous if teachers who utilize digital textbooks frequently in classes are systematically different from teachers who do not. Teachers’ decisions to adopt new technologies in classrooms may depend on teachers’ beliefs, attitudes, pedagogical ideologies, and pedagogical approaches (Fullan and Stiegelbauer 1991). Therefore, we take several approaches to address the potential endogeneity concern. First, we observe a rich set of data on student, teacher, and school characteristics, and control for them in our empirical models. Second, we employ a teacher fixed-effects model to control for time-invariant differences in teacher quality and personality. Third, we employ a propensity score weighting method to correct for observable characteristics that may have affected the treatment assignment status. Finally, we show that our results are robust to various alternative empirical strategies and specifications, such as employing alternative propensity score matching methods, an endogenous treatment model, and a student fixed-effects model (see the “Robustness Checks” subsection). Another potential source of an endogeneity concern would be measurement errors in the measurement of digital textbook utilization intensity. Possible sources of measurement errors include errors in students’ reporting, in recording of answers, and in the wording of questionnaires (Bound, Brown, and Mathiowetz 2001). We employ an IV approach to account for the potential endogeneity problem due to possible measurement errors.
Teacher Fixed-Effects Model
To address the potential endogeneity in teachers’ decisions, we estimate a teacher fixed-effects model in which the within-teacher variations in digital textbook usage frequency are exploited to examine changes in student performance. The teacher fixed-effects model is employed in the education literature to control for all time-invariant unobservable teacher characteristics, such as teachers’ unobservable inherent quality and personality, that may remain in the error term even after we control for various observable teacher characteristics (e.g., Buddin and Zamarro 2009; Chingos and Peterson 2011). For example, if highly effective teachers are those who are more likely to employ digital textbooks in class, the teachers’ effectiveness may confound the effects of digital textbook utilization. In such cases, the teacher fixed effects can account for teachers’ time-invariant unobservable inherent effectiveness. As the teacher fixed effects cannot account for time-varying teacher characteristics, the model also includes teacher-level time-variant covariates, such as teachers’ experience and training levels (Wooldridge 2016). Web Appendix C1 validates the teacher fixed-effects model and ensures that enough variations in the data remain after employing the fixed effects (Lovell 1963; Mummolo and Peterson 2018).
Propensity Score Weighting Method
To further mitigate possible endogeneity in teachers’ decisions, we use propensity score weighting—inverse probability of treatment weighted (IPTW) estimation—to account for differences in observable characteristics that arise from nonrandom treatment assignment and to construct a treatment group and control group that are comparable (Rosenbaum and Rubin 1983). Under the assumption that unobservable characteristics that affect outcomes are uncorrelated with the treatment status after conditioning on observable characteristics, propensity score matching estimators yield unbiased estimates of treatment effects (Dehejia and Wahba 2002; Heckman, Ichimura, and Todd 1997; Khwaja et al. 2011).
The literature on educational computer use shows that teachers’ observable characteristics, such as a teacher's gender (Shapka and Ferrari 2003; Van Braak, Tondeur, and Valcke 2004) and age (Bradley and Russell 1997), are related to in-class computer use. Teachers who frequently utilize digital textbooks in class may differ in multiple observable dimensions. Therefore, we utilize rich available data on observable teacher and school characteristics to predict the treatment status using a logistic model. The model finds a set of parameters that maximize the likelihood of observing the treatment assignments in the data. Then, we estimate each student i's propensity score,
In employing the IPTW estimation, it is important to evaluate how well the treatment and control groups are balanced in the weighted sample. If the treatment and control groups are poorly balanced, the propensity score needs to be respecified (Austin 2009; Ho et al. 2007). To assess the covariate balance, we examine the standardized differences and variance ratios. The tests of covariates balance in Web Appendix C2 indicate that the propensity score weighting method balances the covariates of the treatment group and the control group.
Instrumental Variable
To account for the potential endogeneity problem due to possible measurement errors that may be present in the measurement of digital textbook utilization intensity, we employ an IV approach. In addressing possible measurement errors in a key independent variable, a second measurement of the variable is often used as a valid IV when available (Wooldridge 2016). Therefore, we employ a second measurement of in-class digital textbook utilization as an IV. In the teacher survey, teachers were asked how often digital textbooks are utilized in class in a week. Following the nationally mandated curriculum, schools are required to have three social studies and three science classes each week. Therefore, possible answers to the teacher's survey questions were 0, 1, 2, and 3. We operationalize the alternative digital textbook utilization intensity variable, Alternative DT Utilization, as a binary variable with two levels (T0 = low utilization and T1 = high utilization), where the high-utilization group refers to students whose teachers used digital textbooks more than twice a week. As both teachers’ and students’ responses measure students’ exposure to digital textbooks, the two variables are correlated. At the same time, measurement errors from the two variables will not be correlated because the two measurements were collected independently from two different respondents using different question wordings. In Web Appendix C3, we present several tests that support the validity of our IV. We test for the strength of the IV using the Angrist–Pischke multivariate F-statistic, and the results confirm that the IV is strongly correlated with the DT Utilization variable (F = 121, p < .01).
Results
Main Results
We first operationalize the digital textbook utilization intensity as a binary variable with two levels (T0 = low utilization and T1 = high utilization), where the high-utilization group refers to students who used digital textbooks more frequently than printed textbooks or used only digital textbooks for in-class social studies and science education. The low-utilization group refers to students who used digital textbooks less frequently than printed textbooks for in-class education. The high-utilization group includes 1,133 observations, and the low-utilization group includes 794 observations. Table 4 presents the estimation results for utilization effects for the different models we employ. The estimates of coefficients on the variable DT Utilization, which is an indicator variable for the high-utilization group, reveal that greater in-class utilization of digital textbooks has positive and statistically significant effects on students’ academic performance, academic interest, and learning skills.
Utilization Effects of Digital Textbooks on Academic Outcomes.
*p < .10, **p < .05, ***p < .01.
Notes: Standard errors are clustered at the class level.
Model 1 controls for all observable student, teacher, and school characteristics, and students in the high-utilization group have higher academic outcomes. Model 2 employs the teacher fixed-effects model and consistently shows that students in the high-utilization group have higher academic outcomes after controlling for time-invariant teacher characteristics. For example, students’ social studies examination grades are .12 standard deviations (significant at 10%) higher for students who utilize digital textbooks more frequently in class. The IPTW model (Model 3) reveals that positive treatment effects are achieved after balancing the covariates of the treatment group and the control group to account for nonrandom treatment status based on observable characteristics. For example, the first-stage logistic regression result for the IPTW model reveals that male teachers are more likely to utilize digital textbooks (Web Appendix C2). This is comparable to the findings of Van Braak, Tondeur, and Valcke (2004), who report that male teachers are more likely to utilize computers in class. After correcting for observable characteristics, the IPTW model shows that students’ social studies examination grades and science examination grades are each .09 standard deviations (significant at 10%) and .08 standard deviations (significant at 10%) higher for students who utilize digital textbooks more frequently in class. The IV model (Model 4) also shows that digital textbooks have positive effects on students’ learning outcomes. The results of the first stage are presented in Web Appendix C3. After accounting for measurement errors by using IVs, we find that greater utilization of digital textbooks in class improves students’ social studies examination grades by .28 standard deviations (significant at 10%) and science examination grades by .32 standard deviations (significant at 10%). Furthermore, higher utilization of digital textbooks increases students’ academic interest in social studies by .43 standard deviations (significant at 5%) and academic interest in science by .35 standard deviations (significant at 10%). For the outcome variables that measure students’ learning skills, the results show that greater utilization of digital textbooks in class has positive effects on students’ learning skills in academic, cognitive, affective, social, and psychomotor domains.
We use separate dummy variables that indicate different levels of treatments (T0 = used printed textbooks only, T1 = used printed textbooks more frequently than digital textbooks, T2 = used digital textbooks and printed textbooks equally, T3 = used digital textbooks more frequently than printed textbooks, and T4 = used digital textbooks only) to examine the extent to which different digital textbook utilization levels have differential effects on students’ academic outcomes. We find that greater usage of digital textbooks is generally associated with higher and more statistically significant effects on students’ academic outcomes. For example, when the outcome variable measures students’ social studies examination grades, using printed textbooks more frequently in class has positive but statistically insignificant effects, compared with not utilizing digital textbooks. At the same time, using only digital textbooks increases the grade by .22 standard deviations (significant at 5%), compared with not utilizing digital textbooks.
Robustness Checks
We conduct several robustness checks to ensure that our results are robust to alternative specifications, alternative measurements, and additional controls. First, we replicate our estimations using a Tobit model to take into account that our outcome variables are bounded (Table 4, Model 5). Students’ grades and responses to survey questionnaires are censored at the left and right. Second, we validate the measurement of digital textbook usage intensity and show that our results are not sensitive to alternative measurements of usage intensity (Web Appendix C4). For example, instead of employing our original DT Utilization variable, we utilize teachers’ responses and show that our results are robust to using Alternative DT Utilization as our main independent variable. The robustness check using Alternative DT Utilization, which is defined at the class level, shows that the results are not driven by within-class variations in students’ responses. We also show that the results are robust to employing another alternative utilization intensity measurement, Mean DT Utilization, which is the mean of the original DT Utilization variable, where the mean is computed at the class level (Web Appendix C4). Third, we show that results are robust to employing alternative propensity score matching methods, including kernel matching and nearest neighbor matching. Web Appendix C2 presents details regarding the robustness of propensity score matching methods. Fourth, our results are robust to using an endogenous treatment model following the Heckman (1976, 1978) two-step approach, where a teacher's decision to utilize digital textbooks frequently in class is modeled separately from students’ academic outcomes. Web Appendix D presents details regarding the endogenous treatment model. Fifth, we show that our results are robust to alternative specifications, including a teacher fixed-effects model with IVs, a student fixed-effects model, and a student fixed-effects model with IVs (Web Appendix E). Although the main models control for student characteristic variables, time-invariant unobservable student characteristics can lead to an endogeneity concern. Therefore, we employ a student fixed-effects model to account for students’ time-invariant unobservable characteristics, such as motivation, personality, and innate abilities. In addition, the student fixed-effects model can account for survey response biases that are specific to an individual student. 9 Finally, we replicate our estimations using an alternative measurement of students’ learning skills. More specifically, instead of using the averages of students’ responses to questionnaires, we use the first question of each domain, which may be less subject to the potential respondent fatigue problem (Web Appendix E).
Heterogeneity in Effects Across Students’ Initial Academic Performance Levels
We explore the heterogeneity in the effects of digital textbooks across students’ academic performance levels. The examination of heterogeneity in effects provides insights into the underlying mechanisms and provides practical implications for educators who seek to understand the potential roles of digitization in reducing education inequality. As many education policies and interventions aim to reduce the education gap, policy makers and school administrators are interested in understanding the heterogeneity in the effects of interventions for academically weaker and disadvantaged students. To examine whether low-achieving students who initially had lower than average grades show greater improvements in their academic performance, we estimate
The results presented in Table 5 consistently show that the estimates of the coefficient on the interaction term are positive and statistically significant when the outcome variables measure students’ academic performance, which indicates that greater in-class utilization of digital textbooks has greater positive effects on low-achieving students’ academic performance. For example, when compared with high-achieving students, low-achieving students improve their social studies examination grades additionally by around .12 (Model 4, significant at 10%) and .22 (Model 1, significant at 5%) standard deviations, depending on the models we employ. Similarly, low-achieving students’ science examination grades improve by around .14 (Models 1–4) standard deviations more when compared with high-achieving students. The findings suggest that digital textbooks can help low-achieving students who may lack educational support gain access to high-quality, interactive, and self-directed learning materials and improve their academic outcomes. In Web Appendix E, we additionally explore the heterogeneity in the effects of digital textbooks on students whose parents’ highest level of education was less than high school education, and we find that students with lower parental education show greater improvements in their academic outcomes.
Heterogeneity in Effects Across Students’ Initial Academic Performance Levels.
*p < .10, **p < .05, ***p < .01.
Notes: Standard errors are clustered at the class level. In Model 4, DT Utilization and DT Utilization × Low are instrumented with Alternative DT Utilization and Alternative DT Utilization × Low.
Digital Textbooks and Students’ Learning Experience
In this section, we explore potential mechanisms through which digital textbooks improve students’ academic outcomes by employing questionnaires that aim to assess how digital textbooks affect students’ learning experiences. Students are asked to express on a five-point Likert scale how much they agree or disagree with the following four statements when studying with digital textbooks: “Class materials are easier to learn,” “Class materials are more fun to learn,” “I interact more frequently with teachers in classes by actively participating, volunteering answers, and asking questions,” and “I interact more frequently with other students in classes.” These four statements distinguish and measure four important channels through which digital textbooks can enhance students’ learning experience. Table 6 presents the estimation results when outcome variables measure students’ responses to the four questions. We find that utilizing digital textbooks more frequently in class makes learning easier and more fun (Columns 1 and 2). Furthermore, students interact more frequently with teachers and other students (Columns 3 and 4), which suggests that digital textbooks can help promote an interactive and collaborative learning environment.
Digital Textbooks and Students’ Learning Experience.
*p < .10, **p < .05, ***p < .01.
Notes: Standard errors are clustered at the class level. In Model 4, DT Utilization and DT Utilization × Low are instrumented with Alternative DT Utilization and Alternative DT Utilization × Low.
In addition, we examine whether low-achieving students, who show greater improvements in academic performance from utilizing digital textbooks, are also more likely to respond that utilizing digital textbooks improves their learning experiences and increases their interactions with teachers and other students. Table 6, Panel B, shows positive and statistically significant coefficients on the interaction term (DT Utilization × Low) for most models we estimate, which indicates that improvements in the learning experience are indeed greater for low-achieving students compared with those for high-achieving students. The results suggest that digital textbooks, which provide an interactive and collaborative learning environment, make learning easier, more fun, and more engaging for all students. At the same time, the positive effects are greater for low-achieving students.
Earliness Effects of Digital Textbooks
In addition to the utilization effect, educators need to consider the effects of the age of students at their entry into the digital textbook intervention. Early educational interventions produce meaningful and lasting effects on cognitive, social, and schooling outcomes (Barnett 2011). For example, Butler (1991) finds that poor reading competency compounds over the school years, and a reading intervention program for elementary students helps break the cycle of reading failure. A well-established finding in the cognitive-developmental literature is that brain plasticity, which refers to the brain's ability to change its function and structure with experience, reduces with age (Jacobson 1969). Digital textbooks can have greater impacts on younger students, whose brain plasticity is higher. Therefore, we examine whether exposure to digital textbooks at an earlier age is positively associated with improvements in students’ academic outcomes (i.e., earliness effect).
To do so, we exploit the panel nature of the data and a unique research setting in which different student cohorts were differentially exposed to digital textbooks. Digital textbooks were introduced to two cohort groups in the same calendar year (i.e., 2014), but a younger cohort group (i.e., early exposure cohort) was exposed to digital textbooks starting from third grade whereas an older cohort group (i.e., late exposure cohort) was exposed to digital textbooks from fourth grade. We compare students’ academic outcomes measured in the second year of exposure relative to the first year of exposure between the early exposure and late exposure cohorts, while holding the number of exposure years constant between the two groups.
11
We estimate
Table 7 shows that coefficients on the interaction term between the early exposure cohort and second exposure are positive and statistically significant for most models we estimate. This indicates that the early exposure cohort, who started using digital textbooks at an earlier age, shows greater improvement in academic outcomes when we compare outcomes measured in the second year of exposure relative to the first year of exposure.
Earliness Effects of Digital Textbooks.
*p < .10, **p < .05, ***p < .01.
Notes: Among 771 students included in the panel data, the early exposure cohort includes 385 students, and the late exposure cohort includes 386 students. Standard errors are clustered at the class level.
Cost–Benefit Analysis and Conclusions
We conduct a back-of-the-envelope cost–benefit analysis to analyze the benefits and costs of digital textbooks. We also discuss the managerial and policy implications of our findings for major stakeholders in the education sector, including teachers, school administrators, students, and policy makers.
Cost–Benefit Analysis
Earnings benefits
To quantify monetary benefits associated with digital textbooks, we follow approaches taken in the education literature and associate improvements in exam grades with an increase in students’ future earnings (e.g., Chetty et al. 2011; Chetty, Friedman, and Rockoff 2014; Kane and Staiger 2002). The education literature robustly shows that a one-standard-deviation increase in exam grade is associated with a 7% to 18% increase in earnings (e.g., Chetty et al. 2011; Chetty, Friedman, and Rockoff 2014; Murnane, Willett, and Levy 1995; Neal and Johnson 1996), and Kim and Moon (2007) use Korean data to show consistently that a one-standard-deviation increase in Korean students’ exam grades is associated with a 7.3% increase in earnings. By incorporating the estimates from Kim and Moon (2007), along with our utilization effect estimates, we compute the effect of digital textbooks on undiscounted lifetime earnings to be $26,884 per student. When we follow Krueger (1999) and assume a 3% annual discount rate, the present value of the increase in lifetime earnings is $10,145 at age ten, which is the average age of the intervention (see Web Appendix F and Chetty et al. [2011] for calculation details). 12 The calculation, which reveals significant gains, may be an underestimation of the benefit, as it ignores the nonmonetary gains from improved academic outcomes, such as reduced crime (Machin, Marie, and Vujić 2011) and improved health prospects (Currie 2009; Cutler and Lleras-Muney 2010). 13
Fiscal costs
Next, we examine costs associated with digital textbook implementation. In calculating the cost, we categorize the costs into infrastructure costs (covering costs associated with the internet access, devices, and teacher training) and digital textbook content production costs. To cover the costs of our digital textbook experiment, the Ministry of Education allocated and spent approximately $923 per student ($823 per student on the infrastructure and $100 per student on content production), which is significantly lower than the earnings gains. We note that large components of the infrastructure costs are nonrecurring upfront costs, which can complement other policies and initiatives that encourage the adoption of ICT in education. In addition, as prices of digital devices decline over time with improvements in technologies, we expect lower fiscal costs in the future.
Turning to the content production cost, we note that digital textbooks can be more cost-efficient than traditional printed textbooks because digital textbooks are not subject to costly printing and distribution processes. A key characteristic that differentiates digital goods from traditional durable goods is that although the cost of producing the first digital content (i.e., fixed cost) can be high, once the digital content is produced, the marginal production and distribution costs are extremely low. More specifically, a total of $173 million is spent every year to cover the production, shipping, and distribution of traditional printed textbooks. 14 In contrast, the Korea Education and Research Information Service (KERIS), which oversees the development and dissemination of digital textbooks, estimates $63 million as the total initial digital textbook content development costs for all subjects in mandatory curricula, 15 but negligible maintenance and distribution costs after the initial development cost.
Additional Managerial and Policy Implications
The digitization of services has significantly reshaped the education landscape. Advances in digital technology can provide better and more equitable learning opportunities for students and help teachers deliver education more effectively. Although governments, school districts, and private sectors make significant investments in providing technology tools to students, there has been limited empirical evidence on the effectiveness of such investments. The key players in education, including educators, publishers, consumers, and policy makers, seek to make informed decisions on digital textbook implementation. Korea's digital textbook experiment provides a unique opportunity to examine the effect of digitization on students’ academic outcomes.
Our findings have several important practical implications for major stakeholders in the education sector. First, schools and policy makers should make adequate investments in digital textbooks, as they have positive effects on students’ academic outcomes. The cost–benefit analysis of digital textbooks also shows substantial potential benefits. Second, educators, policy makers, and teachers who seek to close the educational gap can consider prioritizing the adoption of digital textbooks for academically disadvantaged students as digital textbooks offer greater learning opportunities to academically weaker students who may have lacked educational support. Compared with traditional printed textbooks, digital textbooks offer students more equitable access to high-quality learning materials and tools that are more self-directed and interactive. Schools in socioeconomically disadvantaged areas can also prioritize the adoption of digital textbooks to offer more equitable and inclusive education to their students. Third, teachers who adopt digital textbooks can employ our findings when choosing their pedagogy styles and designing their courses. As we find that digital textbooks help foster more engaging and collaborative learning environments, teachers can utilize digital textbooks as communication channels with students, use digital textbooks to introduce more interactive in-class activities, and encourage lagging students to use digital textbooks to review materials. Relatedly, digital textbook publishers should consider including more interactive and collaborative features in their digital textbooks. Finally, students should understand the potential benefits of digital textbooks and effectively utilize them to enhance their learning experience and academic outcomes.
Future Research
Our findings can be further extended by future research in various ways. First, there has been a growing interest in using more advanced technology, such as augmented reality and virtual reality, in classrooms to help students learn through immersive experiences. It is estimated that $700 million will be invested in augmented reality and virtual reality applications in education by 2025 (Silagadze 2018). Future research on the effectiveness of other uses of technology in digital textbooks can enhance our understanding of how to implement and design a curriculum for digital natives. Second, the effectiveness of digital textbooks can potentially be further increased when complemented with high-quality teachers and pedagogy. The literature shows that teachers have significant effects not only on students’ test scores but also on their noncognitive skills and long-term lifetime income (Chetty, Friedman, and Rockoff 2014; Jackson 2018). As technology can help teachers focus on their comparative advantage, the interaction between teachers and digital textbooks is an important aspect of education that should be further studied. Finally, future research can examine the long-term effects of digital textbooks and other technology on students’ future college choices, occupations, labor productivity, and other long-term outcomes.
Supplemental Material
sj-pdf-1-mrj-10.1177_00222437221130712 - Supplemental material for The Effects of Digital Textbooks on Students’ Academic Performance, Academic Interest, and Learning Skills
Supplemental material, sj-pdf-1-mrj-10.1177_00222437221130712 for The Effects of Digital Textbooks on Students’ Academic Performance, Academic Interest, and Learning Skills by Stephanie Lee, Ju-Ho Lee and Youngsik Jeong in Journal of Marketing Research
Footnotes
Acknowledgments
The authors thank seminar participants at the Journal of Marketing Research Special Issue Conference on Education and Marketing for helpful comments. The authors also thank Sang Hoon Jee for research assistance. Ju-Ho Lee gratefully acknowledges financial support from the KDI School of Public Policy and Management. The authors thank the JMR review team for their helpful suggestions and comments.
Associate Editor
Shrihari Sridhar
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the KDI School of Public Policy and Management.
Notes
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
