Deep Reinforcement Learning for Adaptive Learning Systems

Abstract

The adaptive learning problem concerns how to create an individualized learning plan (also referred to as a learning policy) that chooses the most appropriate learning materials based on a learner’s latent traits. In this article, we study an important yet less-addressed adaptive learning problem—one that assumes continuous latent traits. Specifically, we formulate the adaptive learning problem as a Markov decision process. We assume latent traits to be continuous with an unknown transition model and apply a model-free deep reinforcement learning algorithm—the deep Q-learning algorithm—that can effectively find the optimal learning policy from data on learners’ learning process without knowing the actual transition model of the learners’ continuous latent traits. To efficiently utilize available data, we also develop a transition model estimator that emulates the learner’s learning process using neural networks. The transition model estimator can be used in the deep Q-learning algorithm so that it can more efficiently discover the optimal learning policy for a learner. Numerical simulation studies verify that the proposed algorithm is very efficient in finding a good learning policy. Especially with the aid of a transition model estimator, it can find the optimal learning policy after training using a small number of learners.

Keywords

adaptive learning system transition model estimator Markov decision process deep reinforcement learning deep Q-learning neural networks model free

Introduction

In a traditional classroom, a teacher uses the same learning material (e.g., textbook, instruction pace) for all students. However, the selected material may be too hard for some students and too easy for some other students. Further, some students may take a longer time in learning than the others. Such a learning process may not be efficient. These issues can be solved if the teacher can make an individualized learning plan for each individual student: Select an appropriate learning material according to each student’s proficiency and let a student learn at their own pace. Considering that a very high teacher–student ratio is required, such an individualized adaptive learning plan may be too expensive to be applied to all students. As such, adaptive learning systems are developed to provide individualized adaptive learning for all students/learners. In particular, with the fast growth of digital platforms, globally integrated resources, and machine learning algorithms, adaptive learning systems are becoming increasingly more affordable, applicable, and efficient (Zhang & Chang, 2016).

An adaptive learning system—also referred to as a personalized/individualized learning or intelligent tutoring system—aims at providing a learner with optimized and individualized learning experience or instructional materials so that the learner can reach a certain achievement level in the shortest time or reach as high as possible an achievement level in a fixed time period. An adaptive learning system typically works as follows. First, learners’ historical data are used to estimate their proficiency. Then, according to the level of their proficiency, the system selects the most appropriate learning material for the learner. After the learner finishes the learning material, an assessment is given to the learner and their proficiency level is updated and is used by the adaptive learning system to choose the next most appropriate learning material for the learner. Such process repeats until the learner achieves a certain proficiency level.¹ With a good adaptive learning system, teachers can focus on designing high-quality learning materials and tests without worrying about actively collecting feedback from students and selecting the most suitable materials for them. Meanwhile, such a system leverages the state-of-the-art technologies and can easily provide high-quality learning environments around the world, which in particular would benefit students in areas lacking (experienced) teachers, promoting not only efficiency but also equity in education.

In previous studies, the proficiencies, which are latent traits, were typically characterized as vectors of binary latent variables (Chen, Li, et al., 2018; Li et al., 2018; Tang et al., 2019). However, it is important to consider the granularity of the latent traits in a complex learning and assessment environment, in which a knowledge domain consists of several fine-grained abilities. In some cases, it would be insufficient to model learners’ abilities as mastery or nonmastery. For example, when a learner is answering an item designed to measure several latent traits, even if the learner is measured to have mastered all the traits individually, there is no guarantee that the learner will answer the item correctly. A possible reason is that the so-called mastery is not full mastery of a latent trait. By measuring learners’ traits as continuous scales, the adaptive learning system can be designed to help learners to learn and improve until they reach the target levels of certain abilities, so that the learners can achieve target scores in assessments. Especially in practice, most assessments are designed to measure learners’ latent traits (McGlohen & Chang, 2008). In such scenarios, it is better to use a continuous scale to measure the latent traits as the item response theory (IRT) does. In this article, we will develop an adaptive learning system that estimate learners’ abilities using measurement models in order to provide them with most appropriate materials for further improvements.

Existing studies have focused on modeling learners’ learning paths (Chen, Culpepper, et al., 2018; Wang et al., 2018), accelerating learners’ memory speed (Reddy et al., 2017), providing model-based sequence recommendation (Chen, Li, et al., 2018; Lan & Baraniuk, 2016; Xu et al., 2016), tracing learners’ concept knowledge state transitions over time (Lan et al., 2014), and selecting materials for learners optimally based on model-free algorithms (Li et al., 2018; Tang et al., 2019). However, explicit models are typically needed to characterize learners’ learning progresses in these studies. While there exist studies that aim to find the optimal learning plan/policy (the term policy is adopted throughout the rest of this article) that chooses the most appropriate learning materials for learners using model-free algorithms, they all assume discrete latent traits. In addition, when the number of learners is too small for the system to learn an optimal policy from, these algorithms are inapplicable.

This article studies the important, yet less-addressed adaptive learning problem of finding the optimal learning policy based on continuous latent traits, and applies machine learning algorithms to deal with challenges, including limitation of the number of learners in historical data. Specifically, we formulate the adaptive learning problem as a Markov decision process (MDP), in which the state is the (continuous) latent traits of a learner, and the action is the (discrete) learning material given to the learner. However, the state transition model is unknown in practice, thus making the MDP unsolvable using conventional model-based algorithms, such as the value iteration algorithm (Sutton & Barto, 2018). To solve the issue, we apply a model-free deep reinforcement learning (DRL) algorithm, the so-called deep Q-learning algorithm, to search for the optimal learning policy. The model-free DRL algorithm is a class of machine learning algorithms that solve an MDP by learning an optimal policy represented by neural networks from a sequence of state transitions directly when the transition model itself is unknown (François-Lavet et al., 2018). DRL algorithms have been widely applied in solving a variety of problems in many different fields, such as playing Atari games (Mnih et al., 2015), bidding and pricing in electricity market (Xu et al., 2019), manipulating robotics (Gu et al., 2017), and localizing objects (Caicedo & Lazebnik, 2015). We refer interested readers to François-Lavet et al. (2018) for a detailed review on the theories and applications of DRL. Therefore, the adaptive learning system is embedded with the well-developed measurement models and the model-free DRL algorithm so as to be more flexible.

However, a deep Q-learning algorithm typically requires a large amount of state transition data so as to find an optimal policy, which may be difficult to obtain in practice. To cope with the challenge of insufficient state transition data, we develop a transition model estimator that emulates the learner’s learning process using neural networks. The transition model that is fitted using available historical transition data can be used in the deep Q-learning algorithm to further improve its performance with no additional cost.

The purpose of this article is to develop a fully adaptive learning system, in which (1) the learning material given to a learner is based on their continuous latent traits that indicate the levels of certain abilities and (2) the learning policy that maps the learner’s latent traits to the learning materials is found adaptively with minimal assumptions about the learners’ learning process. First, an MDP formulation for the adaptive learning problem by representing latent traits in a continuum is developed. Second, a model-free DRL algorithm—the deep Q-learning algorithm—is applied, to the best of our knowledge, for the first time in solving the adaptive learning problem. Third, a neural network-based transition model estimator is developed, which can greatly improve the performance of the deep Q-learning algorithm when the number of learners is inadequate. Last, some interesting simulation studies are conducted to serve as demonstration cases for the development of adaptive learning systems.

The remainder of this article is organized as follows. In the Preliminaries section, we briefly review measurement models and make some assumptions on the adaptive learning problem. In the Adaptive Learning Problem section, we introduce the conventional adaptive learning systems and develop an MDP formulation for the adaptive learning problem. Then, we apply the deep Q-learning algorithm to solve the MDP in the Optimal Learning Policy Discovery section, where a transition model estimator that emulates the learners is also developed. Two simulation studies are conducted in the Numerical Simulation section and some concluding remarks are made at the end of this article.

Preliminaries

In this section, we give a brief introduction on measurement models for continuous latent traits, which is an important component in adaptive learning systems. The representation of learners’ latent traits and assumptions on them is also presented.

Measurement Models

In an adaptive learning system, a test is given to a learner/student after each learning cycle. The learner’s responses to the test items are collected by the system and their latent traits are estimated using measurement models, specifically IRT models (Lord et al., 1968; Rasch, 1960).

An appropriate IRT model needs to be chosen based on the test’s features, such as the test’s dimensional structure (Zhang, 2013) and its response categories. To be more specific, in the case when item responses are recorded as binary values indicating correct or incorrect answers, the test that evaluates only one latent trait will use the unidimensional IRT models (Birnbaum, 1968; Lord, 1980; Rasch, 1960), whereas tests that associate more than one trait will use the multidimensional IRT (MIRT) models (Mulaik, 1972; Reckase, 1972; Sympson, 1978; Whitely, 1980). When item responses have more than two categories, polytomous IRT models, such as the partial credit model (Masters, 1982), the generalized partial credit model (Muraki, 1992), and the graded response model (Samejima, 1969), are used for unidimensional case. Their extensions can be applied in multidimensional cases.

The basic representation of an IRT model is expressed as

ℙ (U = u | θ) = f (θ, η, u),

where $ℙ$ denotes probability, U is a random variable representing the score on the test item, u is the possible value of U, $θ$ is a vector of parameters describing the learner’s latent traits, $η$ is a vector of parameters indicating the characteristic of the item, and f denotes a function that maps $θ, η, and u$ to a probability in $[0, 1]$ . As pointed out in Ackerman et al. (2003), many educational tests are inherently multidimensional. Therefore, we will use the MIRT as the intrinsic model to build up the adaptive learning system. As an illustration, the multidimensional two-parameter logistic IRT model is given by

ℙ (U_{i j} = 1 | θ_{i}, a_{j}, d_{j}) = \frac{e^{a_{j}^{Τ} θ_{i} + d_{j}}}{1 + e^{a_{j}^{Τ} θ_{i} + d_{j}}},

where $U_{i j}$ is the response given by the $i th$ test taker to $j th$ item, $θ_{i} = [θ_{i 1}, θ_{i 2}, \dots, θ_{i D}]^{Τ}$ is a vector in $ℝ^{D}$ describing a set of D latent traits, a _j is a vector of D discrimination parameters for the $j th$ item, indicating the relative importance of each trait in correctly answering the item, and the intercept parameter d_j is a scalar for item j. An applicable item j takes each element of a _j to be nonnegative. Therefore, as each element’s value of $θ_{i}$ increases, the probability of correct response increases.

With an online calibration design, an accurately calibrated item bank can be acquired using previous learners’ response data for an adaptive learning system without large pretest subject pools (Makransky & Glas, 2014; Zhang & Chang, 2016). After item parameters are precalibrated, a variety of latent trait estimation methods can be applied to estimate learners’ abilities. Conventional methods such as maximum likelihood estimation (Lord et al., 1968), weighted likelihood estimation, and Bayesian methods (e.g., expected a posteriori estimation, maximum a posteriori) can accurately estimate latent traits in MIRT models. Their variations are also extended for estimating the latent traits in multiple dimensions. Many latent trait estimation methods result in a bias on the order of as small as $O (n^{- 1})$ , where n denotes the test length, while approaches that further reduce the bias as well as the variance of estimates have also been identified and proposed (Firth, 1993; Tseng & Hsu, 2001; Wang, 2015; Warm, 1989; Zhang et al., 2011).

Assumptions

Denote $θ^{(t)} = [θ_{1}^{(t)}, \dots, θ_{D}^{(t)}]^{Τ}$ as a learner’s latent traits at time step t, where D is the number of dimensions and $Τ$ indicates transpose. Throughout this article, we make the following simplified yet practical assumptions:

A1. No retrogression exists in latent traits. That is, $θ_{d}^{(t + 1)} \geq θ_{d}^{(t)}$ , $\forall d \in {1, \dots, D}$ .

A2. The number of learning materials is finite.

Adaptive Learning Problem

In this section, we first describe the adaptive learning problem and then formulate this problem as an MDP.

Problem Statement

A conventional adaptive learning system is illustrated in Figure 1. Such an adaptive learning system is typical in traditional classrooms and online courses like massive open online courses (MOOCs; Lan & Baraniuk, 2016). In an adaptive learning system, the learner takes some learning materials to improve their latent traits. After the learner finishes learning the materials, a test or homework assignment is given to the learner. Then, the learner’s latent traits are estimated. Based on the estimated latent traits, the learning system adaptively determines the next learning material for the learner, which may take one of the many forms, such as a textbook chapter, a lecture video, an interactive task, examples, exercises, and so on. Note that the same learning materials may be reused to improve the students’ understanding of a subject. In fact, this is important as in classroom teaching, and reviewing the same material happens a lot as one effective way to help students understand the knowledge better. Meanwhile, examples and exercises can be abundant for students to use as a learning material. This cyclic learning process continues until the learner’s latent traits reach or are close to a prespecified level of proficiency, that is, when the values of the latent traits $θ$ exceed certain thresholds.

Figure 1.

Conventional adaptive learning system.

The tests in an adaptive learning system can be computerized adaptive testing (CAT). The CAT is a test mode that administers tests adapted to test takers’ trait levels (Chang, 2015). It provides more accurate trait estimates with much smaller number of items (Weiss, 1982) by sequentially selecting and administering items tailored to each individual learner. Therefore, a relatively short test can assess learners’ latent traits with high accuracy.

Conventionally, the learning policy is provided by a teacher as illustrated in Figure 1. As aforementioned, however, it is too expensive for teachers to make an individualized adaptive learning policy for each learner. In this article, we use a DRL algorithm to search for an optimally individualized adaptive learning policy for each learner. The algorithm selects the most appropriate learning material among all available materials for each learner based on their provisional estimated latent traits that are obtained from their learning history and performances in tests. The adaptive selection of learning materials guarantees the learner reaches a prespecified proficiency level in a shortest number of learning cycles or reaches proficiency level as high as possible in a fixed number of learning cycles. That is, instead of resorting to an experienced teacher for the construction of a learning policy as illustrated in Figure 1, we will develop a systematic method to enable the adaptive learning system to discover an optimal learning policy from the data that have been collected, which include historical learning materials, test responses, and estimated latent traits.

MDP Formulation

Primer on MDP

Before presenting the formulation for the adaptive learning problem, we first briefly review MDPs. An MDP is characterized by a 5-tuple $(S, A, P, R, and γ)$ , where $S$ is a set of states, $A$ is a set of actions, $P$ is a Markovian transition model, $R : S \times A \times S \to ℝ$ is a reward function, and $γ \in [0, 1]$ is a discount factor² (Sutton & Barto, 2018). A transition is defined as $(s, a, r, s')$ , where $s, s' \in S$ , and $a \in A$ , $r = R (s, a, s')$ is a scalar reward when the state transitions into state $s'$ from state s after taking action a.

Let $S^{(t)}$ and $A^{(t)}$ denote the state and action at time step t, respectively, and $R^{(t)}$ denote the reward obtained after taking action $A^{(t)}$ at state $S^{(t)}$ . Note that $S^{(t)}$ , $A^{(t)}$ , and $R^{(t)}$ are random variables. When both $S$ and $A$ are finite, the transition model $P$ can be represented by conditional probability, that is

P^{(t)} (s' | s, a) = ℙ (S^{(t + 1)} = s' | S^{(t)} = s, A^{(t)} = a) .

The Markovian property of the transition model is that for any time step t

ℙ (S^{(t + 1)} | A^{(t)}, S^{(t)}, \dots, A^{(0)}, S^{(0)}) = ℙ (S^{(t + 1)} | A^{(t)}, S^{(t)}) .

Essentially, the Markovian property requires that a future state is independent of all past states given the current state. Assume $P$ is time-homogeneous, that is, for any two time steps t ₁ and t ₂

P^{(t_{1})} (s' | s, a) = P^{(t_{2})} (s' | s, a) .

Then, we can drop the superscript t and write the transition model as $P (s' | s, a)$ . Note that when $S$ is continuous, the transition model can be represented by a conditional probability density function.

Let $π : S \to A$ denote a deterministic policy for the MDP defined above. The action-value function for the MDP under policy $π$ is defined as follows:

Q^{π} (s, a) = E [\sum_{t = 0}^{\infty} γ^{t} R^{(t)} | S^{(0)} = s, A^{(0)} = a; π],

where $E$ denotes the expectation. The action-value function $Q^{π} (s, a)$ is the expected cumulative discounted reward when the system starts from state s, takes action a, and follows policy $π$ thereafter. The maximum action-value function over all policies is defined as $Q (s, a) = {max}_{π} Q^{π} (s, a)$ . A policy $π$ is said to be optimal if $Q^{π} (s, a) = Q (s, a)$ for any $s \in S$ and $a \in A$ . In particular, the greedy policy with respect to $Q (s, a)$ , defined as $π^{*} (s) = arg {max}_{a} Q (s, a)$ , is an optimal policy (Sutton & Barto, 2018). The MDP is solved if we find $π^{*}$ .

Theorem 1 (Bertsekas & Tsitsiklis, 1996): The optimal action-value function $Q (s, a)$ satisfies the Bellman optimality equation:

Q (s, a) = E [R^{(0)}] + γ \sum_{s' \in S} P (s' | s, a) max_{a' \in A} Q (s', a') .

Furthermore, there is only one Q function that solves the Bellman optimality equation.

The Bellman optimality equation is of central importance to solving the MDP. When both $S$ and $A$ are finite and $P$ is known, model-based algorithms such as the value iteration algorithm can be applied to solve the MDP (Sutton & Barto, 2018).

Adaptive Learning Problem as MDP

We next formulate the adaptive learning problem as an MDP as follows.

State Space: Define the vector of parameters describing the learner’s latent traits as the state, that is, $s = θ$ , which has D continuous variables, where D represents the dimension of the latent traits. For the simplicity of the algorithm construction in the following, the state space is defined as $S {= [0, 1]}^{D}$ when each element of $θ$ satisfies $θ \in [0, 1]$ , in which a smaller value of $θ$ indicates a lower proficiency and a larger value indicates a higher proficiency. Although a latent trait variable is typically defined on $ℝ$ in IRT, a closed interval, say [−5, 5], is used as the range of a latent trait variable in practice. This makes sense since once the latent trait variable goes beyond a certain value, the difference in the actual proficiency is negligible. Let h_d be the prespecified target proficiency level of the $d th$ latent trait, which is the level the learners try to reach, where $d = 1, \dots, D$ . Because there is a bijection between $[- 5, h_{d}]$ and $[0, 1]$ , an estimated trait $θ \in [- 5, h_{d}]$ can be directly transformed into the scale of [0, 1]. Thus, without loss of generality, we consider the state space as $S {= [0, 1]}^{D}$ .

Action Space: We can categorize learning materials into different sets, within which the materials are close in the sense that they cover the same topic/skill and have a similar level of difficulty. Let the learning material sets available in the adaptive learning system be indexed by $1, 2, \dots, L$ . The action a in the adaptive learning system is the index of a learning material set, which is discrete, and the action space is $A = {1, \dots, L}$ . When the action is i, a learning material is randomly selected from the $i th$ learning material set.

Reward Function: Recall that the objective of the adaptive learning system is to minimize the learning steps it takes before a learner’s latent traits reach the maximum, that is, for $θ$ to reach $1_{D}$ , where $1_{D}$ is an all-ones vector in $ℝ^{D}$ . As such, the reward function is defined as follows:

r = R (s, a, s') = \{\begin{array}{l} - 1, & if | | s' - 1_{D} {| |}_{\infty} \geq 10^{- 3}, \\ 0, & otherwise, \end{array}

where $| | \cdot {| |}_{\infty}$ indicates the infinite norm. Note that the prespecified level of proficiency is defined as a value that has an infinite-norm distance to $1_{D}$ that is smaller than 0.001. Intuitively, the sum of rewards over one episode (the entire learning process of a learner) is equal to the negative of the total steps a learner takes before all of their latent traits are very close to 1, which indicates that the learner has reached the target levels of all prespecified abilities. We define the total steps a learner takes to reach the target levels of all prespecified abilities as time-to-master (TTM). This TTM metric is meaningful in showing how efficient a learning process is. Note that it is equal to the negative of the sum of the rewards.

Transition Model: The probability distributions of the latent trait and the change of trait are unknown. As a result, the transition model $P (s' | s, a)$ is not known a priori.

Based on this MDP formulation, the adaptive learning problem is essentially to find an optimal learning policy, denoted by $π^{*} : S \to A$ , that determines the action (learning material selection) based on the state (latent traits), such that the expected cumulative discounted reward is maximized. Note that the larger the expected cumulative discounted reward is, the less the total learning steps a learner takes to reach the target proficiency level(s) is. Since the transition model $P$ is unknown, the MDP cannot be solved using model-based algorithms such as the value iteration algorithm. We will resort to a model-free DRL algorithm to solve it in the next section.

Optimal Learning Policy Discovery

In this section, we solve the adaptive learning problem by using the deep Q-learning algorithm, which can learn the action-value function directly from historical transition data without knowing the underlying transition model. To utilize the available transition information more efficiently, we further develop a transition model estimator and use it to train the deep Q-learning algorithm.

Action-Value Function as Deep Q-Network

Recall that the optimal learning policy can be readily obtained if we know the action-value function. When the state is continuous and the action is discrete, which is the case in the adaptive learning problem, the action-value function $Q (s, a)$ cannot be exactly represented in a tabular form. In such cases, the action-value function can be approximated by some functions, such as linear functions (Sutton & Barto, 2018) or artificial neural networks (simply referred to as neural networks, which are essentially nonlinear functions with a special structure; Mnih et al., 2015). In the former case, the approximate action-value function is represented as an inner product of the parameter vector and a feature vector that is constructed from the state. It is important to point out the choice of the features is critical to the performance of the approximate action-value function. Meanwhile, neural networks are capable of extracting useful features from the state directly and have stronger representation power than linear functions (Goodfellow et al., 2016). They have been widely used on a variety of tasks, such as image classification, natural language processing, and so on (Goodfellow et al., 2016). We refer readers who are less familiar with neural networks to the subsection “Neural Network Basics” in the Online Appendix to understand what neural networks are via a simple example. For the purpose of understanding the rest of this article, readers can simply treat a neural network as a nonlinear function with a layered structure, which takes one or more inputs, processes them in some hidden layers, and gives one or more outputs.

Recall that in the adaptive learning problem, the state is continuous in ${[0, 1]}^{D}$ , while the action is discrete $A = {1, \dots, L}$ . The approximate action-value function, denoted by $\hat{Q} (s, a)$ , can be represented using a neural network as follows. The input layer is the state s, or equivalently, the latent trait vector $θ$ , which has D units. The output has L units, each of which corresponds to the action value for one action. To more be specific, the $ℓ th$ unit in the output layer gives $\hat{Q} (s, a = ℓ)$ , that is, the action value for state s and action $ℓ$ . The number of hidden layers and the number of units in each hidden layer can be determined through simulation, which is to be detailed in the numerical simulation section. Such a neural network is also referred to as a deep Q-network (DQN; Mnih et al., 2013). Let w denote the parameter vector of the DQN, which includes all parameters in the DQN. To emphasize that $\hat{Q} (s, a)$ is parameterized by w, we write $\hat{Q} (s, a)$ as $\hat{Q} (s, a; w)$ .

Once we have $\hat{Q} (s, a; w)$ , the optimal learning policy becomes readily available, which is $π^{*} (s) = arg {max}_{a} \hat{Q} (s, a; w)$ . Then, the “Teacher” block in Figure 1 can be replaced with the DQN as shown in Figure 2.

Figure 2.

Adaptive learning system with DQN. (DQN is used as action-value function.) DQN = deep Q-network.

Learning Policy Discovery With Deep Q-Learning

The parameters of the DQN can be learned from the sequence of latent traits (states) and learning materials (actions) using the deep Q-learning algorithm proposed by Mnih et al. (2013). In order not to distract readers from grasping the main idea of the adaptive learning system we proposed, we defer technical details of this algorithm to the subsection “Deep Q Learning Algorithm” in the Online Appendix. We highlight here the key idea, which is formulating the problem of finding optimal parameters of the DQN as an optimization problem that aims to minimize the approximation error of the action-value function based on Theorem 1, given a set of state transitions extracted from learners’ learning processes. Recall that one episode represents a complete learning process of one learner. One important feature of this algorithm is that it improves the parameters of the DQN iteratively in each episode by interacting with the learner. Another important feature of this algorithm is that it needs to sufficiently explore the state-action space in order to obtain a good approximate action-value function, which is achieved by the so-called $ε$ -greedy exploration strategy.

Transition Model Estimator

The deep Q-learning algorithm requires a sufficiently large historical transition data in order to find a good approximate of the action-value function, based on which the learning policy is then derived. However, we may not be able to obtain adequate transitions due to several reasons including the lack of adequate learners (sample size) and the long time it takes to acquire an individual learner’s learning path (transitions). Thus, it is more desirable to develop an adaptive learning system, which can efficiently discover the optimal learning policy after training on a relatively small number of learners. To this end, we develop a transition model estimator, which emulates the learning behavior of learners. Specifically, the transition model estimator can take a state s and an action a as inputs and output the next state $s'$ . This can be cast as a supervised learning task (a regression task), which can be solved using neural networks. The input layer of the neural network that represents the transition model is a pair of state and action, and the output layer is the next state. The number of hidden layers can be adjusted through the parameter tuning process (see, e.g., Goodfellow et al., 2016, for more details).

Conceptually, we can write the neural network that represents the transition model as $s' = ψ (s, a)$ , the parameter vector of which is denoted by v. The optimal value of v can be found by solving the following problem using the backpropagation algorithm:

min_{v} \sum_{(s, a, r, s') \in H} | | ψ (s, a) - s {' | |}^{2},

where $H$ is the set of historical transition (data).

The adaptive learning system with the DQN and a transition model estimator is shown in Figure 3, where the DQN is trained against the transition model, instead of the actual learners.

Figure 3.

Adaptive learning system with deep Q-network and transition model estimator.

Discussion on Potential Real-World Applications

Due to the difficulty in developing a production-grade system to be used in the real world and acquiring real-world data, the scope of this work is limited to the conceptual and methodological development of an adaptive learning system with continuous latent traits and the illustration of its advantages via some numerical examples. However, we would like to discuss its potential real-world applications, which is ultimately what we aim for, before we proceed to the simulation results. As mentioned earlier, an adaptive learning system can be used to substitute part of the functionalities of teachers. Specifically, it can continuously collect feedback from students and select the most suitable materials for each student based on the student’s proficiency level, while teachers can focus on designing high-quality learning materials and tests that are fed into the adaptive learning system.

We envision the adaptive learning system could sit as the core in a software that can be used online by an online learning platform such as MOOCs. For a specific skill/topic to be learned, the latent traits are defined first. Then, their estimator is selected from existing ones that have proven effective in practice. Meanwhile, sets of learning materials need to be developed by teachers and provided to the adaptive learning system. In the initial stages of an MOOC, the assignment of learning materials is determined by a teacher as a routine that is not adapted to each student. As shown in Figure 3, the adaptive learning system could train a transition model from the limited data samples accumulated during the initial stage. Based on the estimated transition model, a deep Q-network can be further trained to determine the learning policy, which can now substitute the teacher in assigning learning materials to each student adaptively according to the student’s proficiency level on a skill/topic. The transition model and deep Q-network will be iteratively improved as more data on students’ learning process accumulates. Eventually, the transition model becomes unnecessary once sufficient data on students’ learning process have been observed, and the deep Q-network can be learned from all real state transition processes, as shown in Figure 2. Note that models are trained for each topic/skill.³ The modeling of latent traits as continuous variables and the low requirements on historical data make this adaptive learning system highly flexible. Such an adaptive learning system is very helpful to MOOCs as it can provide more personalized and efficient learning experiences to students participating in MOOCs, which ideally emulates a learning environment provided by an experienced teacher.

Numerical Simulation

In this section, we show the performance of the adaptive learning system with and without the transition model estimator and also investigate the impacts of latent trait estimation errors through two simulation studies.

Simulation Overview

Consider a group of learners in a two-dimensional assessment and a learning environment with three sets of learning materials. We model the group of learners as a homogeneous MDP. Let the random vector $Θ^{(t)} = [Θ_{1}^{(t)}, Θ_{2}^{(t)}]^{Τ}$ denote a learner’s state $S^{(t)}$ at time step t, which represents the latent traits in our study. Consider three sets of learning materials regarding the two-dimensional latent trait levels, that is, $A = {1, 2, 3}$ . Each set of learning materials contains contents with regard to different latent traits. Denote the change of the latent traits from time step t to $t + 1$ by $Δ Θ^{(t)} = [Δ Θ_{1}^{(t)}, Δ Θ_{2}^{(t)}]^{Τ}$ . The probability of having $Δ θ = [Δ θ_{1}, Δ θ_{2}]^{Τ}$ transitioning from state $θ = [θ_{1}, θ_{2}]^{Τ}$ to $θ^{'} = [θ_{1}^{'}, θ_{2}^{'}]^{Τ}$ can be represented as

P (θ^{'} | θ, a) = ℙ (Δ Θ^{(t)} = Δ θ | Θ^{(t)} = θ, A^{(t)} = a),

where a is the index of the set which the selected learning material belongs to. In the following sections, we only consider the set which the selected learning material belongs to, denoted as a. Assume $θ_{1}, θ_{2} \in [0, 1]$ , where the value of 0 indicates extremely low proficiency on the corresponding dimension and the value of 1 indicates the target proficiency.

In addition, under Assumption A1 of no retrogression, we have $Δ θ_{1} \in [0, 1 - θ_{1}]$ and $Δ θ_{2} \in [0, 1 - θ_{2}]$ . As we model the transition of the latent traits to be a continuous-state MDP, the change of ${Δθ}_{1}$ and ${Δθ}_{2}$ only depends on current latent trait $θ$ and the selected learning material set a. Therefore, we let ${Δθ}_{1}$ and $Δ θ_{2}$ follow Beta distributions, such that $Δ θ_{1} \sim B e t a (1, g_{1} (θ, a))$ , where $a \in {1, 3}$ , and $Δ θ_{2} \sim B e t a (1, g_{2} (Δ θ_{1}, θ, a))$ , where $a \in {2, 3}$ . ${Δθ}_{2} = 0$ when $a = 1$ and $Δ θ_{1} = 0$ when $a = 2$ , which means the first set of materials only helps improving $θ_{1}$ while the second set is only related to $θ_{2}$ . Parameters of $g_{1} (θ, a)$ and $g_{2} (Δ θ_{1}, θ, a)$ in the Beta distribution are calculated by

g_{1} (θ, a) = \{\begin{array}{l} 3 + 8 θ_{1} - 0.2 θ_{2}, & a = 1 \\ 15 + 15 θ_{1} - 0.4 θ_{2}, & a = 3 \end{array},

and

g_{2} (Δ θ_{1}, θ, a) = \{\begin{array}{l} 10 - θ_{1} + 5 θ_{2}, & a = 2 \\ 20 - 28 θ_{1} e^{- \frac{{(θ_{1} - 0.6)}^{2}}{0.3}} + 30 θ_{2} - 0.3 Δ θ_{1}, & a = 3. \end{array}

An intuitive example is how a learner learns addition “+” and subtraction “–.” A learning process usually takes a long time, and thus, a monotonic decreasing, zero-concentrated distribution is adopted to simulate the proficiency increase. In that case, each learning step will most likely lead to a small increase of the proficiency. Besides, in the distribution $B e t a (1, b)$ , the larger b is, the more the curve approaches 0, which results in a higher chance in generating a smaller $Δ θ$ . It implies that a higher proficiency the learner has acquired on either dimension, the harder for them to further improve the same dimension. Thus, $g_{1} (θ, a)$ has a positive coefficient in front of $θ_{1}$ and $g_{2} (Δ θ_{1}, θ, a)$ has a positive coefficient in front of $θ_{2}$ . Meanwhile, we assume that a higher proficiency on one dimension helps to increase the other dimension’s proficiency, which results in a negative coefficient ahead of $θ_{2}$ in $g_{1} (θ, a)$ and a negative coefficient ahead of $θ_{1}$ in $g_{2} (Δ θ_{1}, θ, a)$ . In particular, assume the third learning material contains contents related to both abilities and especially helps learners with intermediate or high proficiency level of addition to improve further on subtraction. This assumption is included in calculating $g_{2} (θ, a)$ when $a = 3$ in Equation 12. In addition, if the learner makes a big progress in mastering addition, there is a higher chance for the one to improve more on learning subtraction. Thus, the coefficient of ${Δθ}_{1}$ in $g_{2} (Δ θ_{1}, θ, a)$ is negative, which gives a curve that is less zero-concentrated as ${Δθ}_{1}$ increases. Consequently, ${Δθ}_{2}$ has a higher possibility in increasing more as ${Δθ}_{1}$ is large. Note that the transition model is not required for adaptive learning system. The simulation gives an example in validating the model-free deep Q-learning algorithm in discovering the optimal learning policy.

Estimation errors ranging from 1% to 15% are also added to estimated latent traits to evaluate their impacts on the adaptive learning system. Denote the estimation error vector by $e = [e_{1}, e_{2}]^{Τ}$ , where e ₁ and e ₂ are generated by the same normal distribution, such that $e_{1}, e_{2} \sim N (0, σ^{2})$ . As a result, 99.7% of $e_{1}, e_{2}$ lie in the range of $(- 3 σ,3 σ)$ . In the simulation, the estimated latent traits are calculated by the sum of the true latent traits and the estimation errors, which are ${[θ_{1} + e_{1}, θ_{2} + e_{2}]}^{Τ}$ . For instance, if the standard deviation (SD) $σ$ is 0.03, the observation is ${{[θ}_{1} + e_{1} {,θ}_{2} + e_{2}]}^{Τ}$ , where $e_{1}, e_{2} \sim N {(0, 0.03}^{2})$ , and 99.7% of $e_{1}, e_{2}$ lie in the range of (−0.09, 0.09).

Two simulations cases are studied. In the first case, the DQN is trained against actual learners whose abilities’ changes follow the MDP with the transition kernels described above. In this case, it is presumed that the optimal learning policy can be trained on sufficient number of learners. The resulting optimal learning policy is compared with a heuristic learning policy, which selects the next learning material that can improve the not-fully mastered ability, and a random learning policy which selects any material randomly from the set of three. The impact of different estimation errors is also assessed. In the second case, the DQN is trained against an estimated transition model that is obtained using a small group of learners. The resulting optimal learning policy is compared with that obtained by training against actual learners. The parameters for the deep Q-learning algorithm used in the simulation are detailed in the subsection “Simulation Parameters” in the Online Appendix.

Simulation Study I

Assume all learners are beginners on the two latent traits when using the adaptive learning system, that is, $Θ^{(0)} {= [0, 0]}^{Τ}$ . The DQN is trained against 2,000 episodes (hence 2,000 learners) that are simulated according to the method discussed earlier.

Figure 4 presents the smoothed episode reward—the sum of reward at all steps in one episode or the negative of the TTM—under the deep Q-learning algorithm across the first 1,500 episodes with a smoothing window of 20. It can be seen that the reward converges to −15 after 600 episodes, which indicates the optimal learning policy is found after the DQN is trained using 600 learners.

Figure 4.

Smoothed rewards under the deep Q-learning algorithm.

Figure 5 and Table 1 compare smoothed rewards across 200 new learners, labeled as episodes in Figure 5, with a smoothing window of 20 between the optimal learning policy found by the deep Q-learning algorithm after being trained in 2,000 episodes—referred to as the DQN learning policy, the heuristic learning policy, and the random learning policy. The larger the reward is, the fewer learning steps a learner takes to fully master the two latent traits, or in another word, the better the learning policy is. Clearly, the rewards obtained by the deep Q-learning algorithm have a higher mean and smaller SD than those obtained by the heuristic learning policy and the random learning policy. These results show that the learning policy found by the deep Q-learning algorithm is much better than the other two.

Figure 5.

Smoothed rewards under deep Q-network, heuristic, and random learning policies.

Table 1.

Comparison of Deep Q-Network (DQN), Heuristic, and Random Learning Policies

Time-to-Master	DQN	Heuristic	Random
Mean	13.49	21.55	24.85
SD	4.59	4.76	5.59

Figure 6 presents an example of two state transition paths that show how the latent traits change with a sequence of actions taken under the DQN learning policy obtained without considering estimation error. In Path 1, the initial state is (0, 0), and the optimal action sequence is 1,1,1,1,3,3,3,3,3,2,2,2,2,2,2. Meanwhile, in Path 2, the initial state is (0.5, 0.5) and the optimal action sequence is 2,2,2,2,1,1,3,3. Take the addition and subtraction test as an example. The first learning material is repeatedly selected to improve the learner’s proficiency of addition at the beginning. Then, the third material related to both addition and subtraction is selected. In the last few steps, the second learning material is chosen to further improve the learner’s proficiency of subtraction.

Figure 6.

State transition path examples with different initial states.

Figure 7 compares rewards under the DQN and the heuristic learning policies when estimation errors with various SDs ( $σ$ ) exist. It shows that the mean rewards obtained by the DQN learning policy under various estimation errors are consistently higher than those of the heuristic learning policy when estimation errors exist. That is, the DQN learning policy still outperforms the heuristic learning policy even in the presence of estimation errors, which demonstrates that the deep Q-learning algorithm is reliable and stable in finding the optimal learning policy in the presence of estimation errors.

Figure 7.

Comparison of rewards under deep Q-network and heuristic learning policies with various estimation errors.

Simulation Study II

Next, we show the performance of the adaptive learning system with a transition model estimator, which is represented using a neural network with one hidden layer that has 32 units. The prediction accuracy indices are presented in Table 2. The train and test scores are defined as the coefficient of determination in the training and test sets respectively, calculated by

1 - \frac{\sum_{i = 1}^{H} | | s^{(i)} - {\hat{s}}^{(i)} {| |}^{2}}{\sum_{i = 1}^{H} | | s^{(i)} - \bar{s} {| |}^{2}},

where s is the true state, $\bar{s}$ is the average value of the true state, $\hat{s}$ is the predicted state using previous state and the action taken, and H is the number of the transitions. The best possible score is 1. The root mean square error is calculated by

R M S E = \sqrt{\frac{\sum_{i = 1}^{H} | | s^{(i)} - {\hat{s}}^{(i)} {| |}^{2}}{H}} .

Table 2.

Accuracy of Transition Model Trained Against Various Numbers of Learners

No. of Learners	10	20	30	40	50	100	150	200	2,000
Train score	.96	.97	.97	.97	.97	.97	.97	.97	.97
Test score	.95	.97	.96	.96	.97	.97	.97	.97	.97
Root mean square error	.11	.08	.09	.09	.08	.08	.08	.08	.08

A DQN is trained on 2,000 episodes against the estimated transition model that is fitted using a certain number of actual learners; the learning policy corresponding to this DQN is referred to as the virtual DQN learning policy. For the purpose of comparison, another DQN is trained on the same number of actual learners; the learning policy corresponding to this DQN is referred to as the actual DQN learning policy. Essentially, these two learning policies differ in the way how the same set of actual learners is utilized. The actual learners are simulated according to the method discussed in “Simulation Overview” section and are used to train the actual DQN learning policy directly. In contrast, these actual learners are used to first fit a transition model, which is then used to train the virtual DQN learning policy; this allows the virtual DQN learning policy to be trained over as many episodes as it needs. Figure 8 compares rewards obtained by the two DQN learning policies when various numbers of actual learners are utilized. It is shown that with no more than 200 actual learners, the utilization of the transition model can significantly improve the performance of the learning policy, generating much larger mean rewards compared than the algorithm without using the transition model. When the number of learners is large enough, both two approaches found optimal learning policies and yield similar rewards.

Figure 8.

Comparison of rewards under actual and virtual deep Q-network learning policies.

Concluding Remarks and Future Directions

In this article, we developed an MDP formulation for an adaptive learning system by describing learners’ latent traits as continuous instead of simply classifying learners as mastery or nonmastery of certain skills. The objective of the system is to improve learners’ abilities to the prespecified target levels. We developed a deep Q-learning algorithm, which is a model-free DRL algorithm that can effectively find the optimal learning policy from data on learners’ learning process without knowing the transition model of the learner’s latent traits. To cope with the challenge of insufficient state transition data, which may result in a poor performance of the deep Q-learning algorithm, we developed a transition model estimator that emulates the learner’s learning process using neural networks, which can be used to further train the DQN and improve the its performance.

The two simulation studies presented in the article verified that the proposed methodology is very efficient in finding a good learning policy for adaptive learning systems without any help from a teacher. The optimal learning policy found by the DQN algorithm outperformed the heuristic and random methods with much higher rewards, or equivalently, much fewer learning steps/cycles for learners to reach the target levels of all prespecified abilities. Particularly, with the aid of a transition model estimator, the adaptive learning system can find a good learning policy efficiently after training using a few learners.

The directions for extending the adaptive learning research include applying the adaptive learning system on actual learners to further assess the efficiency of the proposed methodology. Both the DQN algorithm and the transition model estimator can be adopted and evaluated through real data analysis on an online learning platform. Second, to design an adaptive learning system excluding the component of the latent trait estimator could also be interesting. The adaptive learning system in this article consists of a latent trait estimator, which uses measurement models to estimate latent traits. However, the latent trait estimator will for sure introduce errors due to the nature of estimation methods. Some studies instead constructed the system assuming that learning materials influence learners’ responses to test items directly, without the latent trait estimator being incorporated (Lan & Baraniuk, 2016; Lan et al., 2014). Following this direction, a learning process could be modeled and traced directly and model-free algorithms could be proposed to find the optimal learning policy. Third, because each group of learners is assumed to follow a homogeneous MDP, future works can be conducted to classify learners into groups before they use the adaptive learning system in order to find the optimal learning policy for each group. Finally, future studies can include the modeled learning paths (Chen, Culpepper, et al., 2018; Wang et al., 2018) as learners’ historical data to search the optimal learning policy more efficiently.

Supplemental Material

Supplemental Material, sj-docx-1-jeb-10.3102_10769986221129847 - Deep Reinforcement Learning for Adaptive Learning Systems

Supplemental Material, sj-docx-1-jeb-10.3102_10769986221129847 for Deep Reinforcement Learning for Adaptive Learning Systems by Xiao Li, Hanchen Xu, Jinming Zhang and Hua-hua Chang in Journal of Educational and Behavioral Statistics

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Xiao Li

Notes

References

Ackerman

T. A.

Gierl

M. J.

Walker

C. M.

(2003). Using multidimensional item response theory to evaluate educational and psychological tests. Educational Measurement: Issues and Practice, 22(3), 37–51.

Bertsekas

D. P.

Tsitsiklis

J. N.

(1996). Neuro-dynamic programming (Vol. 5) . Athena Scientific Belmont.

Birnbaum

(1968). Some latent trait models and their use in inferring an examinee’s ability. In Lord

F. M.

Novick

M. R.

(Eds.), Statistical theories of mental test scores (pp. 397–479). Addison-Wesley.

Caicedo

J. C.

Lazebnik

(2015). Active object localization with deep reinforcement learning. Proceedings of the IEEE International Conference on Computer Vision (pp. 2488–2496).

Chang

H.-H.

(2015). Psychometrics behind computerized adaptive testing. Psychometrika, 80(1), 1–20.

Chen

Culpepper

S. A.

Wang

Douglas

(2018). A hidden Markov model for learning trajectories in cognitive diagnosis with application to spatial rotation skills. Applied Psychological Measurement, 42(1), 5–23.

Chen

Liu

Ying

(2018). Recommendation system for adaptive learning. Applied Psychological Measurement, 42(1), 24–41.

Firth

(1993). Bias reduction of maximum likelihood estimates. Biometrika, 80(1), 27–38.

François-Lavet

Henderson

Islam

Bellemare

M. G.

Pineau

(2018). An introduction to deep reinforcement learning. Foundations and Trends® in Machine Learning, 11(3–4), 219–354.

10.

Goodfellow

Bengio

Courville

. (2016). Deep learning. MIT press.

11.

Holly

Lillicrap

Levine

(2017). Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. 2017 IEEE International Conference on Robotics and Automation (ICRA) (pp. 3389–3396).

12.

Lan

A. S.

Baraniuk

R. G.

(2016). A contextual bandits framework for personalized learning action selection. Proceedings of the 9th International Conference on Educational Data Mining, Raleigh, NC: EDM (pp. 424–429).

13.

Lan

A. S.

Studer

Baraniuk

R. G.

(2014). Time-varying learning and content analytics via sparse factor analysis. Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 452–461).

14.

Zhang

Chang

H. H.

(2021). Optimal hierarchical learning path design with reinforcement learning. Applied Psychological Measurement, 45(1), 54–70.

15.

Lord

F. M

. (1980). Application of item response theory to practical testing problems. Lawrence Eribaum Associates.

16.

Lord

F. M.

Novick

M. R.

Birnbaum

. (1968). Statistical theories of mental test scores. 1968. Reading: Addison-Wesley Google Scholar.

17.

Makransky

Glas

C. A.

(2014). An automatic online calibration design in adaptive testing. Journal of Applied Testing Technology, 11(1), 1–20.

18.

Masters

G. N.

(1982). A Rasch model for partial credit scoring. Psychometrika, 47(2), 149–174.

19.

McGlohen

Chang

H.-H.

(2008). Combining computer adaptive testing technology with cognitively diagnostic assessment. Behavior Research Methods, 40(3), 808–821.

20.

Mnih

Kavukcuoglu

Silver

Graves

Antonoglou

Wierstra

Riedmiller

. (2013). Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 . https://arxiv.org/abs/1312.5602

21.

Mnih

Kavukcuoglu

Silver

Rusu

A. A.

Veness

Bellemare

M. G.

Graves

Riedmiller

M. A.

Fidjeland

A. K.

Ostrovski

Petersen

Beattie

Sadik

Antonoglou

King

Kumaran

Wierstra

Legg

Hassabis

(2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529.

22.

Mulaik

. (1972). A mathematical investigation of some multidimensional Rasch models for psychological tests. Annual Meeting of the Psychometric Society, Princeton, NJ.

23.

Muraki

. (1992). A generalized partial credit model: Application of an EM algorithm. ETS Research Report Series, 1992(1), i–30.

24.

Rasch

(1960). Probabilistic models for some intelligence and attainment tests. Danish Institute for Educational Research.

25.

Reckase

M. D

. (1972). Development and application of a multivariate logistic latent trait model. [Doctoral dissertation, Syracuse University]. Dissertation Abstracts International, 33.

26.

Reddy

Levine

Dragan

. (2017). Accelerating human learning with deep reinforcement learning. Nips’17 Workshop: Teaching Machines, Robots, and Humans.

27.

Samejima

(1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika, 34(Suppl 1), 1–97.

28.

Sutton

R. S.

Barto

A. G

. (2018). Reinforcement learning: An introduction. MIT press.

29.

Sympson

J. B

. (1978). A model for testing with multidimensional items. Proceedings of the 1977 Computerized Adaptive Testing Conference.

30.

Tang

Chen

Liu

Ying

(2019). A reinforcement learning approach to personalized learning recommendation systems. British Journal of Mathematical and Statistical Psychology, 72(1), 108–135.

31.

Tseng

F.-L.

Hsu

T.-C

. (2001). Multidimensional adaptive testing using the weighted likelihood estimation: A comparison of estimation methods. Annual Meeting of NCME, Seattle.

32.

Wang

(2015). On latent trait estimation in multidimensional compensatory item response models. Psychometrika, 80(2), 428–449.

33.

Wang

Yang

Culpepper

S. A.

Douglas

J. A.

(2018). Tracking skill acquisition with cognitive diagnosis models: A higher-order, hidden Markov model with covariates. Journal of Educational and Behavioral Statistics, 43(1), 57–87.

34.

Warm

T. A.

(1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54(3), 427–450.

35.

Weiss

D. J.

(1982). Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement, 6(4), 473–492.

36.

Whitely

S. E.

(1980). Multicomponent latent trait models for ability tests. Psychometrika, 45(4), 479–494.

37.

Sun

Nikovski

Kitamura

Mori

Hashimoto

. (2019). Deep reinforcement learning for joint bidding and pricing of load serving entity. IEEE Transactions on Smart Grid, 10(6), 6366–6375.

38.

Xing

Van Der Schaar

(2016). Personalized course sequence recommendations. IEEE Transactions on Signal Processing, 64(20), 5340–5352.

39.

Zhang

(2013). A procedure for dimensionality analyses of response data from various test designs. Psychometrika, 78(1), 37–58.

40.

Zhang

Xie

Song

(2011). Investigating the impact of uncertainty about item parameters on ability estimation. Psychometrika, 76(1), 97–118.

41.

Zhang

Chang

H.-H.

(2016). From smart testing to smart learning: How testing technology can assist the new generation of education. International Journal of Smart Technology and Learning, 1(1), 67–92.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.27 MB