Personalized exercise recommendation algorithm combining learning objective and assignment feedback

Abstract

Intelligent exercise recommendation is a research focus in the field of online learning that can help learners quickly find exercises suitable for them from the exercise bank. However, exercise recommendation differs from product or film recommendation because of some special requirements. First, the recommended exercises must cover all knowledge points geared toward the learning objective of the learner. Second, the difficulty of exercises must match the knowledge level of the target learner. In response to the above requirements, this study proposes an exercise recommendation algorithm that integrates learning objective and assignment feedback. This algorithm considers not only the coverage of knowledge points but also the knowledge level of learners to help them find highly suitable exercises. According to this algorithm, the learning objective of the learner must be initially identified to obtain a course knowledge set that suits his/her learning objective. Second, the understanding of the learner about the knowledge set must be judged based on the assignment feedback. Third, suitable exercises are recommended based on the knowledge level of the learner and the course knowledge structure. The proposed algorithm is experimentally verified by using a real-world dataset and by comparing it with other algorithms. The experimental results show that the proposed algorithm significantly outperforms the other algorithms in both precision and recall. Based on these results, the proposed algorithm can achieve an excellent recommendation performance.

Keywords

Recommender systems online learning assignment feedback course knowledge set

1 Introduction

With the acceleration of education informationalization, the Internet has been playing an increasingly important role in learning activities. However, the mass learning resources often disorient learners. For example, learners often have to spend much time searching for resources (such as exercises) that serve their learning needs. This phenomenon leads to the so-called “information overload.” Meanwhile, learners expect more from online learning systems apart from feedback on their test scores. Specifically, they want to receive better learning services based on their assignment feedback. Under this background, intelligent exercise recommendation has emerged to supplement teaching and promote the development of education in a highly targeted way. The US promulgated the No Child Left Behind Act of 2001, which stipulates that American schools must give parents, teachers, and students with feedback on all tests that they carry out. Besides, as mentioned above, the Internet offers too many learning resources. Thus, examining how to recommend learning resources that are geared toward the learning status, behavior, and objective of learners is crucial [1].

As the most effective tools for filtering information, recommendation systems [2] can satisfactorily solve the problem of information overload. The implementation of these systems has generated favorable effects in many fields. Many e-commerce platforms, most notably Netflix.com, Amazon.com, and Taobao.com, recommend products based on user interest. However, the recommendation of online learning exercises differs from traditional recommendations, such as product or film recommendations. Apart from user interest, exercise recommendation must consider two other issues, namely, (1) whether the recommended exercises can cover all required knowledge points, and (2) whether the difficulty of the recommended exercises is in line with the knowledge level of the learner. The existing exercise recommendation algorithms are mainly developed on the basis of traditional ones [3 –8] or according to cognition diagnosis [9]. To be specific, various cognition diagnosis models [10 –13] combining cognition psychology and psychometrics are used to diagnose the current cognition status and recommend exercises according to the cognition diagnosis results. Although proven to affect exercise recommendation, the existing algorithms have failed to address the above two issues. Thus, this study aims to propose a new exercise recommendation algorithm that combines learning objective and assignment feedback. This algorithm not only considers the coverage of knowledge points and the knowledge level of the learner but also maintains the course knowledge structure in order to efficiently recommend personalized exercises for the learner.

2 State of the art

The existing exercise recommendation algorithms can be divided into two types.

The first type is based on conventional recommendation algorithms, such as collaborative filtering (CF), and has been widely investigated by scholars. In [3], CF is adapted in the recommender systems of e-learning. In [4], a new material recommender system framework based on CF is proposed, and the proposed method outperforms the other algorithms on all classification accuracy measures and can accurately satisfy the learning preferences of the learner according to real-time and up-to-date contextual information. In [5], Wu et al. proposed a migration bagging exercise recommendation algorithm based on multiple classifiers. Their algorithm examines the issue of exercise recommendation under the framework of transfer learning and model classification. The abundant amount of historical information in the auxiliary domain can help recommend exercises for the target users that do not have any historical information. The algorithms of this type are mainly based on traditional product recommendation algorithms. Although simple and efficient, these algorithms do not consider the coverage of knowledge points, the hierarchical structure of knowledge points, or the knowledge level of learners.

The second type is based on the learning capability of learners. These algorithms have been studied by representative scholars in the field. For instance, in [6], Pavel et al. proposed a time-limited exercise recommendation algorithm that considers the time interval of the same exercise recommended for a learner and then recommends exercises based on the knowledge level of the learner and the difficulty of exercises. In [7], a system that recommends appropriate massive open online courses (MOOCs) according to the request of the learner is proposed. Using the case-based reasoning approach and the information retrieval technique, this system proposes the most appropriate MOOCs for the learner that fit her/his request based on his/her profile, needs, and knowledge. In [9], a personalized exercise recommendation algorithm is proposed based on cognition diagnosis. This algorithm first clarifies the learner’s grasp of personalized knowledge points through the cognition diagnosis model. Afterward, this algorithm adopts probabilistic matrix factorization to factorize the test score matrix of the learner into his/her characteristic matrix and test exercise characteristic matrix for the sake of accurately predicting his/her test performance. Some exercises that can be correctly performed by the learner are then recommended based on his/her knowledge. Although these algorithms focus on the learning capability of learners, they all ignore other information, such as the course knowledge points.

Some scholars have also examined exercise recommendation based on the hierarchical structure of knowledge points. For example, in [8], Jiang et al. proposed a personalized exercise recommendation algorithm based on the knowledge point hierarchy chart. In this algorithm, the knowledge point hierarchical structure is initially built by experts. Afterward, the weight of knowledge points is computed one by one. The knowledge point error rate matrix of a learner is obtained in view of his/her test results, and then the error rate of candidate exercises for the learner is predicted. Those exercises with a high degree of recommendation are eventually recommended to the learner. Nevertheless, this algorithm ignores the knowledge level of the learner and do not consider other information, such as course knowledge points.

In sum, although the above scholars have produced fruitful results in the field of exercise recommendation, they have all failed to address two issues, namely, 1) whether the recommended exercises cover all the required knowledge points of a course, and 2) whether the difficulty of the recommended exercises matches the current knowledge level of the learner. To address this research gap, this study proposes an exercise recommendation algorithm that integrates both the learning objective of the learner and the assignment feedback. By considering both the coverage of knowledge points and the knowledge level of the learner, this algorithm aims to help learners find suitable exercises. In this algorithm, the learning objective of the learner must be identified first to obtain the course knowledge set that suits his/her learning objective. Afterward, based on assignment feedback, the learner’s grasp of the knowledge set can be judged immediately. Suitable exercises are eventually recommended to the users based on the knowledge level of learners and the course knowledge structure.

The rest of this paper is organized as follows. Section 3 introduces the exercise recommendation algorithm that combines learning objective and assignment feedback. Section 4 verifies the proposed algorithm through contrastive experiments. Section 5 concludes the paper.

3 Methodology

3.1 Definitions

3.1.1 Course knowledge set

Different knowledge points are interconnected [14], and interconnected knowledge points constitute a learning unit or a course. A knowledge set embraces all knowledge points of a course and all connections among these knowledge points. Figure 1 shows the knowledge set of a triangle in primary school mathematics. Introduced by American scholars Novak and Gowin in 1965, a concept map [15] can vividly express a series of concepts and their connections in the proposition network.

Fig.1

Course knowledge set.

The course knowledge set and other relevant concepts are defined as follows.

Definition 1. Knowledge point

A knowledge point is an independent unit that transfers teaching information during the process of teaching activities. This unit is represented by k. The granularity of different knowledge points may vary and may be either a definition or a theoretical system. Experts are invited to divide the knowledge points. The relationships among knowledge points can be classified into inclusion, pre-order, and post-order.

Definition 2. Inclusion

If the connotation of a knowledge point k_j is included in that of another knowledge point k_i, then the knowledge point k_i includes k_j, which can be written as k_i ⊇ k_j.

For example, the knowledge point, “attributes of a triangle,” covers the knowledge point about the physical attributes of a triangle. In this sense, inclusion reflects the hierarchical structure of knowledge points.

Definition 3. Pre-order and post-order

If a knowledge point k_i is built on another knowledge point k_j, then k_j can be derived by. In other words, if a learner must first understand k_ibeforek_j, then the relationship between k_i and k_j in terms of their learning order can be written as k_i → k_j, where k_i represents the pre-order knowledge of k_i while k_j denotes the post-order knowledge of k_i

When ¬ ∃ k_g (k_i → k_g) ∧ (k_g → k_j) is substantiated, then k_i is the direct pre-order knowledge of, while k_j is the direct post-order knowledge of k_i. The relationship between these two can be written as k_ixrightarrowdk_j.

The three relationships— inclusion, pre-order, and post-order— have the following attribute (Attribute 1):

Attribute 1: A knowledge point set K constitutes the partial order relationship on the inclusion relationship set RI and on the pre-order/post-order relationship set RP. In other words, both <K, RI> and <K, RP> are partially ordered sets.

Proof. According to Definition 3, the inclusion relationship RI demonstrates three attributes, namely, reflectivity, anti-symmetry, and transfer, on K. Therefore, RI is partially ordered on set K, while RP is partially ordered on set K.

Based on the pre-order and post-order relationship between knowledge points, a directed acyclic graph (DAG) can be extracted from a course knowledge set. According to the DAG, all the potential topological collating sequences can be obtained where every sequence corresponds to a learning path.

Definition 4. Course knowledge set

A course knowledge set is defined as a triple, C = (K, RI, RP), where K denotes the set of all knowledge points of a course, RI denotes a set of inclusion relationships on k, and RP denotes the set of pre-order and post-order relationships on K.

Figure 1 ows part of the triangle in the course knowledge set.

Definition 5. Exercise

Exercise is the smallest unit of an assignment. An exercise includes the exercise number, point, exercise type, difficulty coefficient, knowledge sets tested, predicted time for finishing exercise, and distinction degree.

Definition 6. Assignment

An assignment unit is represented as the set of its global blocks where each block is represented by its set of indexed keywords [16].

In this study, Assignment is defined as a set of exercises for examining, consolidating, or improving a learner’s grasp of a or some knowledge points. This set is represented by J. An assignment comprises exercises and inherits the attributes of various exercises.

Assume that an assignment consists of m exercises. In this case, the assignment can be written into a set J = {t₁, t₂, …, t_m}. n knowledge points, namely, {k₁, k₂, …, k_n}, are then examined. The correlation between exercise t_i and knowledge point k_j can be written as r_ij. If r_ij = 1, then the exercise t_i tests the knowledge point k_j; otherwise, r_ij = 0. Based on this relationship, the exercise– knowledge point incidence matrix can be written as R = [r_ij] _M×N.

An assignment has the following attribute:

Attribute 2: The knowledge point set tested by the assignment is a union set of the knowledge points tested by the exercises in the assignment.

Proof. Given that an assignment comprises several exercises, the set of knowledge points tested by an assignment can be obtained by the joint of the set of knowledge points that are tested by the exercises.

Definition 7. Assignment feedback

Assignment feedback refers to the process of judging the cognition of a learner based on the assignment finished by the learner. This process involves the following steps:

Step 1. After a learner submits his/her assignment for review, an exercise– point matrix R is formed as follows: $R = (\begin{matrix} r_{11} & \dots & r_{1 m} \\ ⋮ & ⋱ & ⋮ \\ r_{n 1} & \dots & r_{nm} \end{matrix}),$ (1) where r_ij represents the points that learner i obtains by finishing exercise j. In this case, the exercise– point ratio matrix G can be defined [9]. The exercise– point ratio is the ratio of the actual points obtained by the learner to the full points of an exercise. $G = (\begin{matrix} g_{11} & \dots & g_{1 m} \\ ⋮ & ⋱ & ⋮ \\ g_{n 1} & \dots & g_{nm} \end{matrix})$ (2)

Step 2. The exercise– knowledge point incidence matrix can be written as $Q = (\begin{matrix} q_{11} & \dots & q_{1 m} \\ ⋮ & ⋱ & ⋮ \\ q_{n 1} & \dots & q_{nm} \end{matrix}),$ (3) where q_ij = 1 means that exercise t_i tests the knowledge point k_j; otherwise, q_ij = 0.

Step 3.α_sk denotes the grasp of the knowledge point k by a learner s. Every learner s corresponds to a knowledge level vector α_s = (α_s1, α_s2, …, α_sm), where a_ik can be expressed as $α_{ik} = \frac{\sum_{j = 1}^{J} g_{ij} q_{ij}}{\sum_{j = 1}^{J} q_{jk}}$ (4) where J denotes the number of exercises.

We provide the following example to further explain the assignment feedback process. First, suppose that g_i = (0.6, 0.7, 0.8, 0.9, 1.0, 0.5, 0.9, 0.4, 0.3, 0.8) and q_k = (1, 1, 1, 1, 1, 0, 0, 0, 0, 0) ^T. Then, α_ik can be given as $\begin{matrix} α_{ik} & = & \frac{0.6 \times 1 + 0.7 \times 1 + 0.8 \times 1 + 0.9 \times 1 + 1.0 \times 1 + 0.5 \times 0 + 0.9 \times 0 + 0.4 \times 0 + 0.3 \times 0 + 0.8 \times 0}{1 + 1 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 0} \\ = & 0.8 \end{matrix}$

In other words, based on the performance of learner i on the 10 exercises, his/her grasp of the knowledge point k can be computed as 0.8.

3.2 An exercise recommendation algorithm that combines learning objective and assignment feedback

The proposed algorithm for exercise recommendation that combines learning objective and assignment feedback (ER-LOAF) mainly covers two periods (see Fig. 2).

Fig.2

ER-LOAF framework.

Period 1 (realized by Algorithm 1): A learning objective is set. The knowledge point set under the learning objective is extracted from the course knowledge set along with its inclusion, pre-order, and post-order. Afterward, a personalized knowledge set can be generated in accordance with the requirements of the learner.

Period 2 (realized by Algorithm 2): An assignment is randomly generated according to the personalized knowledge set obtained from Period 1. Assignment feedback can update the knowledge level of the learner on a real-time basis. In view of the knowledge level and learning characteristics of the learner, exercise sets suitable for the learner will be generated, and these exercise sets will eventually form a list of recommended exercises.

3.3 Algorithm design

3.3.1 Generation of a personalized knowledge set

According to the recommendation framework in Fig. 2, generating a personalized knowledge set in the first period aims to extract relevant knowledge points from the course knowledge set and to determine the connections among knowledge points based on the given learning objective. The initial knowledge level of a learner can be tested by the system in advance. A knowledge set suitable for the learner can be screened out eventually. The whole process can be realized through the following algorithm:

Algorithm 1: Generation of a personalized knowledge set

Input: The learner’s learning objective, k_s, and the course knowledge set, C = (K, RI, RP).

Output: Personalized knowledge set, C′ = (K′, RI′, RP′).

1 K′ = φ; RI′ = φ; //Define the personalized knowledge set and initialize it;

2 For i = 1 to |K|] do //Search the course knowledge set, C, based on the learning objective, k_s;

3 If <k_s, k_i > ∈ RI, then

4 K′ = K′ ∪ {k_i}; //Obtain the knowledge point set under the objective;

5 RI′ = RI′ ∪ {< k_s, k_i >}; //Inherit the original inclusion;

6 End if

7 End For

8 For j = 1 to |K′| do //Extract the pre-order and post-order relationships of the knowledge point set, K′;

9 If <k_s, k_j > ∈ RP, then

10 RP′ = RP′ ∪ {< k_s, k_j >}; //Update the pre-order and post-order relationship set;

11 End if

12 End For

13 Return C′ = (K′, RI′, RP′); //Obtain the personalized knowledge set, C′ = (K′, RI′, RP′);

The time complexity of Algorithm 1 can be written as T (n) = O (n), where n denotes the number of knowledge points of the course.

3.3.2 Exercise recommendation list

The task in the second period can be realized based on what is obtained in the first period. According to the learner’s knowledge set C′, suitable exercises are extracted from the exercise bank based on the learning objective. Afterward, an exercise recommendation list can be generated based on the personalized information of the learner.

The specific steps are outlined as follows:

Step 1. Randomly generate an assignment under the knowledge set according to the personalized knowledge set C′;

Step 2. Judge the learner’s grasp of knowledge based on the assignment feedback (refer to 3.1.2 for more details);

Step 3. Extract exercises that match the knowledge level of the learner according to the pre-order and post-order relationships of knowledge points; and

Step 4. Form the exercise recommendation list, List.

Algorithm 2: Exercise recommendation list

Input: Personalized knowledge set, C′ = (K′, RI′, RP′), exercise– knowledge point incidence matrix, Q, and exercise bank, T;

Output: Exercise recommendation list, List = {t₁, t₂, …, t_n};

1 List = φ; //Initialize the recommendation list

2 Randomly generate an assignment under the current knowledge set, C′;

3 Judge the learner’s grasp of knowledge points using the assignment feedback and obtain the learner’s initial knowledge level vector, α_u, (refer to 3.1.2 for more details);

4 For each k_i ∈ K′ do

5 Temp = φ //Initialize the temporary list;

6 Sum = 0

7 For each t_j ∈ T do

8 If q_ij = 1 and t_j. diff ≤ α_ui, then //Add the exercises connected with knowledge points and which difficulty well matches the learner’s level into the temporary list;

9 Temp = Temp ∪ {t_j}

10 Sum = Sum + 1;

11 If Sum ≥β, then //Obtain the maximum number of exercises under every knowledge point, the threshold β is predefined.

12 Exit for

13 End if

14 End If

15 End For

16 List = List ∪ Temp

17 End For

18 Return List

The time complexity of Algorithm 2 can be written as T (n) = O (n²), where n represents the number of exercises under the exercise bank.

4 Result analysis and discussion

In this section, a set of experiments have been conducted to set the parameters and examine the recommendation accuracy and quality of the proposed recommender method.

4.1 Experiment design

4.1.1 Dataset

Learning records from a real-world dataset taken from an online learning platform are applied in the experiments. The authenticity of these data can be guaranteed because they are based on the mathematical learning of Grade 4 students from February 23, 2016 to June 24, 2016. The learning objective and assignment feedback of learners are integrated to form the two-period exercise recommendation algorithm, ER-LOAF, which can recommend exercises for the learner based on his/her learning objective. The dataset contains 63 knowledge sets of mathematics that are published by the People’s Education Publishing House for Grade 4 students. A total of 753 learners and 1,056 exercises are chosen. Table 1 shows the statistical information of this dataset.

Table 1
Statistical information of the dataset

Dataset Number of learners Number of exercises Number of knowledge points

Learning records 753 1056 63

Dataset	Number of learners	Number of exercises	Number of knowledge points
Learning records	753	1056	63

4.1.2 Experimental settings

The dataset is randomly divided into the training set and test set. Under the training set, the parameters of the exercise recommendation algorithm that integrates learning objective and assignment feedback are trained. Meanwhile, under the test set, the recommendation reliability of the recommendation algorithm is tested. Precision and recall are the most popular metrics [17] in this category that have been used by various researchers. The recall and precision of recommender systems can be defined as $Pr ecision = \frac{| R \cap T |}{| R |} and$ (5) $Re call = \frac{| R \cap T |}{| T |},$ (6) where R represents the set of recommended exercises, while T represents the set of exercises (from the relevant exercises under the knowledge point) that the learner can perform correctly.

Given that increasing the size of the recommendation set will increase recall and reduce precision at the same time, we can compute the F1 measure [4], a famous combination metric, as follows: $F 1 = \frac{2 \times Pr ecision \times Re call}{Pr ecision + Re call} .$ (7)

4.1.3 Benchmark

To validate ER-LOAF, this algorithm is compared with the following recommendation algorithms:

CF [2]: According to the record of points already obtained by the learner, the cosine similarity of the points can be obtained. The points of the learner can be predicted based on his/her most similar points. The exercises are recommended based on the points ranked in a descending order.

Cognitive diagnosis (deterministic inputs, noisy, and gate [11] (DINA)): The initial learning level of a learner can be judged by using the DINA model. Based on his/her initial learning level, corresponding exercises can be recommended for the learner.

The DINA model can be expressed as $P (X_{ij} = 1 | α_{i}) = (1 - s_{j})^{η_{ij}} g_{j}^{1 - η_{ij}},$ (8) where P (X_ij = 1|α_i) is the probability for a learner (whose knowledge level is α_i) to correctly perform exercise j, g_j = P (X_ij = 0|η_ij = 1) (also called error rate) is the probability for learner i to perform the exercise incorrectly despite grasping all knowledge points tested by item j, and g_j = P (X_ij = 1|η_ij = 0) (also called guess rate) represents the probability for learner i to perform the exercise correctly despite failing to grasp all knowledge points tested by item j. η_ij judges whether learner i grasps all knowledge points tested by item j. This parameter is computed as $η_{ij} = \prod_{k = 1}^{K} α_{ik}^{q_{jk}} .$ (9)

If η_ij = 1, then learner i grasps all knowledge points tested by item j. If η_ij = 0, then learner i fails to grasp at least one knowledge point tested by the exercise. According to the “local independency hypothesis” of item response theory, the likelihood function of the DNA model can be represented as $L (s, g; α) = \prod_{i = 1}^{I} \prod_{j = 1}^{J} {p_{j} (α_{i})^{X_{ij}} [1 - p_{j} (α_{i})]^{1 - X_{ij}}},$ (10) where J denotes the number of exercises, while I denotes the number of learners.

4.2 Result analysis

Figure 3 shows that the recommendation precision decreases and the recall rate steadily increases along with a constantly increasing threshold β value (the maximum number of exercises recommended for a knowledge point) set in the experiment. During the recommendation process, when the number of exercises under every knowledge point is small, the learner has a high probability to perform the exercises correctly. Therefore, the recommendation precision is also high. Meanwhile, given that the number of recommended exercises is small, the recall rate of exercises is relatively low. However, as the value of β increases, the recall rate gradually increases. This observation is in line with practical situations. Therefore, the proposed exercise recommendation algorithm is in line with practical situations and is thereby valid and reliable. In this experiment, the value of β is set to β = 8, which can well guarantee the recommendation performance and user satisfaction of the proposed algorithm.

Fig.3

Effects of the value of β on ER-LOAF.

To conduct the contrastive experiment, 80% of learners are randomly chosen as the users of the training set, while the remaining 20% are chosen as the users of the test set. The proposed algorithm is then compared with CF and DINA by setting the proportion of exercises in the test set to 10%, 20%, 30%, and 40%. The experimental results are then compared (see Fig. 4). As shown in Fig. 4, ER-LOAF outperforms the two other algorithms in terms of precision, recall, and F1. Meanwhile, CF slightly outperforms DINA. A possible cause might be the dataset itself. Probably, under a more ordinary learning scenario, the dataset might be sparser. Under the condition, CF might show a less satisfactory performance.

Fig.4

Experimental results of different recommendation algorithms.

5 Conclusions

The existing exercise recommendation algorithms cannot cover all knowledge points of a course, and the difficulty of recommended exercises often do not match the learning level of learners. To address these problems, this study proposes a new exercise recommendation algorithm, ER-LOAF, which integrates learning objective and assignment feedback. The learning objective and knowledge level of the learner are comprehensively taken into account in this algorithm. The original characteristics of the course knowledge structure can also be maintained. The experiment adopts the real-world online learning data of Grade 4 students, and the experimental results suggest that adopting the proper parameters can help improve the recommendation performance of the recommendation algorithm.

Based on the above analysis and discussion, this study concludes the following:

Assignment feedback can diagnose the actual knowledge level of a learner, thereby ensuring the expandability of recommendations;

The coverage of knowledge points is fully considered by the exercise recommendation algorithm, thereby clarifying the learning objective of the learner; and

Several concepts, such as the course knowledge set, are introduced in order for the recommended exercises to maintain the hierarchical structure of the course knowledge.

The above conclusions can provide references for investigating other relevant issues in the field.

This study also has several limitations. When recommending exercises, the proposed algorithm presets the learning objective, which means that this algorithm cannot automatically recognize the learning objective of the learner. In the future, the author will further investigate how to recognize the learning objective of the learner automatically and update such objective on a real-time basis based on the learning behaviors and learning situations of the learner. By doing so, the author can achieve a more intelligent exercise recommendation algorithm that can further improve the learnišng efficiency and user satisfaction of the learner.

Footnotes

Acknowledgments

The authors thank the Natural Science Foundation of China (Grant Nos.: 71461013 and 61262033), the Natural Science Foundation of Jiangxi Province (Grant No.: 20132BAB201028), and the Humanities and Social Science Research in Colleges and Universities of Jiangxi Province (Grant No.: TQ1303).

References

Klasšnja-Milićević

, Ivanovć

and Nanopoulos

, Recommender systems in e-learning environments: A survey of thestate-of-the-art and possible extensions, Artificial Intelligence Review44(4) (2015), 571–604.

Adomavicius

and Tuzhilin

, Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions, IEEE Transactions on Knowledge and Data Engineering17(6) (2005), 734–749.

Bobadilla

, Serradilla

and Hernando

, Collaborative filtering adapted to recommender systems of e-learning, Knowledge-Based Systems22(4) (2009), 261–265.

Salehi

, Kamalabadi

I.N.

and Ghoushchi

M.B.G.

, Personalized recommendation of learning material using sequential pattern mining and attribute based collaborative filtering, Education and Information Technologies19(4) (2014), 713–735.

, Feng

, Sun

et al., Online transfer-bagging question recommendation based on hybrid classifiers, Journal of Computer Applications33(07) (2013), 1950–1954.

Michlík

and Bieliková

, Exercises recommending for limited time learning, Procedia Computer Science1(2) (2010), 2821–2828.

Bousbahi

and Chorfi

, MOOC-Rec: A case based recommender system for MOOCs, Procedia-Social and Behavioral Sciences (195) (2015), 1813–1822.

Zhu

T.Y.

, Huang

Z.Y.

, Chen

E.H.

et al., Cognitive Diagnosis Based Personalized Question Recommendation, Chinese Journal of Computers40(01) (2017), 176–191.

Jiang

C.F.

, Feng

, Sun

et al., Personalized Exercises Recommendation Algorithm based on Knowledge Hierarchical Graph, Computer Engineering and Applications (4) (2017), 1–9.

10.

Cai

, Tu

and Ding

, A simulation study to compare five cognitive diagnostic models, Acta Psychologica Sinica45(11) (2013), 1295–1304.

11.

De La Torre

, DINA model and parameter estimation: A didactic, Journal of Educational and Behavioral Statistics34(1) (2009), 115–130.

12.

Chen

and de la Torre

, A general cognitive diagnosis model for expert-defined polytomous attributes, Applied Psychological Measurement37(6) (2013), 419–437.

13.

Kuo

B.C.

, Chen

C.H.

, Yang

C.W.

et al., Cognitive diagnostic models for tests with multiple-choice and constructed- response items, Educational Psychology36(6) (2016), 1115–1133.

14.

Durand

, Belacel

and LaPlante

, Graph theory based model for learning path recommendation, Information Sciences (251) (2013), 10–21.

15.

Novak

J.D.

, Bob Gowin

and Johansen

G.T.

, The use of concept mapping and knowledge vee mapping with junior high school science students, Science education67(5) (1983), 625–645.

16.

Moussiades

, Discovering Clusters of Plagiarism in Students’Source Codes, Journal of Engineering Science and Technology Review9(1) (2016), 8–12.

17.

S.Y.

and Raahemi

, A Semantic-based service discovery framework for collaborative environments, International Journal of Simulation Modelling15(1) (2016), 83–96.