Improved rank discrepancy-aware sampling based on bidirectional distillation for top- K recommender system

Abstract

Nowadays, the model compression method of knowledge distillation has drawn great attentions in Recommender systems (RS). The strategy of bidirectional distillation performs the bidirectional learning for both the teacher and the student models such that these two models can collaboratively improve with each other. However, this strategy cannot effectively exploit representation capabilities of each item and lack of the interpretability for the importance of items. Thus, how to develop an effective sampling scheme is still valuable for us to further study and explore. In this paper, we propose an improved rank discrepancy-aware item sampling strategy to enhance the performance of bidirectional distillation learning. Specifically, by employing the distillation loss, we train the teacher and student models to reflect the fact that a user has partiality for the unobserved items. Then, we propose the improved rank discrepancy-aware sampling strategy based on feedback learning mechanism to transfer just the useful information which can effectively enhance each other. The key part of the multiple distillation training aims to select valuable items which can be re-distilled in the network for training. The proposed technique can effectively solve the problem of high ambiguity in nature for recommender system. Experimental results on several real-world recommender system datasets well demonstrate that the improved bidirectional distillation strategy shows better performance.

Keywords

Bidirectional distillation student-teacher learning rank discrepancy aware items selection recommender system

1 Introduction

At present, with the expanding scale of Recommendation System (RS), researchers usually adopt complicated models to understand the relationships between users and items [2 , 12]. A recommender with large learning parameters can show superior performance. However, it may suffer from high computational and long inference costs. The drawbacks of previous models show more severe for web level applications with a good many of users and items for the reason that the number of parameters can increase sharply.

To deal with the problems, some works [2 –4] have introduced Knowledge Distillation (KD) learning in the case of recommendation system. KD is combined with a mechanism that learns a compact student model under the guidance of a pretrained sophisticated teacher model. The related works can be divided into two types: unidirectional distillation (UD) and bidirectional distillation (BD) as listed in Fig. 1 (a) and (b), respectively. First, the UD in Fig. 1 (a) adopts a unidirectional information transferring. This learning strategy just transfers useful information from the teacher model to the student model based on the assumption that the teacher model is consistently better than that of the student [2, 3]. However, this learning strategy can not fully exploit the performance in the case that the student model shows better than the teacher model. Notably, in the application scenarios of recommendation system, Kweon et al [1] found that the knowledge of the student model can also help to enhance the performance of the teacher model, i.e., the teacher cannot be optimal on the testing set. The second strategy in Fig. 1 (b) is the bidirectional distillation, which can perform the collaborative learning for both the teacher and the student. Notably, the method of BD in [1] shows that some items ranked highly by the teacher can not exhibit strong ability to effectively enhance the other student model, vice versa. Considering that focusing on the high-ranked items of teacher and student will reduce the performance of knowledge distillation performance, the BD model aims at distilling useful information of items ranked relatively higher by the teacher, while along with lower rank determined by the student. However, it cannot fully exploit representation capabilities of the items and lack of the interpretability for the importance of items.

Fig. 1

Two knowledge transferred strategies, (a) Unidirectional distillation and (b) Bidirectional Distillation.

Based on the above analysis, how to develop an effective sampling scheme is still an open problem. In this paper, we propose an improved rank discrepancy-aware item sampling strategy to enhance the performance of bidirectional distillation learning. First, by employing the distillation loss, we train the teacher and student models to reflect the fact that a user has partiality for the unobserved items. Then, we propose the improved rank discrepancy-aware sampling strategy based on feedback learning mechanism to transfer just the useful information which can effectively enhance both the teacher and student models. To be note, the improved sampling framework contains the shallow distillation stage and the deep interaction with multiple distillation training stage, which can distill only the informative knowledge and find potential knowledge to exactly predict the preference on the unobserved items for a target user in a bidirectional manner. Compared with the previous unidirectional distillation learning strategy which only considers the assumption that the teacher model is invariably better than that of the student, the proposed method further adopts bidirectional distillation analysis to exploit that the student is superior than the teacher for a large part of the testing set in the case of recommender system, enabling that the teacher and the student can collaboratively improve with each other during the training. Compared with the related bidirectional distillation learning strategy, our method proposes an improved rank discrepancy-aware item sampling strategy with multiple distillation training to enhance the performance of bidirectional distillation learning such that the representation capabilities of the items and the interpretability for the importance of items can be exploited. Thus, our method can effectively solve the problem of high ambiguity in nature for recommender system.

In brief, the main contributions in this paper are provided as follows:

We develop a new sampling strategy to perform bidirectional distillation for top-K recommender system, named improved rank discrepancy-aware sampling based on Bidirectional Distillation learning (iBD).

By integrating feedback learning mechanism in the rank discrepancy-aware sampling scheme, the multiple distillation training can definitely select and make full use of the valuable items such that only the important and useful information can be distilled to effectively enhance each other.

Experimental results on several real-world recommender system datasets well demonstrate that the proposed method achieves superior performance compared with related knowledge distillation methods.

The rest of this paper is described as follows. The related works of knowledge distillation in RS are introduced in Section 2. Section 3 presents the motivation of our work, describes the optimization process of the proposed model. The effectiveness of the proposed method is experimentally evaluated in Section 4. Finally, the conclusion is provided in Section 5.

2 Related work

There are several strategies to reduce the model size and reasoning time of recommender system [5–8 , 12]. These works aim to adopt discrete quantization for users and items to generate handy recommendations and can successfully reduce the model size. Some works [2 –4] have used knowledge distillation in the field of RS. The strategy of KD can be regarded as a model-agnostic way, which can be used as the base model of RS. For example, to design effective and efficient model, the method of RD [3] uses a KD strategy such that the student with small model size can obtain a similar recommendation list compared with the teacher model. Then, considering the items ranked by the larger teacher model, CD [2] lets the student model make use of the prediction scores to enhance the recommendation performance. Recently, RRD in [4], further adopts the distillation learning as a less rigid regime of ranking matching task between the ranking list of the teacher model and the ranking list of the student model. As learning all the predictions from a teacher for student can be daunting, only focusing on the high ranked items, will affect the performance of recommendations. Through this auxiliary supervision of teachers, they can eventually enhance the performance of the student. However, some drawbacks should be noted: 1) these methods mainly focus on the strategy of unidirectional distillation, 2) the high-ranked items selected by the teacher can not provide enough information into a small and efficient student model.

Furthermore, ensemble models have been jointly trained to obtain good results in the field of computer vision/image processing tasks and natural language processing (NPL) [9 –11]. For instance, the proposed Deep Mutual Learning (DML) [9] aims to learn a series of classifiers jointly in the peer-teaching phrase. Specifically, the result obtained from other classifiers can help to train the target classifier. Also, Collaborative Learning [11] learns multiple classifier headers from the same network, which are simultaneously learned through shared mid-layer representations. Based on the principle of consistency, this model shows that using the multiple view data is definitely able to exploit the supplementary information between different views.

In the task of NPL, Dual Learning [10] argues that two different machine translators can help to teach each other in order to get rid of the time-consuming of human labeling. It is pointed out that the process of translating language can be regarded as a double line. The dual learning provides information feedback via transferring the outputs to each other, which can lead to improved performance.

3 Our method

3.1 Problem statement

Considering that the set of users is P = {p₁, p₂, . . . , p_n} and the set of items is Q = {q₁, q₂, . . . , q_m}, then, we denote M as the user-item interaction matrix as follows: $\begin{matrix} m_{p, q} = {\begin{matrix} 1 & p has an interaction with q \\ 0 & else, \end{matrix} \end{matrix}$ (1) where p is the user, q is the item. For p, $Q_{p}^{-}$ is the set of unobserved items with m = 0 and $Q_{p}^{+}$ is the item set of interacted ones. Then, we can describe that the task of a top-RS tries to obtain a recommendation list of unobserved items for each user. Specifically, for each single item p_i in $Q_{p}^{-}$ , the RS aims to assign a score m along with all the users. We can further obtain the recommendation list by ranking the items via the corresponding m.

3.2 Structure of iBD

In this section, based on the bidirectional distillation learning strategy, we propose a novel method named named improved rank discrepancy-aware sampling based on Bidirectional Distillation learning (iBD) for top-K recommender system. Iconically, the Fig. 2 gives an illustration of the proposed iBD. Notably, both the teacher and the student in our model aim to transfer their knowledge to each other. Specifically, for each user, the teacher model and the student model learn two recommendation lists involving all the items. Then, the IRDS-DB transfers the useful information via two distillation ways by integrating feedback learning mechanism in the rank discrepancy-aware item sampling strategy. At last, both the two models (teacher and student) are iteratively trained under the guide of the collaborative filtering loss and the distillation losses from teacher to student and student to teacher. Thus, it can definitely select and make full use of the noise items such that only the important and useful information can be distilled to effectively enhance each other.

Fig. 2

Block diagram of the proposed method. Based on the distillation loss of BD, in the first shallow distillation stage, we train the teacher and student models to reflect the fact that a user has partiality for the unobserved items. The recommendation lists m1 and m2 for user p is obtained. Then, in the second multiple distillation stage, we obtain the recommendation list matrices M1 and M2. We check which items associated with the user can lead to a rank decline. Thus, these selected items labeled as noise ones are re-distilled to the network for training based on feedback learning mechanism. Notably, this framework can definitely select and make full use of the noise items such that only the important and useful information can be selected to fully enhance each other.

3.3 The joint learning loss

In this section, to obtain an effective student model with smaller size, we describe the joint learning loss inspired by the bidirectional distillation learning strategy [1], which can be listed as follows: $\begin{matrix} Φ_{T} (α_{T}) = Φ_{CF} (α_{T}) + γ_{S \to T} \cdot Φ_{BD} (α_{T}; α_{S}), \\ Φ_{S} (α_{S}) = Φ_{CF} (α_{S}) + γ_{T \to S} \cdot Φ_{BD} (α_{S}; α_{T}), \end{matrix}$ (2) where T is the teacher model and S is the student model, respectively, α_T denotes the parameter of the teacher learning model and α_S is the parameter of the student learning model. Φ_CF is the collaborative filtering loss, which can be considered as the base model represented by any recommender model and Φ_BD denotes the BD loss by integrating the feedback learning mechanism. And the γ_S→T and γ_T→S are two hyperparameters which are used to balance the distillation learning parts of Φ_BD (α_T ; α_S) and Φ_BD (α_S ; α_T), respectively. The bidirectional distillation loss [1] for the user p can be defined as followed: $\begin{matrix} Φ_{BD} (α_{T}; α_{S}) \\ = \sum_{j \in Ω_{S \to T} (Q_{p}^{-})} Φ_{BCE} (ρ (m_{pj}^{T} = 1 | p, j), \\ ρ (m_{pj}^{S} = 1 | p, j)), \\ Φ_{BD} (α_{S}; α_{T}) \\ = \sum_{j \in Ω_{T \to S} (Q_{p}^{-})} Φ_{BCE} (ρ (m_{pj}^{S} = 1 | p, j), \\ ρ (m_{pj}^{T} = 1 | p, j)), \end{matrix}$ (3) where Φ_BCE denotes the widely used binary cross-entropy loss, ρ (m_pj = 1|p, j) can be regarded as the prediction of a recommender, $Ω_{*} (Q_{p}^{-})$ denotes the items set containing the unobserved samples by employing the sampling strategy. Under the distillation learning strategy, each recommender definitely obtain the other’s useful information of the positive and negative reference for the use to a ambiguous item. Eventually, it can enhance the learning performance for the recommender system.

3.4 Improved rank discrepancy-aware item selection

In this part, we describe an effective rank discrepancy-aware sampling strategy based on feedback learning mechanism to distill just the informative information such that the models can enhance each other in a bidirectional manner. Formally, some previous learning strategies [2, 3] simply select the high-ranked items in unidirectional distillation leading to undesired performance. Notably, considering the bidirectional distillation learning strategy, the performance gap between two models for teacher and student needs to be taken into account carefully to obtain the transferred knowledge, for the reason that not all knowledge are useful to enhance the performance. For the item i, the teacher model assigns the rank ${rank}_{T}^{p} (i)$ for i to a user p and the student model shows the rank ${rank}_{S}^{p} (i)$ for i to a user p. Specifically, rank (i) =1 denotes the highest rank. Then, we introduce the improved rank discrepancy-aware item selection strategy, which has the following two stages:

1) The shallow distillation training

Under the shallow distillation stage, the bidirectional learning strategy is considered to obtain the rank list for users, which can be described as follows.

Distillation direction: teacher to student

We select the item which is ranked with a higher assessment value by the teacher model while with a lower assessment value ranked according to the student model by: $\begin{matrix} ρ ({rank}_{S} (i) - {rank}_{T} (i)), \end{matrix}$ (4) with this probability representation in Equation (4), we can obtain the items with high teacher ranking and low student ranking. Therefore, the student obtains the teacher’s rich and useful knowledge based on the rank-discrepant items. In other word, the teacher model helps to enhance the student model.

Distillation direction: student to teacher

Based on the assumption that the student is superior than the teacher for a large part of the testing set, the student model can also help to improve the teacher model. Different from the distillation direction in teacher to student, we select the item which is ranked with a higher assessment value by the student model while with a lower assessment value ranked according to the teacher model by, $\begin{matrix} ρ ({rank}_{T} (i) - {rank}_{S} (i)), \end{matrix}$ (5) similarly, based on the probability function in Equation (5), the items ranked highly by the student model while ranked lowly by the teacher model are sampled. Definitely, Equation (5) encourages the teacher to learn useful information from student to enhance itself.

2) The deep interaction with multiple distillation training

According to Equation (4) and Equation (5), in the training of shallow distillation learning, each recommender model gets informative knowledge by learning on the items ranked highly by the other model, but ranked lowly by itself. Thus, for each user p, we can obtain two modules to record the ranking of teacher and the ranking of student as follows: recommendation list m1 and recommendation list m2.

m1: Teacher’s recommendation former list for user p.

m2: Student’s recommendation former list for user p.

Considering that the shallow distillation learning does not always precisely reflect the fact that a user has partiality for the unobserved items due to the noise items. With multiple distillation training based on the above shallow distillation, we can obtain two matrices to record the ranking of teacher and the ranking of student: recommendation list matrix M1 and recommendation list matrix M2, where m1∈M1 and m2∈M2.

Then, we need to check which items associated with the user p in M1 and M2 can lead to a rank decline. These selected items labeled as noise items are re-distilled to the network for training based on feedback learning, for the reason that these items provide more information to interpret that the user actually dislikes the item or potentially likes the item. Actually, the items with the ranking changed much will be filtered to enhance the effectiveness of bidirectional distillation strategy.

Algorithm 1 provides a training process of the proposed method iBD.

4 Experiment

4.1 Experimental data

We use three real-world datasets to conduct experiments.

CiteULike [13] has 5,219 users, 25187 items.

Foursquare-TYO [14] contains 1939 users, 61,858 items.

Foursquare-NYC [14] contains 824 users, 38336 items.

Algorithm 1 iBD
Input: : Training sample set D, the number of total epochs e, and improved rank updating period v.

Output: : α_T ,α_S.

  repeat

   Each rank updating period v: Teacher and Student update their recommendation lists.

   Train Teacher

   Step 1 Select the items from the teacher to the student based on feedback learning mechanism.

   Step 2 Compute the loss in Equation (2).

   Step 3 Update α_T

   Train Student

   Step 1 Select the items from the student to the teacher based on feedback learning mechanism.

   Step 2 Compute the loss in Equation (2).

   Step 3 Update α_S.

  until e reached.
4.2 Comparative models

We compare our method with three widely used base architectures. To be specific, the competitive methods are introduced as follows:

NeuMF [15]: A deep recommender system which adopts Matrix Factorization (MF) and Multi-Layer Perceptron (MLP) to exploit complicated user-item relationships.

CDAE [16]: This deep architecture uses Denoising Autoencoders (DAE) [17] for the collaborative filtering learning.

Three state-of-art KD techniques are adopted to compared with our method:

Ranking Distillation (RD) [3] first gives a rank for unlabeled items by a teacher model and then considers the rank information as extra knowledge to help train the student model.

Collaborative Distillation (CD) [2] adopts probabilistic rank-aware sampling to handle the ambiguity of missing feedback.

Bidirectional Distillation (BD) [1] samples some items by the teacher and student to effectively enhance the each other.

4.3 Evaluation indicators and settings

In this section, we adopt two metrics to evaluate the ranking performance of these recommender systems. Specifically, Hit Ratio (H@K) tests whether the testing item is listed in the top-K list [18] and Normalized Discounted Cumulative Gain (N@K) [19] gives a higher assessment value to the items at higher rankings in the top-K list.

In this section, we introduce the experimental setup used throughout the paper. Specifically we use PyTorch [20] for the implementation. For each dataset, we use the parameters experimentally which resulted in the best performance for tasks. We train our models with the Adam optimizer [21] with L2 norm regularization and we select the learning rate from {0.00001, 0.0001, 0.001, 0.002} by following the work in [1]. The batch size is set as 128. For NeuMF, we adopt two-layer MLP to implement the network. For CDAE, we adopt two-layer MLP for the model of the encoder and decoder. The dropout ratio is set as 0.5. For the method of CD and RD, we refer to the experimental results followed by [1] with the best performance.

4.4 Result analysis and discussion

Tables 6 list the recommendation performance of CDAE and NeuMF methods on different real-world datasets, including, CiteULike, Foursquare-TYO and Foursquare-NYC, respectively. Notably, we denote "Teacher" and "Student" as the base architectures which are trained individually without the process of distillation learning, and we denote "BD-Teacher" and "BD-Student" are the teacher model and the student model which are both pre-trained jointly with bidirectional distillation. Notabley, the "iBD-Teacher" and "iBD-Student" are the teacher and student models trained with our improved bidirectional distillation strategy of iBD, respectively. Improv.BD-T is the performance improvement of the teacher under the iBD learning strategy and Improv.BD-S is the performance improvement of the student under the iBD learning strategy. From the results in Table 1, we can obtain that the teacher model can be effectively enhanced by the improved bidirectional distillation (iBD) learning strategy, by up to 1.26% under H@50 and 3.25% under H@100 on CiteULike dataset. Similarly, we can see that the student model can be effectively enhanced by the improved bidirectional distillation (iBD) learning strategy, by up to 4.23% under H@50 and 3.93% under H@100 on CiteULike dataset. Moreover, from the Tables 3, we can also find that the proposed method outperforms other methods. As shown in Tables 6, the student model of our solution shows 5.35%, 6.69% and 5.14% better than the other student model under H@50 on CiteULike, Foursquare-TYO and Foursquare-NYC datasets, respectively.

Table 1
Performance comparison of base model CDAE on CiteULike dataset

CiteULike

Model H @ 50 H @ 100 N @ 50 N @ 100

Teacher 0.1759 0.2405 0.0481 0.0586

BD-Teacher 0.1983 0.2646 0.0559 0.0677

BD-Student 0.1040 0.1554 0.0309 0.0392

CD 0.0861 0.1332 0.0250 0.0324

RD 0.0848 0.1266 0.0241 0.0318

iBD-Teacher 0.2008 0.2732 0.0566 0.0683

iBD-Student 0.1084 0.1615 0.0324 0.0410

Student 0.0818 0.1255 0.0244 0.0314

Improv.BD-T 1.26% 3.25% 1.25% 0.89%

Improv.BD-S 4.23% 3.93% 4.85% 4.59%

	CiteULike
Teacher	0.1759	0.2405	0.0481	0.0586
BD-Teacher	0.1983	0.2646	0.0559	0.0677
BD-Student	0.1040	0.1554	0.0309	0.0392
CD	0.0861	0.1332	0.0250	0.0324
RD	0.0848	0.1266	0.0241	0.0318
iBD-Teacher	0.2008	0.2732	0.0566	0.0683
iBD-Student	0.1084	0.1615	0.0324	0.0410
Student	0.0818	0.1255	0.0244	0.0314
Improv.BD-T	1.26%	3.25%	1.25%	0.89%
Improv.BD-S	4.23%	3.93%	4.85%	4.59%

Table 2

Performance comparison of base model CDAE on Foursquare-TYO dataset

	Foursquare-TYO
Model	H @ 50	H @ 100	N @ 50	N @ 100
Teacher	0.1949	0.2416	0.0668	0.0743
BD-Teacher	0.2045	0.2529	0.0742	0.0825
BD-Student	0.1688	0.2124	0.0700	0.0770
CD	0.1629	0.2104	0.0553	0.0651
RD	0.1670	0.2146	0.0572	0.0675
iBD-Teacher	0.2085	0.2595	0.0768	0.0847
iBD-Student	0.1753	0.2212	0.0725	0.0806
Student	0.1579	0.2072	0.0649	0.0728
Improv.BD-T	1.96%	2.61%	3.50%	2.67%
Improv.BD-S	3.85%	4.14%	3.57%	4.68%

Table 3

Performance comparison of base model CDAE on Foursquare-NYC dataset

	Foursquare-NYC
Model	H @ 50	H @ 100	N @ 50	N @ 100
Teacher	0.0826	0.1043	0.0251	0.0306
BD-Teacher	0.0942	0.1191	0.0303	0.0346
BD-Student	0.0785	0.0997	0.0271	0.0334
CD	-	-	-	-
RD	-	-	-	-
iBD-Teacher	0.0960	0.1256	0.0313	0.0356
iBD-Student	0.0801	0.1034	0.0284	0.0349
Student	0.0711	0.0951	0.0185	0.0224
Improv.BD-T	1.91%	5.46%	3.30%	2.89%
Improv.BD-S	2.04%	3.71%	4.80%	4.49%

Table 4

Performance comparison of base model NeuMF on CiteULike dataset

	CiteULike
Model	H @ 50	H @ 100	N @ 50	N @ 100
Teacher	0.1102	0.1641	0.0298	0.0385
BD-Teacher	0.1242	0.1899	0.0341	0.0447
BD-Student	0.0822	0.1245	0.0232	0.0301
CD	0.0820	0.1318	0.0227	0.0307
RD	0.0755	0.1242	0.0197	0.0268
iBD-Teacher	0.1255	0.1926	0.0346	0.0455
iBD-Student	0.0866	0.1274	0.0247	0.0313
Student	0.0606	0.0851	0.0172	0.0227
Improv.BD-T	1.05%	1.42%	1.47%	1.79%
Improv.BD-S	5.35%	2.33%	6.47%	3.99%

Table 5

Performance comparison of base model NeuMF on Foursquare-TYO dataset

	Foursquare-TYO
Model	H @ 50	H @ 100	N @ 50	N @ 100
Teacher	0.1774	0.2206	0.0689	0.0710
BD-Teacher	0.1901	0.2420	0.0751	0.0835
BD-Student	0.1688	0.2202	0.0678	0.0761
CD	0.1548	0.2063	0.0592	0.0677
RD	0.1442	0.1819	0.0547	0.0611
iBD-Teacher	0.1945	0.2433	0.0775	0.0853
iBD-Student	0.1801	0.2324	0.0711	0.0814
Student	0.1260	0.1670	0.0532	0.0614
Improv.BD-T	2.31%	0.54%	3.20%	2.16%
Improv.BD-S	6.69%	5.54%	4.87%	6.96%

Table 6

Performance comparison of base model NeuMF on Foursquare-NYC dataset

	Foursquare-NYC
Model	H @ 50	H @ 100	N @ 50	N @ 100
Teacher	0.0637	0.0868	0.0219	0.0235
BD-Teacher	0.0739	0.0914	0.0267	0.0296
BD-Student	0.0720	0.0905	0.0249	0.0278
CD	-	-	-	-
RD	-	-	-	-
iBD-Teacher	0.0764	0.0959	0.0281	0.0307
iBD-Student	0.0757	0.0931	0.0265	0.0292
Student	0.0556	0.0658	0.0189	0.0201
Improv.BD-T	3.38%	4.92%	5.24%	3.72%
Improv.BD-S	5.14%	2.87%	6.43%	5.04%

Based on the above experimental results in Tables 3, for both BD and iBD, the information of the student helps to enhance the model of teacher. The proposed iBD can effectively transfer the useful knowledge from the student compared with the model BD for the reason that these selected items labeled as noise items are redistilled in the network for training, which provides more information to interpret that the user actually dislikes the item or potentially likes the item.

Figure 3 shows the performance of BD and iBD with different 10 epochs of base model of CADE on CiteULike dataset. We can obtain that: 1) when the number of epoch is less than 300, these two methods achieve similar results, 2) when the number of epoch is large than 300, our proposed method obtains the best result for H@50 and N@50 on CiteULike, which can reflect that our learning strategy based on feedback learning mechanism can definitely select noise items such that only the informative knowledge can be distilled to fully enhance each other.

Fig. 3

The performance of BD and iBD with different 10 epochs: (a) H@50 and (b) N@50.

4.5 Analysis of item selection strategy

We further examine the effect of the improved rank discrepancy-aware sampling scheme based on feedback learning mechanism on the performance of iBD to evaluate the superiority of the proposed sampling strategy. Figures 4(a) and 4(b) show the number of knowledge items in teacher model and student model to be distilled within iBD based on CDAE on Foursquare-NYC dataset. Figures 5(a) and 5(b) show the number of knowledge items in teacher model and student model to be distilled within iBD based on NeuMF on Foursquare-TYO dataset. Specifically, we show the proportional relationship between the number of items sampled by each user during the bidirectional distillation process and the number of items subsequently corrected. In Figs. 5(a) and 5(b), the blue height is the total number of items screened by each user before 10 rounds, and the red height is the total number of items subsequently corrected according to the feedback learning mechanism. As we can see, the noise item with multi-round learning contains information which can help the models to better solve the problem of high ambiguity in nature for recommender system.

Fig. 4

Items analysis and selection strategy of improved rank discrepancy-aware sampling. (a) Teacher model within iBD based on CDAE on Foursquare-NYC dataset. (b) Student model within iBD based on CDAE on Foursquare-NYC dataset.

Fig. 5

Items analysis and selection strategy of improved rank discrepancy-aware sampling. (a) Teacher model within iBD based on NeuMF on Foursquare-TYO dataset. (b) Student model within iBD based on NeuMF on Foursquare-TYO dataset.

4.6 Hyperparameter analysis

Here, we give the hyperparameter analysis of γ_T→S and γ_S→T to evaluate the effectiveness of iBD. We show the results of NeuMF conducted on CiteULike dataset. For the proposed improved rank discrepant items selection based on bidirectional distillation, γ_T→S and γ_S→T are both chosen from the set of {0.1,0.5,0.9}. Specifically, Fig. 6 (a) and (b) show the performance of the teacher and the student with varying γ_T→S and γ_S→T, which balance the parts of the distillation losses. From Fig. 6 (a), when γ_T→S and γ_S→T are around 0.5, the student model can obtain the best performance under H@50 on CiteULike dataset. For the teacher model, we can also obtain that the trend of the experimental result is very similar with that of the student, but the corresponding values are more robust with the γ_T→S.

Fig. 6

Hyperparameter analysis of the student and teacher models on CDAE. (a) γ_T→S for the student model under H@50 and (b) γ_S→T for the teacher model under H@50.

4.7 Running time analysis

In this section, we validate the proposed iBD with two base models of CDAE and NeuMF on three datasets, including, CiteULike, Foursquare-TYO and Foursquare-NYC. The experiments were run on a workstation with Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz and CUDA with NVIDIA GeForce RTX 2080 Ti GPU for acceleration. The results of running time of pre-train phrase, distillation phrase and predict phrase are shown in Table 7. From the results in Table 7, the running time of Pre-train and Distillation depends heavily on two factors: the datasets and the base model architectures. The model in larger dataset requires more running time. Notably, the predict phrase of the proposed method based on different base model architectures can show relatively high efficiency, with less than 5 seconds.

Table 7
Time analysis ( s)

Datasets Model Pre-train Distillation Predict

CiteULike CDAE-T 1824.66 23258.35 4.27

CDAE-S 1659.35 3.80

NeuMF-T 1476.63 22278.47 4.32

NeuMF-S 1257.88 3.98

Foursquare-TYO CDAE-T 1315.17 13037.17 2.17

CDAE-S 1075.89 1.58

NeuMF-T 1116.25 12897.35 1.81

NeuMF-S 1003.69 1.25

Foursquare-NYC CDAE-T 441.95 4054.03 1.17

CDAE-S 402.72 0.65

NeuMF-T 434.08 3463.02 1.34

NeuMF-S 391.92 0.84

Datasets	Model	Pre-train	Distillation	Predict
CiteULike	CDAE-T	1824.66	23258.35	4.27
	CDAE-S	1659.35		3.80
	NeuMF-T	1476.63	22278.47	4.32
	NeuMF-S	1257.88		3.98
Foursquare-TYO	CDAE-T	1315.17	13037.17	2.17
	CDAE-S	1075.89		1.58
	NeuMF-T	1116.25	12897.35	1.81
	NeuMF-S	1003.69		1.25
Foursquare-NYC	CDAE-T	441.95	4054.03	1.17
	CDAE-S	402.72		0.65
	NeuMF-T	434.08	3463.02	1.34
	NeuMF-S	391.92		0.84

5 Conclusions

In this paper, we develop a new sampling strategy to perform bidirectional distillation for top-K recommender system, named improved rank discrepancy-aware sampling based on bidirectional distillation learning. Under our framework, the teacher model and the student model are collaboratively improved with each other in the training phrase. Based on feedback learning mechanism, both the teacher model and the student model can definitely select and make full use of the noise items such that only the important and useful information can be distilled to effectively enhance each other. Thus our method can effectively exploit the high ambiguity in nature for recommender system. At last, experimental results on several real-world recommender system datasets well demonstrate that the proposed iBD achieves superior performance compared with related knowledge distillation methods.

References

Kweon

, Kang

and Yu

, Bidirectional distillation for Top-K recommender system, In Proceedings of the Web Conference (2021), 3861–3871.

Lee

J.W.

, Choi

, Lee

and Shim

, Collaborative distillation for top-N recommendation, In 2019 IEEE International Conference on Data Mining (ICDM). IEEE (2019), 369–378.

Tang

and Wang

, Ranking distillation: Learning compact ranking models with high performance for recommender system, In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2018), 2289–2298.

Kang

, Hwang

, Kweon

and Yu

, DE-RRD: A knowledge distillation framework for recommender system, In Proceedings of the 29th ACM International Conference on Information and Knowledge Management (2020), 605–614.

Kang

W.C.

and McAuley

, Candidate generation with binary codes for large-scale top-n recommendation, In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (2019), 1523–1532.

Lian

, Liu

, Ge

, Zheng

, Xie

and Cao

, Discrete content-aware matrix factorization, In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2017), 325–334.

Zhang

, Shen

, Liu

, He

, Luan

and Chua

T.S.

, Discrete collaborative filtering, In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (2016), 325–334.

Zhang

, Lian

and Yang

, Discrete personalized ranking for fast collaborative filtering from implicit feedback, In Proceedings of the AAAI Conference on Artificial Intelligence (2017).

Zhang

, Xiang

, Hospedales

T.M.

and Lu

, Deep mutual learning, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018), 4320–4328.

10.

, Xia

, Qin

, Wang

, Yu

, Liu

T.Y.

and Ma

W.Y.

, Duallearning for machine translation, Advances in NeuralInformation Processing Systems (2016), (29).

11.

Song

and Chai

, Collaborative learning for deep neuralnetworks, Advances in Neural Information Processing Systems (2018), (31).

12.

Zhou

and Zha

, Learning binary codes for collaborative filtering, In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2012), 498–506.

13.

Wang

, Chen

and Li

W.J.

, Collaborative topic regression with social regularization for tag recommendation, In Twenty-Third International Joint Conference on Artificial Intelligence (2013).

14.

Yang

, Zhang

, Zheng

V.W.

and Yu

, Modeling user activitypreference by leveraging user spatial temporal characteristics inLBSNs, IEEE Transactions on Systems, Man, and Cybernetics: Systems 45(1) (2014), 129–142.

15.

, Liao

, Zhang

, Nie

, Hu

and Chua

T.S.

, Neural collaborative filtering, In Proceedings of the 26th International Conference on World Wide Web (2017), 173–182.

16.

, DuBois

, Zheng

A.X.

and Ester

, Collaborative denoising auto-encoders for top-n recommender systems, In Proceedings of the Ninth ACM International Conference on Web Search and Data Mining (2016), 153–162.

17.

Vincent

, Larochelle

, Bengio

and Manzagol

P.A.

, Extracting and composing robust features with denoising autoencoders, In Proceedings of the 25th International Conference on Machine Learning (2008), 1096–1103.

18.

, Hong

, Lian

, Wu

, Wang

and Ge

, A relaxed ranking-based factor model for recommender system from implicit feedback, In IJCAI (2016), 1683–1689.

19.

Järvelin , Kalervo and Kekäläinen , Jaana, Cumulatedgain-based evaluation of IR techniques, ACM Transactions onInformation Systems (TOIS) 20(4) (2002), 422–446.

20.

Paszke

, Gross

, Massa

, Lerer

, Bradbury

, Chanan

and Chintala

, Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems (2019), 32.

21.

Kingma

D.P.

and Ba

, Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).