An interpretable mechanism for personalized recommendation based on cross feature

Abstract

The existing recommender system provides personalized recommendation service for users in online shopping, entertainment, and other activities. In order to improve the probability of users accepting the system’s recommendation service, compared with the traditional recommender system, the interpretable recommender system will give the recommendation reasons and results at the same time. In this paper, an interpretable recommendation model based on XGBoost tree is proposed to obtain comprehensible and effective cross features from side information. The results are input into the embedded model based on attention mechanism to capture the invisible interaction among user IDs, item IDs and cross features. The captured interactions are used to predict the match score between the user and the recommended item. Cross-feature attention score is used to generate different recommendation reasons for different user-items.Experimental results show that the proposed algorithm can guarantee the quality of recommendation. The transparency and readability of the recommendation process has been improved by providing reference reasons. This method can help users better understand the recommendation behavior of the system and has certain enlightenment to help the recommender system become more personalized and intelligent.

Keywords

Intelligent recommendation interpretability XGBoost attention mechanism cross feature

1 Introduction

The development of information technology and the Internet industry have resulted in “information overload”. Users require to quickly and accurately find what they need from massive resources, while merchants may win more sales volume and profits by displaying proper items to users. However, Google, Baidu and other keyword-based search engines can only meet the simple requirements of the public rather than their personalized demands. The birth of the recommender system has greatly solved this problem [15]. The recommender system is an information filtering system that can predict the users’ rating or preference for a given item based on their basic information and historical behaviour records. The system changes the communication between businesses and users, increases their frequency of interaction. Compared with traditional recommender systems, explainable recommender systems will give the reason while giving the recommendation result. The explanation helps users understand how the model works and provides potential user feedback [10]. This method also increases the probability that users will accept the recommended item. Explainable recommendations are generally divided into three categories: item-based, user-based, and content-based. Item-based and user-based explainable recommendations are usually based on implicit or explicit feedback from users. User-based recommendation reasons are that users who are similar to current users like an item [14]. Nevertheless, users don’t know anything about users who closely resemble them. Consequently, the reason for recommendation may not be convincing. In addition, the disclosure of other users’ information may involve privacy issues in commercial systems. In item-based collaborative filtering, the recommendation reason is that the currently recommended item is analogous to the item you have purchased or browsed [27]. It often displayed as a list of recommended items similar to those have been purchased or viewed. However, users cannot see unused items which have a certain relevance to the item they have experienced.

Research shows that content-based recommendations are more popular with users than user-based and item-based recommendations. The content-based recommendation needs to determine the features of users and items and how interested users are with different features. It can identify the most suitable feature for interpretation and establish a finer modelling. For example, if a movie is recommended to a user, the recommendation reason may be that the movie belongs to a romantic, love, and literary type of movie that female users over the age of 18 enjoy. The feature information of the user and the item can indicate what recommendation is suitable for the user. Therefore, the recommender system should: find effective cross-features from the rich side information of users and items; predict the matching scores of users and items in an interpretable manner. The existing recommender system cannot satisfy these two conditions at the same time.

Gradient Boosting Decision Tree (GBDT) [16] and other tree-based methods make predictions by iteratively reducing the loss function. Generating decision rules is interpreted in this process. However, only a small number of possible feature interactions can be explored by this model, and its exploration ability is weaker than the embedding-based model [20]. Neural factorization machine (NFM) [11] and Wide & Deep [7] also embed the user ID, item ID, and cross features into the shared embedding space. Then the embedding vector is transformed non-linearly. Invisible interactions between user IDs, project IDs, and cross features are captured by the powerful representation capabilities of the non-linear implied layer.

Embedding based methods have strong generalization ability when capturing implicit features on user IDs and project ids. However, when the captured object is side information, due to the non-linear learning process, the captured cross-feature is implicit. The model does not explain the high-order cross features. On the other hand, tree-based models can generate clear decision rules for prediction, which are suitable for learning side information. However, it is not possible to determine the user IDs and project id-related attributes. Since leaves are the most significant cross feature, tree-based models are self-explanatory in nature. Therefore, to build a general and explainable recommender system, a better solution is to combine the two models. We use the active leaf node as a cross-feature of side information, embedding it in the shared embedded space with the user IDs and project IDs, capturing the complex interaction between the three. At the same time, the Attention mechanism was introduced to adjust the model as a weight by learning the importance of each interaction feature from the data rather than using a global weighting mechanism. This improves the performance of the model and helps to recommend that the system choose a more appropriate cross-features for different users-items as an explanation.

The structure of this paper is as follows: Section 2 introduces the related work of the research. Section 3 details the model structure used in this paper: First, we build eXtreme Gradient Boosting (XGBoost) trees [5] on the side information of users and items to obtain effective cross-features. Then we input user IDs, item IDs and cross-features into an embedding-based model. The model is a neural attention network that re-weights the cross-features according to the current predictions. The attention score chooses the most predictive cross features, while using the model to capture unseen interactions between IDs and cross features. The entire forecasting process is completely transparent and self-explanatory. Section 4 introduces the experimental design process and results. Section 5 draws conclusions and proposes future research directions.

2 Related work

Content-based recommendations use established user-preferred profiles and select items with analogous features. It is usually explained intuitively from the perspective of different project functions. A lot of recent work has conducted research from the perspectives of project tags, user comments, visual images and social information.

From the perspective of project tags, the system provides suggestions by matching user profiles with candidate project tags. Vig et al. [31] used movie tags to generate suggestions and explanations. The system shows the movie tags to users and tells which tags are related to them.

Comment-based recommendations use user comments to sentiment analysis and opinion mining for recommendation reasons. Zhang et al. [36] proposed and open-sourced a phrase-level sentiment analysis tool Sentiers. The tool can extract the “feature-view-emotion” triad from product reviews to express users’ positive or negative emotions about the product characteristics. Based on this toolkit, researchers have developed different models to interpret recommendations. Neuro-Linguistic Programming (NLP) tools are used to process comments. Some papers [9, 32] obtain the most important features of each item by mining user comments. Literatures [3 , 35] obtain the user’s sentiment towards the item by analysing the content of user comments. For example, Pappas et al. [24] analysed the sentiment of each comment, constructed the user-item sentiment matrix and used collaborative filtering for processing. The paper [17] proposed a capsule network based model for rating prediction with user reviews, named CARP. CARP extracts the informative logic units from the user and item review documents and infer their corresponding sentiments for rating prediction. In addition, researchers tried to extract user opinions from comments. Integrated Topic and Latent Factor Model (ITLFM) [34] and Rating-Boosted Latent Topics (RBLT) [29] linearly combined comments and ratings to establish a model of user preferences and item features. The results were put into a matrix decomposition model for analysis. Chen et al. [6] designed an encoder-selector-decoder architecture for multi-task learning. The model uses a hierarchical co-attentive selector to identifie reviews and concepts that are important for the user- item pair based on co-attention.

Researchers also tried to use project images to provide users with image-based explanations. Lin et al. [19] studied the interpretable clothing recommendation problem. A convolutional neural network with attention mechanism was proposed to extract visual features of clothing. The extracted features are input into the neural prediction network to predict the recommendation score. McAuley et al. [22] developed a recommender system, which collects users’ visual preferences for modelling. These preferences are used to analyse the image information of the project (such as clothing patterns, styles, fabrics, etc.). The research on visually interpretable recommendations is still at the initial stage, and not all visual features are useful in recommendation prediction [8].

Some researchers used social information such as the social relationships of target users to generate explanations. These methods take into account the friendship and trust relationship between users. Park et al. [25] proposed a unified graph structure. The structure uses friends of the target user with similar preferences as the reason for recommendation. Ratings and social information are used to generate interpretable product recommendations. Chen et al. [4] proposed a SEcure SOcial RECommendation (SeSoRec) framework which is able to collaboratively mine knowledge from social platform to improve the recommendation performance of the rating platform. The research proved the correctness and security theoretically of SeSoRec.

These recommendation methods can provide some specific reasons, and they mainly rely on the content of each item rather than the ratings of other users [1]. However, most of the current work of researchers is to extract features for processing. These features are used as the same independent factors as user ID and item ID for research. Few people notice the impact of the interaction between features.

On the other hand, currently popular recommender system methods, for instance, NFM [11], Wide & Deep Model [7], Memory Augmented Graph Neural Network (MA-GNN) [21], etc. provide the most advanced recommendation performance. Their black box features cannot clearly reflect the reason for recommendation. The new model eXtreme Deep Factorization Machine (xDeepFM) [18] generates higher-order interaction feature in an explicit manner, where multiple interaction networks are stacked. However, xDeepFM has high complexity and is prone to overfitting [30]. Therefore, the paper proposes a new solution that combines XGBoost [5] with the attention mechanism. While mining cross-feature information to improve recommendation performance, it also ensures the interpretability of the recommender system.

3 Interpretable recommendation algorithm based on XGBoost and attention mechanism

3.1 XGBoost tree construction

XGBoost [5] is an efficient implementation of Gradient Boosting, which is a further improvement to GBDT. Boosting is a very effective ensemble learning model. Several weak classifiers with lower accuracy are combined into a strong classifier with higher accuracy. The Gradient Boosting Machine, as an improvement of Boosting, makes the loss function of the algorithm gradient drop in the iterative process when each tree is generated. XGBoost’s base classifier supports both Classification And Regression Trees (CART) and Gradient Boosting linear (GBlinear) classifiers. It has the characteristics of high accuracy, not easy to overfit, scalability, etc. XGBoost also can process high-order sparse features in a distributed manner. For a given dataset D = {(x_i, y_i)}, where ∣D ∣ = N, x_i ∈ R^M, y_i ∈ R, N represents the number of samples. The meaning of M is the number of features in each sample. The XGBoost classification regression tree space Γ is expressed as: $Γ = {f (x) = w_{t (x)}} (t : R^{M} \to S, w \in R^{S})$ (1)

In the equation, S denotes the number of trees, x_i represents the i-th feature vector. The number of leaf nodes in each tree is expressed as L. The structure of each tree t is employed to map each sample to the corresponding leaf node. We can use the following Equation (2) to predict: ${\hat{y}}_{i} = ϕ (x_{i}) = \sum_{s = 1}^{S} f_{s} (x_{i}), f_{s} \in Γ$ (2)

Each f_s corresponds to an independent tree structure t and a leaf node weight w. The instance uses the decision rules of each tree to map to the leaf nodes of the corresponding tree. The weights of all leaf nodes are summed to calculate the final predicted value.

The above tree model cannot be optimized using traditional optimization methods. It is necessary to recursively solve the model of each tree through the training method of Additive Training (Boosting), i.e. the predicted value ${\hat{y}}_{i}^{(p)}$ of the i-th instance at the p-th iteration is equal to the predicted value ${\hat{y}}_{i}^{(p - 1)}$ of the previous p-1 model plus the predicted value f_p (x_i) of the p-th tree, which is also the leaf node weight w_{t(x_i)}.

Thus, the objective function can be written as Equation (3): $L^{(p)} = \sum_{i = 1}^{N} l (y_{i}, {\hat{y}}_{i}^{(p - 1)} + w_{t (x_{i})}) + Ω (f_{p})$ (3)

l represents the loss function, and Ω (f_p) is used as a regularization term. The objective function is expanded by the second-order Taylor formula at ${\hat{y}}_{i}^{(p - 1)}$ , and expressed as Equation (4) in the structure of a tree: ${\hat{L}}^{(p)} = \sum_{i = 1}^{N} [g_{i} w_{t (x_{i})} + \frac{1}{2} h_{i} w_{t (x_{i})}^{2}] + Ω (f_{p})$ (4)

The first derivative and the second derivative of the loss function l respectively are g_i and h_i. Unifying all the leaf nodes, the objective function can be transformed into the cumulative form of leaf nodes. GBDT only uses first-order derivative information when optimizing. However, XGBoost carries out second-order Taylor formula expansion, which use first-order and second-order derivative information at the same time. In addition, the objective function of XGBoost adds the regular term $Ω (f_{t}) = γ L + \frac{1}{2} λ {∥ w ∥}^{2}$ , which includes the number of leaf nodes L and the leaf node weight w. It is equivalent to pruning in the training process to reduce model complexity and prevent overfitting.

3.2 Cross features construction

Recommender systems in the industry use a large number of content features for improving the accuracy of the model. These features can exhibit different information from various dimensions. The cross-features that these features combine also have great significance. Traditional second-order cross features are often designed by hand. After that, the features are input into an interpretable method. For high-order cross-features, the complexity will increase exponentially. It also requires designers who possess a lot of domain knowledge. This kind of labour-intensive work has high costs, large limitations, and poor scalability. In order to avoid these shortcomings, scholars began to study the use of deep models to automatically learn high-order cross-features. However, due to the black-box nature of the depth model, the obtained cross features are implicit and difficult to explain.

Therefore, the paper chooses to use XGBoost to capture effective cross features. Although XGBoost is not specifically designed for extracting cross-features, a leaf node represents a cross-feature. XGBoost optimizes the prediction of cross-features in iterations. It is considered that leaf nodes are useful and cross-features are reasonable.

Formally, the paper expresses XGBoost as a set of decision trees T = {T₁, T₂, . . . , T_S}, where each tree maps a feature vector x to a leaf node with a weight w. The number of leaf nodes in the s-th tree is expressed as L_s. It is different from the original XGBoost tree that sums the weights of the active leaf nodes as the predicted value. In this paper, the active leaf nodes are extracted as effective cross-features and provided to the subsequent model for processing. The embedding_lookup function is used to convert the set of all leaf nodes obtained in the XGBoost tree into a multi-hot vector t = XGBoost (x ∣ T) = [T₁ (x) , . . . , T_S (x)]. The vector is the concatenation of all the leaf nodes in the decision tree. The number of elements with value one in t is equal to the number of trees in XGBoost. Each tree has only one active leaf node, so a single effective cross vector can be obtained.

There are two trees T₁ and T₂ in the Figure 1. The depth of the trees is two, and there are three and four leaf nodes in the trees, respectively. The feature vector of an input sample is expressed as x, with four features x₀, x₁, x₂, x₃. After traversing the two trees, the feature vector x of the sample is mapped to the second node of T₁ and the second node of T₂. Each leaf node corresponds to a cross feature. By traversing the leaf nodes of all trees, all the cross features of the sample are obtained. The multi-hot vector t should be expressed as [0, 1, 0, 0, 1, 0, 0]. t extracts two effective crosses from the feature vector x = [x₀, x₁, x₂, x₃] to input into the subsequent model. Moreover, as the depth of the tree increases, XBGoost can capture high-order cross features. If the above example is substituted into the specific semantics, the obtained cross-features can be vividly displayed:

vL₂ : [Age > 18] & [Occupation = teacher]

vL₅ : [Age < 35] & [Gender ≠ Man]

The first cross feature indicates that the user is a person older than eighteen and whose occupation is a teacher. The second cross feature indicates that the user is a female younger than 35.

3.3 Cross features weighting based on attention mechanism

The paper uses a multi-hot vector t to represent the effectiveness of each cross feature. The intersection feature of value 1 is selected as the interpretation direction of the prediction results. The GBDT with logistic regression (GBDT+LR) model by Facebook [12] has demonstrated the effectiveness of this solution. However, the cross features of all user-item pairs are assigned the same weight. The processing will limit the prediction effect, because the importance of different cross features is different. Some useless feature interactions may even introduce noise. Therefore, the Attentional Factorization Machines (AFM) [33] model introduces the Attention mechanism, which learns the importance of each cross feature from the data as a weight to adjust the model. This mechanism improves the performance of the model, and explainable Attention can improve the explainability of the model.

For capturing the correlation of cross features in sparse multi-hot vectors, firstly, each valid cross feature is associated with an embedding vector. Then the cross features represented by each embedding vector are weighted individually. Inspired by the AFM model, an attention mechanism is designed based on its cross feature weighting method to weight the cross features. The modelling process of personalized weights on cross features is shown in Equation (5): ${\begin{matrix} {a^{'}}_{ijl} = h^{T} Re lu (W ([u_{i} \otimes e_{j}, v_{l}]) + b \\ a_{ijl} = \frac{exp ({a^{'}}_{ijl})}{\sum_{ijl} exp ({a^{'}}_{ijl})} \end{matrix}$ (5) where W ∈ R^a×2k and b ∈ R^a are respectively used as the weight matrix and bias vector of the hidden layer. The embedding size k is used to control the embedding layer size. The attention size a is employed to control the size of the hidden layer. In AFM, the implicit cross feature variables obtained from the inner product of any two feature embeddings are selected in order to ensure the transparency of the model. The feature variables are replaced with the effective cross feature v obtained from the XGBoost tree model. However, in reality, users with similar characteristics may choose items for various reasons. Therefore, when using different user-item pairs to make predictions, the degrees of influence of various cross-features are different. NFM [11] and Wide & Deep [7] can be realized by embedding the user ID, item ID and cross feature into the shared embedding space together. The embedding vectors are transformed nonlinearly. The powerful representation capability of the nonlinear hidden layer ensures the capture of complex interactions between user ID, item ID and cross-features. The paper draws on this idea, integrates user ID vector u, item ID vector e and cross feature vector v. v and u ⊗ e are concatenated in the embedding dimension. The complex interactions between user ID, item ID and cross-features are captured while avoiding the use of hidden layers. The interaction between the three can be explained intuitively.

3.4 Model prediction

The hybrid model structure based on XGBoost and attention mechanism is shown in Figure 2.

After obtaining the weights of cross features, the paper aggregates the embedding vectors of all cross features in each sample by maximum pooling. Each sample obtains a uniform representation for cross features. This approach can retain main features, while reducing the parameters and calculations, preventing from overfitting and improved generalization ability. $B_{max} (i, j, V) = max_poo l_{v_{l} \in V} (a_{ijl} v_{l})$ (6)

Previously, only the cross features about the side information of users and items were considered. Therefore, the embedding vectors x, u, e, V of side information, user ID, item ID and cross feature set are integrated together. These unified embedding vectors use linear regression to make the final prediction. P ∈ R^k and Q ∈ R^k are used as the weights of the final linear regression layer. The sigmod function is used for activation to obtain the probability that the recommender system recommends a certain product for a user. $\hat{y} (i, j, l) = w_{0} + \sum_{k = 1}^{M} w_{k} x_{k} + P^{T} (u_{i} ⊙ e_{j}) + Q^{T} B (i, j, V)$ (7)

In this paper, whether the recommender system recommends a certain item for a user is regarded as a binary classification problem. Consequently, the paper uses Logloss as the loss function and uses L2 regularization to prevent overfitting. Since this model is composed of two models in series, the two models are trained separately. After training XGBoost to minimize the loss value, input cross features into the model based on the attention mechanism for training. Model optimization uses min-batch Adagrad.

This model is only a shallow model. Although there is no fully connected hidden layer, but the embedding mechanism and attention mechanism still make the model have strong representation ability and effectiveness. Because of the additivity of the shallow model, the contribution of each component can be effortlessly evaluated for uncomplicated interpretation.

4 Experimental design and analysis

4.1 Experimental environment and data sources

The experiment used two public datasets, from MovieLens 100k [13] and Myers-Briggs Personality Type Dataset [23]. MovieLens is one of the most commonly used datasets to verify the performance of the recommendation algorithm. There are a total of 100,000 ratings generated by 943 users rating 1682 movies. Each user has at least 20 ratings, and the rating is an integer from one to five. The rating is converted into a binary implicit feedback, indicating whether the user recommends the movie. On a scale of one to three, the rating is converted to zero, which means the movie is not worth recommending. When the rating is four or five, it is converted to one, which means the movie is worth recommending. The information of each user includes gender, age, occupation, region, and movie attributes are composed of 20 movie genre tags. The second data set comes from the Kaggle public database, with a total of 1,028,752 ratings, and 1,820 users rated 35,196 movies. The score is a number from zero to five with a step size of 0.5. The score is converted into a binary implicit feedback. A score from zero to three is not recommended, and a score of 3.5-5 is recommend. The detailed information are shown in Table 1. Table 2 and Table 3 respectively show some samples of these datasets.

Table 1
The description of some representative features

Dataset Features Attribute type Description

Ds1 age numercial attibute The age of the user

occupation discrete attribute There are 21 occupations of users, like administrator, artist, doctor, etc.

state discrete attribute Inferred from the zip code, representing the area of this user

label discrete attribute The label of this movie, such as ’Action’, ’Comedy’, etc.

Ds2 openness discrete attribute An assessment score (from 1 to 7) assessing user tendency to prefer new experience

emotional stability discrete attribute An assessment score (from 1 to 7) assessing user tendency to have psychological stress

extraversion discrete attribute An assessment score (from 1 to 7) assessing user tendency to be outgoing

conscientiousness discrete attibute An assessment score (from 1 to 7) assessing user tendency to be organized and dependable

Dataset	Features	Attribute type	Description
Ds1	age	numercial attibute	The age of the user
	occupation	discrete attribute	There are 21 occupations of users, like administrator, artist, doctor, etc.
	state	discrete attribute	Inferred from the zip code, representing the area of this user
	label	discrete attribute	The label of this movie, such as ’Action’, ’Comedy’, etc.
Ds2	openness	discrete attribute	An assessment score (from 1 to 7) assessing user tendency to prefer new experience
	emotional stability	discrete attribute	An assessment score (from 1 to 7) assessing user tendency to have psychological stress
	extraversion	discrete attribute	An assessment score (from 1 to 7) assessing user tendency to be outgoing
	conscientiousness	discrete attibute	An assessment score (from 1 to 7) assessing user tendency to be organized and dependable

Table 2

A sample about the detail records of MovieLens 100k

User id	Item id	age	gender	occupation	state	movie name	label	rating
1	1	24	Man	technician	8	Toy Story (1995)	Animation, Children’s, Comedy	5
6	1	42	Man	executive	9	Toy Story (1995)	Animation, Children’s, Comedy	4
22	2	25	Man	writer	4	GoldenEye (1995)	Action, Adventure, Thriller	2
49	2	23	Woman	student	7	GoldenEye (1995)	Action, Adventure, Thriller	1
43	3	29	Woman	librarian	2	Four Rooms (1995)	Thriller	2

Table 3

A sample about the detail records of anime recommendation database

User id	Item id	openness	agreeableness	emotional stability	conscientiousness	extraversion	assigned condition	rating
1	310	5.5	6	2	4.5	1	3	4.5
2	734	4.5	4.5	2.5	4.5	4	3	3.5
3	1293	6.5	3.5	3	5	5	4	2.5
4	4617	5.5	4	3	1.5	2.5	2	4
5	13460	6.5	6.5	6	7	5	3	5

For each dataset, this paper randomly selects 20% as test data, and uses k-fold cross-validation to randomly divide the remaining data into training set (70%) and validation set (10%). The verification set is mainly used to adjust the hyperparameters, and finally the test set is used to compare the performance.

The simulation environment of this paper: hardware environment configuration: Intelcorei7 processor, 500G hard disk. Software environment configuration: simulation tool Pycharm Community, Anaconda, Win10 operating system. Anaconda is used to preprocess data, and Pycharm is used to build models to implement algorithms.

4.2 Evaluation metrics

To evaluate the prediction score, two indicators are used: Logloss and Area Under Curve (AUC). We report the average score for all test instances, and the same settings apply to hyperparameter tuning on the validation set. Logloss [28] indicates the generalization ability of each model, measures the probability of user-project interaction deviating from the basic facts, pays more attention to the accuracy of the model, AUC indicates the ability to distinguish positive from negative models, and pays more attention to the sorting results of models. When the data scale is unbalanced, the model may be biased toward predicting more negative samples, and the Logloss results are still acceptable, but the AUC results are not ideal. Therefore, both are selected for comprehensive evaluation.

When there are N samples, for the i-th sample, the true value is y_i and the predicted value is ${\hat{y}}_{i}$ . The calculation formula of Logloss is as Equation 8, the smaller the value, the better. $Logloss = \frac{1}{N} \sum_{i = 1}^{N} - y_{i} log {\hat{y}}_{i} - (1 - y_{i}) log (1 - {\hat{y}}_{i})$ (8) If we classify a sample, there will be four situations, namely:

TP: true positive (positive sample, and is predicted to be positive sample)

FP: false positive (negative sample, but is predicted to be positive sample)

TN: true negative (negative sample, and is predicted to be a negative sample)

FN: False negative (positive sample, but predicted as negative sample)

The AUC curve is the area under the curve. The curve is the receiver operating characteristic curve (ROC). The ordinate of the curve is the true positive rate (TPR), and the abscissa is the false positive rate (FPR).

$TPR = \frac{TP}{TP + FN}$ represents the proportion of correct predictions among all positive samples. It is also called the recall rate.

$FPR = \frac{FP}{FP + TN}$ represents the proportion of all negative samples that are wrongly predicted as positive samples.

The model takes different probability thresholds to get different (FPR, TPR), which is connected to the ROC curve. The AUC curve is the area under the ROC curve. The larger the TPR, the better, the smaller the FPR, and the higher the AUC value, the better.

When a positive sample and a negative sample are randomly selected, the probability of placing the positive sample in front of the negative sample according to the predicted value obtained by the model is the AUC value. The larger the AUC value, the more likely the current classification algorithm will rank the positive samples in front of the negative samples, and can be better classified.

4.3 Evaluation and results

4.3.1 Comparative experiment with other models

For proving the recommendation quality of the model, the model based on XGBoost and attention mechanism proposed in this paper is compared with the existing matrix factorization (MF), GBDT+LR model and AFM model. In order to make a fair comparison, all models are adjusted and optimized in this paper.

MF [26]: A general model-based collaborative filtering method that uses embedding vectors to encode user ID and item ID. It implicitly models cross features through the inner product of feature embedding.

GBDT+LR [12]: This method will input the cross features extracted from GBDT into logistic regression to refine the weight of each cross feature.

AFM [33]: The FM model with the Attention mechanism added, and the importance of each interactive feature is learned from the data through the neural network to adjust the model.

It can be seen from Table 4 that the method in this paper is slightly better than the comparison method in the Logloss and AUC indicators of the model. The model has higher accuracy and better ranking results while ensuring interpretability.

Table 4
Performance comparison of each model

Dataset Model Logloss AUC

Ds1 MF 0.5963 0.7337

GBDT+LR 0.6313 0.7602

AFM 0.5817 0.7715

Method of this paper 0.5800 0.7773

Ds2 MF 0.6235 0.7029

GBDT+LR 0.6319 0.7278

AFM 0.6156 0.7393

Method of this paper 0.6132 0.7420

Dataset	Model	Logloss	AUC
Ds1	MF	0.5963	0.7337
	GBDT+LR	0.6313	0.7602
	AFM	0.5817	0.7715
	Method of this paper	0.5800	0.7773
Ds2	MF	0.6235	0.7029
	GBDT+LR	0.6319	0.7278
	AFM	0.6156	0.7393
	Method of this paper	0.6132	0.7420

4.3.2 Hyper-parameter studies

The value of the hyperparameter affects the overall performance of the model. In the two data sets, the number of model trees is based on {50, 100, 150, 200, 250} and {25, 50, 75, 100, 125} respectively. The maximum depth of a tree is selected among {2, 4, 6, 8, 10}. The attention size is set to be the same as the embedding size based on experience, and the value is taken at {5, 10, 15, 20}. The learning rate range is {0.005, 0.01, 0.03, 0.05, 0.1}. All embedding-based methods are optimized using mini-batch Adagrad. Take the number of trees and the maximum depth as an example for analysis.

The effect of tree number

The number of trees is related to the coverage of cross features, reflecting how much information is derived from the datasets. In Figure 3(a)-3(d), as the number of trees increases, the Logloss value gradually decreases, and the AUC gradually increases. As shown in Figure 3(a), performance is best in the first dataset when the number of trees is 200. When the number of trees is 250, Logloss increases and AUC decreases. Maybe the model is overfitting.

The effect of maximum tree depth

The depth of the tree determines the highest order of cross features, which can dig out more interactive information between features. Models with too small depth will underfit. When the depth is too large, the model will overfit.The experimental results are shown in Figure 3(e)-3(h), both datasets perform best when the maximum depth of the tree is eight.

Fig. 1

XGBoost tree model example.

Fig. 2

Structure diagram of hybrid model based on XGBoost and attention mechanism.

Fig. 3

The influence of tree number and maximum tree depth on model effect.

4.3.3 Model interpretability description

The model predicts by generating explicit decision rules through the XGBoost tree, making the cross-features directly interpretable. The experiment uses the Graphviz package to visualize the decision tree in XGBoost. Part of the decision tree obtained is shown in Figure 4. In this process, the effective cross features can be captured. The sample labels in the figure are binary features. The value of label is zero or one. When the value is greater than 0.5, indicating that the item contains this label. On the contrary, the item does not include this tag.

Fig. 4

Partial decision process of XGBoost tree.

After obtaining cross features, the weights of effective cross features in the attention mechanism represent the importance of the cross feature in prediction. Cross features of the heavy weight can be recommended to users as explanations.

4.3.4 Model scruability

In addition to making the cross-features visible, our model allows users to correct the process to refresh the recommendations as they desire. This property of adjusting recommendation is called scrutability and is the gateway to control the recommendation process [2]. It is illustrated by a sample user in Table 5.

Table 5
Scrutable recommendation for a sampled user on the model

Top Ranked Recommendation List on the model

Original 1. Fargo 2. East of Eden 3. Three Colors: Red

4. Lone Star 5. Short Cuts

Adjusted 1. Alien, 2. Shining 3. Scream

4. Jaws 5. Candyman

Top Ranked Recommendation List on the model
Original	1. Fargo 2. East of Eden 3. Three Colors: Red
	4. Lone Star 5. Short Cuts
Adjusted	1. Alien, 2. Shining 3. Scream
	4. Jaws 5. Candyman

The profile of the sample user is a 42-year-old female entertainer. In addition, most of the films in her historical interactions have the tag of Drama. As a result, the model detected such frequent cross features and recommended films such as Fargo and East of Eden to her. Supposing that the user wants to recommend movies to a scientist who loves horror movies. The model can assign a higher attentive weight to the cross feature that includes [Label = Horror] and [Evidence = scientist], and then obtain the predictions to update the recommendations. The Alien, Glow, Scream, Jaws and Candyman are the top score of prediction in the adjusted recommendation list.

5 Conclusion

This paper first gives an explanatory problem description of the recommender system. On the basis of the problem, an interpretable recommendation model based on XGBoost and attention mechanism is proposed, which solves the self-explanatory problem of the model prediction. The effective cross features are extracted from the XBGoost-based tree part. Through the introduction of the Attention mechanism, it is possible to predict the matching degree between users and items in an interpretable manner. According to the experimental results, the proposed method guarantees the recommendation performance and improves the readability of the recommendation reason. In the following job, we plan to use knowledge graphs to enhance algorithm interpretation capabilities. As a highly readable external knowledge carrier, the knowledge graph provides a great possibility for improving the ability of algorithm interpretation. Common recommended interpretation media are users, items, and features. The model will explore the associations between media, and select the most appropriate media to recommend and explain to users in accordance with the situation. Recommendation results and interpretation capabilities will be optimized. Another future direction is to use contextual information such as text comments and pictures. Comments extract the features of the items that users are most interested in. Features are added to the recommender system for modelling. It is significant to find a way to understand user preferences in different aspects.

References

Al-Ghuribi

S.M.

and Noah

S.A.M.

, Multi-criteria review-based recommender system–the state of the art, IEEE Access 7 (2019), 169446–169468.

Balog

, Radlinski

and Arakelyan

, Transparent, scrutable and explainable user models for personalized recommendation, In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 265–274, 2019.

Bauman

, Liu

and Tuzhilin

, Aspect based recommendations: Recommending items with the most valuable aspects based on user reviews, In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 717–725, 2017.

Chen

, Li

, Wu

, Hong

, Wang

and Zhou

, Secure social recommendation based on secret sharing, arXiv preprint arXiv:2002.02088, 2020.

Chen

and Guestrin

, Xgboost: A scalable tree boosting system, In Proceedings of the 22nd acmsigkdd international conference on knowledge discovery and data mining, pages 785–794, 2016.

Chen

, Wang

, Xie

, Wu

, Bu

, Wang

and Chen

, Co-attentive multi-task learning for explainable recommendation, In IJCAI, pages 2137–2143, 2019.

Cheng

H.-T.

, Koc

, Harmsen

, Shaked

, Chandra

, Aradhye

, Anderson

, Corrado

, Chai

, Ispir

, et al. Wide & deep learning for recommender systems, In Proceedings of the 1st workshop on deep learning for recommender systems, pages 7–10, 2016.

Cheng

, Chang

, Zhu

, Kanjirathinkal

R.C.

and Kankanhalli

, Mmalfm: Explainable recommendation by leveraging reviews and images, ACM Transactions on Information Systems (TOIS) 37(2) (2019), 1–28.

D’Addio

, Conrado

, Resende

and Manzato

, Generating recommendations based on robust term extraction from users’ reviews, In Proceedings of the 20th Brazilian Symposiumon Multimedia and the Web, pages 55–58, 2014.

10.

Gao

, Wang

and Xie

, Explainable recommendation through attentive multi-view learning, In, Proceedings of the AAAI Conference on Artificial Intelligence 33 (2019), 3622–3629.

11.

and Chua

T.-S.

, Neural factorization machines for sparse predictive analytics, In Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 355–364, 2017.

12.

, Pan

, Jin

, Xu

, Liu

, Xu

, Shi

, Atallah

, Herbrich

, Bowers

, et al. Practical lessons from predicting clicks on ads at facebook, In Proceedings of the Eighth International Workshop on Data Mining for Online Advertising, pages 1–9, 2014.

13.

Herlocker

J.L.

, Konstan

J.A.

, Borchers

and Riedl

, An algorithmic framework for performing collaborative filtering, In ACM SIGIR Forum, volume 51, pages 227–234. ACM New York, NY, USA, 2017.

14.

Herlocker

J.L.

, Understanding and improving automated collaborative filtering systems, Citeseer, 2000.

15.

Huang

, Zhang

H.-Y.

and Wang

J.-Q.

, A comprehensive mechanism for hotel recommendation to achieve personalized search engine, Journal of Intelligent & Fuzzy Systems 35(3) (2018), 3733–3745.

16.

, Meng

, Finley

, Wang

, Chen

, Ma

, Ye

and Liu

T.-Y.

, Lightgbm: A highly efficient gradient boosting decision tree, In Advances in neural information processing systems, pages 3146–3154, 2017.

17.

, Quan

, Peng

, Qi

, Deng

and Wu

, A capsule network for recommendation and explaining what you like and dislike, In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 275–284, 2019.

18.

Lian

, Zhou

, Zhang

, Chen

, Xie

and Sun

, xdeepfm: Combining explicit and implicit feature interactions for recommender systems, In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1754–1763, 2018.

19.

Lin

, Ren

, Chen

, Ren

, Ma

, de Rijke

, et al., Explainable fashion recommendation with joint outfit matching and comment generation, arXiv preprint arXiv:1806.08977, 2, 2018.

20.

Liu

, Zhu

, Li

, Zhang

, Lai

, Tang

, He

, Li

and Yu

, Autofis: Automatic feature interaction selection in factorization models for click-through rate prediction, arXiv preprint arXiv:2003.11235, 2020.

21.

, Ma

, Zhang

, Sun

, Liu

and Coates

, Memory augmented graph neural networks for sequential recommendation, In, Proceedings of the AAAI Conference on Artificial Intelligence 34 (2020), 5045–5052.

22.

McAuley

, Targett

, Shi

and Hengel

A.V.D.

, Image-based recommendations on styles and substitutes, In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pages 43–52, 2015.

23.

Nguyen

T.T

, Harper

F.M.

, Terveen

and Konstan

J.A.

, User personality and user satisfaction with recommender systems, Information Systems Frontiers 20(6) (2018), 1173–1189.

24.

Pappas

and Popescu-Belis

, Sentiment analysis of user comments for one-class collaborative filtering over ted talks, In Proceedings of the 36th international ACMSIGIR conference on Research and development in information retrieval, pages 773–776, 2013.

25.

Park

, Jeon

, Kim

, Ahn

and Kang

, Uniwalk: Explainable and accurate recommendation for rating and network data, arXiv preprint arXiv:1710.07134, 2017.

26.

Rendle

, Factorization machines, In 2010 IEEE International Conference on Data Mining, pages 995–1000. IEEE, 2010.

27.

Sarwar

, Karypis

, Konstan

and Riedl

, Item-based collaborative filtering recommendation algorithms, In Proceedings of the 10th international conference on World Wide Web, pages 285–295, 2001.

28.

Shan

, Hoens

T.R.

, Jiao

, Wang

, Yu

and Mao

J.C.

, Deep crossing:Web-scale modeling without manually crafted combinatorial features, In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 255–262, 2016.

29.

Tan

, Zhang

, Liu

and Ma

, Ratingboosted latent topics: Understanding users and items with ratings and reviews, In, IJCAI 16 (2016), 2640–2646.

30.

Tao

, Wang

, He

, Huang

and Chua

T.-S.

, Hoafm: A high-order attentive factorization machine for ctr prediction, Information Processing & Management 57(6) (2020), 102076.

31.

Vig

, Sen

and Riedl

, Tagsplanations: explaining recommendations using tags, In Proceedings of the 14th international conference on Intelligent user interfaces, pages 47–56, 2009.

32.

Wang

, Wang

and Xu

, A sentimentenhanced hybrid recommender systemformovie recommendation: a big data analytics framework, Wireless Communications and Mobile Computing, 2018, 2018.

33.

Xiao

, Ye

, He

, Zhang

, Wu

and Chua

T.-S.

, Attentional factorization machines: Learning the weight of feature interactions via attention networks, arXiv preprint arXiv:1708.04617, 2017.

34.

Zhang

and Wang

, Integrating topic and latent factors for scalable personalized review-based rating prediction, IEEE Transactions on Knowledge and Data Engineering 28(11) (2016), 3013–3027.

35.

Zhang

, Lai

, Zhang

, Liu

and Ma

, Explicit factor models for explainable recommendation based on phrase-level sentiment analysis, In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval, pages 83–92, 2014.

36.

Zhang

, Zhang

, Liu

and Ma

, Do users rate or review? boost phrase-level sentiment labeling with review-level sentiment classification, In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval, pages 1027–1030, 2014.

An interpretable mechanism for personalized recommendation based on cross feature

Abstract

Keywords

1 Introduction

2 Related work

3 Interpretable recommendation algorithm based on XGBoost and attention mechanism

3.1 XGBoost tree construction

3.3 Cross features weighting based on attention mechanism

4.1 Experimental environment and data sources

4.3.1 Comparative experiment with other models

Table 4 Performance comparison of each model Dataset Model Logloss AUC Ds1 MF 0.5963 0.7337 GBDT+LR 0.6313 0.7602 AFM 0.5817 0.7715 Method of this paper 0.5800 0.7773 Ds2 MF 0.6235 0.7029 GBDT+LR 0.6319 0.7278 AFM 0.6156 0.7393 Method of this paper 0.6132 0.7420

Table 5 Scrutable recommendation for a sampled user on the model Top Ranked Recommendation List on the model Original 1. Fargo 2. East of Eden 3. Three Colors: Red 4. Lone Star 5. Short Cuts Adjusted 1. Alien, 2. Shining 3. Scream 4. Jaws 5. Candyman

References

Table 4
Performance comparison of each model

Dataset Model Logloss AUC

Ds1 MF 0.5963 0.7337

GBDT+LR 0.6313 0.7602

AFM 0.5817 0.7715

Method of this paper 0.5800 0.7773

Ds2 MF 0.6235 0.7029

GBDT+LR 0.6319 0.7278

AFM 0.6156 0.7393

Method of this paper 0.6132 0.7420

Table 5
Scrutable recommendation for a sampled user on the model

Top Ranked Recommendation List on the model

Original 1. Fargo 2. East of Eden 3. Three Colors: Red

4. Lone Star 5. Short Cuts

Adjusted 1. Alien, 2. Shining 3. Scream

4. Jaws 5. Candyman