Unsupervised contaminated user profile identification against shilling attack in recommender system

Abstract

A recommender system is susceptible to manipulation through the injection of carefully crafted profiles. Some recent profile identification methods only perform well in specific attack scenarios. A general attack detection method is usually complicated or requires label samples. Such methods are prone to overtraining easily, and the process of annotation incurs high expenses. This study proposes an unsupervised divide-and-conquer method aiming to identify attack profiles, utilizing a specifically designed model for each kind of shilling attack. Initially, our method categorizes the profile set into two attack types, namely Standard and Obfuscated Behavior Attacks. Subsequently, profiles are separated into clusters within the extracted feature space based on the identified attack type. The selection of attack profiles is then determined through target item analysis within the suspected cluster. Notably, our method offers the advantage of requiring no prior knowledge or annotation. Furthermore, the precision is heightened as the identification method is designed to a specific attack type, employing a less complicated model. The outstanding performance of our model, validated through experimental results on MovieLens-100K and Netflix under various attack settings, demonstrates superior accuracy and reduced running time compared to current detection methods in identifying Standard and Obfuscated Behavior Attacks.

Keywords

1. Introduction

In the era of big data, a recommender system has emerged as an effective tool for alleviating information overload by proactively providing potential products and services of interest to users [1]. Despite achieving remarkable performance across various applications [2, 3, 4], recommender systems remain vulnerable to adversarial attacks, a concern due to their openness, anonymity, and interactivity [5]. The shilling attack, also known as profile injection attack, is one of well-known attacks to recommender system, aiming to manipulate the frequency of recommendations for target items [6] by injecting well crafted profiles into the system. Numerous studies [7] indicate that even a small amount of attacked profiles can successfully mislead recommender systems.

Shilling attack detection, as a countermeasure, identifies suspected profiles based on various characteristics such as rating values [5, 8, 9, 10, 11], rating patterns [12, 13, 14, 15] and rating time [10, 16, 31]. Although supervised learning-based methods identify attack profiles successfully, their efficacy heavily relies on the knowledge of attack strategy. The performance drops significantly when an attack is different from the ones in the training set in inference [8, 9]. Moreover, the collection of attack profiles incurs substantial costs [17]. Some studies indicate that unsupervised detection methods, which cluster profiles according to their characteristic without using labels, demonstrate generalizability to unknown attacks in certain applications [18]. However, these methods usually assume that the prior knowledge of the attack, for example, the number of attack profiles [19] and candidate attack users [20], can be obtained in advance. This assumption may not be realistic in some applications. In addition, the unsupervised detection methods may exhibit poor performance in detection when the attack size or filler size is small [16].

This study mainly focuses on standard and obfuscated behavior attacks, recognized as the most prevalent shilling attack types encompassing a wide array of existing attack methods [18, 20, 21]. In standard behavior attacks, rated items are randomly selected, while popular items are deliberately chosen for rating in obfuscated behavior attacks to better simulate the rating behaviors of genuine profiles. An unsupervised attack profile identification method without any prior knowledge is proposed. Acknowledging the distinct characteristics of standard and obfuscated behavior attacks, a divide-and-conquer strategy is employed to address them separately. Given a set of user profiles, the potential attacks are classified as either standard or obfuscated behavior attacks according to their item popularity. Principal Component Analysis (PCA) [19] and Mean Popularity of a User (MPU) [22] are extracted for standard behavior attacks, while Weighted Deviation from Mean Agreement (WDMA) [9] is extracted for obfuscated behavior attacks. Afterwards, a clustering method based on the extracted feature is applied to determine suspicious profiles. Finally, target items are analyzed to identify attack profiles. The advantages of our methods include high efficiency, strong interpretability, and easy implementation. Our proposed method is then evaluated and compared against state-of-the-art (SOTA) methods, including Pop-SAD [22], SpDetector [23] and PCA-VarSelect [19], using two popular recommender system datasets, i.e., MovieLens-100K1 and Netflix.2 Experimental results affirm that our proposed method outperforms all others in terms of prior knowledge, accuracy, and running time when identifying profiles contaminated by standard and obfuscated behavior attacks with different attack and filler sizes.

The contributions of the study are shown as follows:

A divide-and-conquer approach is proposed to address the detection of standard and obfuscated behavior attacks separately. By individually considering the characteristics of each attack type, the detection performance in terms of accuracy and complexity can be enhanced.

The obvious attack profiles are initially preliminarily filtered to approximate the distribution of genuine profiles in our model. The approximated distribution serves as a reference to enhance the precision of attack profile detection.

Experimental results confirm the outstanding performance of our proposed identification method across seven common attack models under different settings on two benchmark datasets, i.e., MovieLens-100K and Netflix, when compared to Pop-SAD, SpDetector and PCA-VarSelect methods.

The rest of the paper is organized as follows. A brief introduction of related work is provided in Section 2, and a detailed description of the proposed method is represented in Section 3. The experimental results are presented and discussed in Section 4. Finally, the conclusions and future works are presented in Section 5.

2. Related work

In this section, shilling attack models adopted in this paper are introduced in section 2.1, and a brief review of the shilling attack detection methods is provided in section 2.2.

2.1 Shilling attack model

Shilling attacks are classified by their objectives into three categories: availability attacks, push attacks, and nuke attacks [20, 24, 25]. Availability attack aims to disfunction a recommender system, whereas push and nuke attacks aim to increase and decrease the recommendation frequency of target items respectively.

Shilling attacks are further distinguished based on the level of knowledge required for their execution, categorizing them into low knowledge attacks and high knowledge attacks [21, 25, 26, 27]. Practically, only limited knowledge of a target system can be obtained by an attacker. The attack with low knowledge usually consider [7, 20, 21, 25] selected ( $I_{s}$ ), filler ( $I_{f}$ ), unrated ( $I_{\emptyset}$ ) and target items ( $I_{t}$ ). $I_{f}$ is randomly selected to enhance the similarity between attack profiles and genuine profiles, thereby reducing the likelihood of the attack profiles being identified. On the other hand, $I_{s}$ is elaborately selected to increase the effectiveness of the attack. $I_{f}$ is randomly selected and rated according to the Gaussian distribution with the average ratings as the mean of all items and $I_{t}$ is assigned as $r_{\max}$ ( $r_{\min}$ ) for push (nuke) attack in random attack [21, 25]. Average attack [21, 25] is the same as random attack except $I_{f}$ is rated according to the Gaussian distribution with the average rating of each filler item. Bandwagon attack [25] rates $I_{s}$ selected from the popular items as $r_{\max}$ , and $I_{f}$ and $I_{t}$ are handled in the same way as average attack. Average-noise injection attack [28] improves average attack by adding a random noise to filler items, while the target items are shifted from $r_{\max}$ to $r_{\max}-1$ for push attacks, and $r_{\min}$ to $r_{\min}+1$ for nuke attack in average-target shifting attack [28]. AoP attack [21, 25] is the same as average attack except $I_{f}$ is randomly selected from the top $x\%$ of the most popular items, where $x\%$ is larger than filler size. Power item attack [21] revises AoP attack by setting that $x\%$ is equal to filler size.

The attack performance is improved by acquiring extensive knowledge about the target recommender system, e.g., the item popularity and item correlation [26, 27]. In addition, Generative Adversarial Network (GAN) trained by a number of real user profiles is also used to generate attack profiles [29, 30]. This kind of attack may not be realistic in some applications, as obtaining detailed information about recommender systems may pose a challenge. Consequently, this study specifically addresses attacks with low prior knowledge.

2.2 Shilling attack detection

Attack profiles are separated from genuine profiles by shilling attack detection according to the rating behavior or item distribution of the profiles [5, 10, 11]. A number of features are proposed to quantify characteristics of a profile, for example, Rating Deviation from Mean Agreement (RDMA) [8], Weighted Deviation from Mean Agreement (WDMA) [9], Weighted Degree of Agreement (WDA) [9] and Degree of Similarity with Top Neighbors (DegSim) [9]. Some features [12, 13, 14] related to item popularity are extracted to capture the difference between item distributions of normal and attack profiles. Additionally, certain features, e.g., expanded RDMA [16] and WDMA [31], incorporate temporal information of a profile to quantify the dynamic rating changes.

A number of supervised learning methods, e.g., the Neural Network [23, 32, 21], Support Vector Machine [31, 33], AdaBoost [11], k-Nearest Neighbourhood [16], Decision Tree [13, 14] and ensemble methods [13], with different kinds of features are used to identify attack profiles. Generally a supervised learning method is able to capture the difference between attack and normal profiles accurately. However, its discrimination ability usually cannot be generalized well to unseen attacks [8, 9]. Moreover, labeling profiles is costly and timely [17, 34]. On the other hand, no label is required by unsupervised detection methods which analyze the characteristics of different groups of profiles [5, 18, 19, 20, 35, 36]. For example, the high correlation of attack profiles [19, 35], highly relevant of target item rating sequence [20], long-term correlation and stable preference [18], correlation of user rated items in the time dimension [5], rating distribution of attack users [36]. A few studies focus on semi-supervised detection methods [37, 38]. A classifier is first trained on the labeled data, and then is updated by considering additional unlabeled data iteratively in these methods. For example, the probability of unlabeled data is estimated by a Naïve Bayes model trained by using labeled data and updates the model iteratively [38]. However, this kind of methods may performs badly due to uncertain information, especially for a small amount of labeled data [23].

3. Proposed method

The divide-and-conquer approach is helpful in detecting attacks for several reasons. Firstly, the selection of filler items varies across different attack models, with random or popular items being chosen. This distinction allows the divide-and-conquer approach to effectively detect attacks based on the specific selection of filler items. Additionally, current detection techniques often rely on investigating shared traits among different attack models, resulting in complex and computationally demanding models. In contrast, the divide-and-conquer approach simplifies the process by extracting unique characteristics for each attack type, eliminating the need for complex models. This approach prevents the issue of characteristics becoming noisy or irrelevant when applied to different attack models. Moreover, by focusing on specific characteristics for each attack type, the divide-and-conquer approach avoids overfitting and enhances the interpretability of the detection models. Consequently, our study devises a divide-and-conquer method, which is an unsupervised detection method without any prior knowledge. The framework of the proposed method is illustrated in Fig. 1. The entire process can be separated into three phases, including attack type classification, suspected profile identification and attack profile refinement. The rating behaviors of obfuscated behavior attacks are much more similar as the genuine users. In the first phase (Section 3.1), our method categorizes the attack model as either standard or obfuscated behavior attacks based on the way of selecting rated items, item popularity distribution and mean popularity of each user. Two features, namely Estimated Mean Popularity of a User ( $\widehat{\textit{MPU}}$ ) [22] and Estimated Weighted Deviation from Mean Agreement ( $\widehat{\textit{WDMA}}$ ) [9] are calculated depending on the attack types. The second phase (Section 3.2) employs a denoising based clustering algorithm, i.e., OPTICS [39], to identify suspicious profiles. In the final phase (Section 3.3), the attack profiles are determined according to the target item analysis.

Figure 1.

The framework of our proposed shilling attack detection method containing three phases including attack type classification, suspected profile identification and attack profile identification.

3.1 Attack type classification

Given a rating matrix $\mathcal{R}$ , containing a set of user profiles, our model first classifies the type of contamination. Standard behavior attacks, e.g., Random Attack [21, 25] and Average Attack [21, 25], rate items randomly, while the rating pattern of obfuscated behavior attacks, e.g., Power Item Attack [21], is similar to genuine profiles. As a result, these two attacks can be distinguished by analyzing the distribution of the rated items. Our model measures the distribution of rated items by Mean Popularity of a User (MPU) [22], which quantifies the average item popularity of a user. The calculation of $\textit{MPU}_{u}$ for user $u$ is defined as follows:

$\displaystyle\textit{MPU}_{u}=\frac{\sum_{i=1}^{n}d_{i}}{n}$ (1)

where $n$ denotes the number of items user $u$ has rated, while $d_{i}$ represents the popularity of $i^{\text{th}}$ item defined as the number of users in $\mathcal{R}$ rating the $i^{\text{th}}$ item. We further define $I^{\mathcal{R}}_{s,e}$ to measure the percentage of users in $\mathcal{R}$ whose MPU in the interval $(s,e]$ .

$\displaystyle I^{\mathcal{R}}_{s,e}=\frac{\sum_{u\in\mathcal{R}}c_{u}(s,e)}{|% \mathcal{R}|}$ (2)

where $c_{u}(s,e)$ indicates whether $\textit{MPU}_{u}$ is in $(s,e]$ , i.e., if $s\leqslant\textit{MPU}_{u}<e$ , $c_{u}(s,e)=1$ ; otherwise, $c_{u}(s,e)=0$ .

The differences of the MPU distributions of contaminated and benign profiles are illustrated in Fig. 2. The dataset MovieLens-100K¹ and Netflix² are chosen as an example. The range [ $\mu-4\sigma$ , $\mu+4\sigma$ ] is separated into k intervals, where $\mu$ and $\sigma$ denote the mean and standard deviation of MPU respectively. The profiles are contaminated by seven popular attack models, including Random Attack [21, 25], Average Attack [21, 25], Bandwagon Attack [25], Target Shift Attack [28], Noise Injection Attack [28], AoP Attack [21, 25], and Power Item Attack [21], with 5% attack size and 5% filler size. AoP Attack and Power Item Attack belong to obfuscated behavior attacks, the rest of them belong to standard behavior attacks.

Figure 2.

MPU distributions of movielens-100k and netflix with different attack models. The x-axis represents a MPU value while the y-axis indicates its frequency. The significant difference between the attack and clean profile sets are highlight by a red box.

When there is no attack, it can be observed that MPU for genuine profiles follows the normal distribution [22], as illustrated in the initial graph in first column of Fig. 2. The x-axis represents MPU values while the y-axis indicates their frequency. For standard behavior attacks, where rated items are randomly selected, MPU values for the attack profiles tend to be smaller than genuine profiles. As a result, a small crest can be found in the histogram when MPU values are low. Conversely, obfuscated behavior attacks predominantly rate popular items to simulate the rating behavior of genuine users. In the case of the AoP attack, which randomly selects popular items, the MPU distribution essentially follows a normal distribution. However, for the Power Item Attack (PIA), where the most popular items are rated, a sharp increase can be observed when the MPU value is large. The change of the distribution caused by obfuscated behavior attacks is less obvious than that resulting from standard behavior attacks. According to Eq. (3), a red box is assigned when the relative difference between the output and a normal distribution, denoted as $\textit{Dev}_{s,e}^{A,B}$ , exceeds the threshold $\theta$ . For Netflix under the AoP attack and Power item attack, noticeable deviations are observed when compared to the original data. This indicates that the value of $I_{s,e}^{A}$ is anomalous. However, as $\textit{Dev}_{s,e}^{A,B}$ remains below the threshold $\theta$ , no red box is marked around the atypical value of $I_{s,e}^{A}$ . Netflix’s dataset is considerably sparser than MovieLens. In AoP attack, the item popularity $d_{i}$ is heightened for the specifically chosen popular items, potentially creating an illusion of a smaller MPU when these popular items receive little ratings from genuine users. Abnormal MPU values for Standard Behavior Attacks are all below 200, whereas the abnormal MPU values in AoP attacks exceed 200. Consequently, an abnormal increase is observed on the left side of the Aop attacks.

By observing that the low rating items of the MPU distribution for profiles contaminated by standard behavior attack deviate from the Gaussian distribution, the given rating matrix ( $\mathcal{R}$ ) is classified as two classes using Algorithm 2, i.e., Standard and Obfuscated Behavior Attacks, according to the deviation of the low rating items in the MPU distribution from the Gaussian Distribution ( $\mathcal{N}$ ). The mean and standard deviation of $\mathcal{N}$ are set to be the same as the ones of $\mathcal{R}$ . The deviation $\textit{Dev}^{\mathcal{A},\mathcal{B}}_{s,e}$ between two distributions ( $\mathcal{A}$ and $\mathcal{B}$ ) in $(s,e]$ is measured as:

$\displaystyle\textit{Dev}^{\mathcal{A},\mathcal{B}}_{s,e}=\frac{\lvert I^{% \mathcal{A}}_{s,e}-I^{\mathcal{B}}_{s,e}\rvert}{I^{\mathcal{B}}_{s,e}+\varepsilon}$ (3)

where $\varepsilon$ is applied to avoid mutational results caused by a smaller value.

The interval $(s,e]$ is partitioned into $k$ intervals for investigation, each with a size of $(s-e)/k$ . The $i^{\text{th}}$ interval $(s_{i},e_{i}]$ is deemed abnormal when $\textit{Dev}^{\mathcal{R},\mathcal{N}}_{s_{i},e_{i}}>\theta$ , where $i=1,2,\ldots k$ and $\theta$ is a threshold representing the sensitivity of our detection. A larger $k$ value results in smaller intervals, offering precise representation of regions. However, this information maybe less convincing as only a few profiles will fall within the small intervals. Conversely, a smaller $k$ value leads to larger intervals. While the information is more convincing, the interval represents a broad region. The rating matrix $\mathcal{R}$ is classified as Standard Behavior Attacks if and only if all its abnormal intervals are located in the first half of the distribution. Otherwise, it is categorized as Obfuscated Behavior Attack. The detailed procedures are outlined in Algorithm 2.

: Determine Attack Type[1] $\mathcal{R}$ : a rating matrix; AttType: standard behavior attack or obfuscated behavior attack; $n\leftarrow$ number of items in $\mathcal{R}$ $i=1\to n$ $d_{i}\leftarrow i^{\text{th}}$ item popularity of $\mathcal{R}$ $\textit{MPU}_{u}\leftarrow$ average rated item popularity of user $u$ by Eq. (1) $i=1\to k$ $I^{\mathcal{R}}_{s_{i},e_{i}}\leftarrow$ the percentage of users with MPU in the interval $(s_{i},e_{i}]$ by Eq. (2) $\textit{Dev}^{\mathcal{R},\mathcal{N}}_{s_{i},e_{i}}\leftarrow$ deviation to the Gaussian Distribution by Eq. (3) $\textit{Dev}^{\mathcal{R},\mathcal{N}}_{s_{i},e_{i}}>\theta$ ${\textit{flag}}_{i}\leftarrow 1$ ${\textit{flag}}_{i}\leftarrow 0$ $\forall i(\textit{flag}_{i}=1$ implies $i\leqslant k/2$ ) $\textit{AttType}=$ Standard behavior attack $\textit{AttType}=$ Obfuscated behavior attack ReturnAttType

3.2 Suspected profile identification

Our model identifies suspected profiles through clustering in feature spaces specially designed for Standard and Obfuscated Behavior Attacks separately. The feature extraction method is thoroughly discussed for each type of attack.

3.2.1 Identification on profiles with standard behavior attack

Given that genuine users usually rate items according to their preferences, the popularity of items tends to follow a long tail distribution [22]. This characteristic may differ in Standard Behavior Attack where items are randomly rated in attack profiles [22]. As discussed in the previous section, MPU can be employed to distinguish attack and genuine profiles. However, the method only can determine whether a dataset is contaminated by Standard Behavior Attack, and cannot precisely identify the attack profiles due to absence of real user behavior.

Our model estimates the genuine MPU distribution by employing PCA-VarSelect [19] to filter out the obvious attack profiles. PCA-VarSelect is chosen for its excellent detection performance when the number of attack profiles is known. As the contaminated samples are usually less than 50% of the collected profiles [11] in reality, we select the top 50% unlikely attack profiles, contained in $\hat{\mathcal{R}}$ , to approximate the real distribution according to Eq. (1). This percentage can be adjusted according to the size of profiles in a given application. $\widehat{\textit{MPU}}_{u}$ is calculated using item popularity, i.e., $\hat{d}_{i}$ , where $i=1,2,\ldots n$ , of $\hat{\mathcal{R}}$ according to Eq. (1), which represents user $u$ .

The profiles are subsequently divided into two distinct clusters using the OPTICS clustering algorithm [39] in the space of $\widehat{\textit{MPU}}$ . In our study, OPTICS, a density-based clustering algorithm, is employed due to its insensitivity to initial parameters and resistance to outliers. The smaller cluster is identified as the attack class, assuming that the size of attack profiles is smaller.

The concept of the proposed feature extraction is exemplified using MovieLens-100K¹. We simulate a shilling attack on MovieLens-100K through average attack with 5% attack size and 5% filler size. The resulting MPU distribution is depicted in Fig. 3a, where red and blue indicate attack and clean profiles respectively. After removing uncertain profiles using PCA-VarSelect, the updated MPU distribution is shown in Fig. 3b. The distribution of $\widehat{\textit{MPU}}$ for all profiles is illustrated in Fig. 3c, revealing an obvious boundary between attack and clean profiles, roughly located at MPU $=$ 65.

Figure 3.

The MPU distributions of the profiles in different situations.

Figure 4.

The WDMA distribution of the profiles under different situations. The x-axis represents a WDMA value ( $\times$ 10,000) while the y-axis shows its frequency.

3.2.2 Identification on profiles with obfuscated behavior attack

In obfuscated behavior attacks, popular items are rated, making the attacks highly similar to genuine profiles. Our method detects obfuscated behavior attacks according to Weighted Deviation from Mean Agreement (WDMA) [9], which measures the average deviation degree of rating value, defined as follows:

$\displaystyle\textit{WDMA}_{u}=\frac{1}{|\mathcal{R}_{u}|}\sum_{j\in\mathcal{R% }_{u}}\frac{|r_{u,j}-\bar{r}_{j}|}{|I_{j}|^{2}}$ (4)

where $\mathcal{R}_{u}$ indicates the set of items rated by user $u$ , $|I_{j}|$ denotes the number of users rated item $j$ respectively. $r_{u,j}$ is the rating of $u$ on $j$ , and $\bar{r}_{j}$ is the average ratings of the rated item $j$ provided by all users.

Similar to the rationale for Standard Behavior Attack identification, the top 50% evidently clean samples are used to approximate the real clean profiles. The 50% of samples with smallest WDMA values are excluded, and the remaining samples are used to calculate $\hat{\mathcal{R}_{u}}$ . $\widehat{\textit{WDMA}}_{u}$ calculated by $\hat{\mathcal{R}_{u}}$ is used to represent user $u$ . The profiles are then separated into two groups according to the clustering method mentioned in Section 3.2.1 in the space of $\widehat{\textit{WDMA}}$ . Similarly, the smaller cluster is assumed to be the contaminated class.

MovieLens-100K is utilized as an example to illustrate the proposed feature extraction for Obfuscated Behavior Attack. AoP attack with 5% attack size and 5% filler size is injected to the dataset, and its WDMA distribution is shown in Fig. 4a, where red and blue indicate attack and clean profiles. Attack profiles exhibit small WDMA values and mix with the clean profiles. Figure 4b displays the distribution of WDMA after removing uncertain profiles, and Fig. 4c illustrates the distribution of $\widehat{\textit{WDMA}}$ of $\mathcal{R}$ . The attack profiles are clearly separated from the clean ones in the domain of $\widehat{\textit{WDMA}}$ , with the threshold roughly equal to 31.

3.3 Attack profile refinement

To enhance the accuracy of attack profile identification, all suspected profiles are filtered according to the target items of the attack. As the attack aims to manipulate the recommendation frequency on the target item, the standard deviation of target items on attack profiles should be small. The precision of attack detection is heightened by analyzing the standard deviation of an item in suspected profiles identified in the previous phase. Assuming the rated value to be between 1 to 5, the suspicion of the $i^{\text{th}}$ item is quantified as:

$\displaystyle\text{sus}_{i}=\frac{\max_{r=\{1,2,4,5\}}(n(i,r))}{e^{\textit{std% }(i)}}$ (5)

where $n(i,r)$ denotes the number of users who rate the $i^{\text{th}}$ item with a rating value $r$ in the suspected cluster. $r=3$ is ignored since the rating is neutral and will not significantly affect the recommendation frequency. $\textit{std}(i)$ represents the standard deviation of the $i^{\text{th}}$ item’s rating values. The exponential function is adopted to increase the difference between the target and non-target items.

The item(s) with the maximum rating suspicion score will be identified as the target items. Subsequently, push and nuke attack are identified according to the rating values of the suspected target items, i.e., if the values are high, it is the push attack; otherwise, it is a nuke attack. A profile in the suspected cluster that rates a high (low) value in the suspected target items of the push (nuke) attack is regarded as the attack profile.

4. Experiments

In this section, experiments are carried out to evaluate and compare the performance of the proposed method with State-of-the-Art (SOTA) methods.

4.1 Experimental settings

Two real datasets, MovieLens-100K¹ and Netflix², are used in the experiments. MoivesLens-100K consists of 100,000 ratings from 943 users on 1,682 movies, with all users providing ratings for at least 20 movies. Netflix includes 103,297,638 ratings on 17,770 movies by 480,189 users. For our experiment, a subset of the dataset is randomly selected, containing 280,015 ratings on 4,000 movies from 2,000 users. In both datasets, each movie is rated as 1 to 5, where 5 indicate the user likes the movie the most and 1 indicate the least. Ten independent runs are carried out to obtain more credible results.

All user profiles in the datasets are assumed as genuine profile. Shilling attack profiles are generated by the attack models, including Random Attack [25, 21], Average Attack [25, 21], Bandwagon Attack [25], Target Shift Attack [28], Noise Injection Attack [28], AoP Attack [25, 21], and Power Item Attack (PIA) [21]. The target items are randomly selected from the items whose average rating number is less than four to generate multi-target items attacks. Only one kind of attack is injected in each experiment. For each attack, three target items are chosen. The attack size is set as 3%, 5%, 10%, and 15%, and the filler size is set as 3%, 5%, and 10%. Filler items in AoP attacks are randomly selected from the top 30% of the popular items.

Three SOTA methods are considered in our experiment. One unsupervised detection method, PCA-VarSelect [19], and two supervised detection methods, namely Pop-SAD [22] and SpDetector [23], are compared with our proposed method. The popularity distribution is conducted solely in Pop-SAD, while our proposed method considers the popularity distribution and also the average deviation degree of rating value. SpDetector is more complex as it requires the construction of user and item hypergraphs and the extraction of spectral features from these hypergraphs. Since the performance of PCA-VarSelect is highly relied on the attack size, we assume this prior information is known by PCA-VarSelect but not by our method, Pop-SAD, and SpDetector. On the other hand, we assume Pop-SAD and SpDetector obtain the attack type and its samples. To be more specific, 80% of the dataset is selected as the training set for Pop-SAD and SpDetector, which is not provided to our method and PCA-VarSelect. Consequently, our method requires the least prior knowledge. F-measure is calculated to quantify the performance.

4.2 Parameter settings

Experiments are conducted to investigate the impact of hyper-parameters $k$ and $\theta$ in Algorithm 2. It has been observered that $\textit{MPU}_{u}$ of genuine users follows a normal distribution, and the mean $\mu$ and standard variation $\sigma$ of $\textit{MPU}_{u}$ for all users are calculated. The probability of $\textit{MPU}_{u}$ falling within the interval ( $\mu-4\sigma$ , $\mu+4\sigma$ ] is 99.99%, so the interval ( $\mu-4\sigma$ , $\mu+4\sigma$ ] is divided into k sub-intervals, i.e., $s$ and $e$ take values of $\mu-4\sigma$ and $\mu+4\sigma$ respectively. $\varepsilon$ is applied to avoid mutational results caused by a smaller denominator, the probability of samples following a normal distribution in the internal [ $-$ 4, $-$ 3] or (3, 4] is 0.13%, and to reduce the impact on normal outcomes, $\varepsilon$ is set to 0.001. According to the Freedman-Diaconis principle, k should take a value around 12 (16) for MNIST (Netflix). While for the SquareRoot principle, k should take a value around 32 (48) for MNIST (Netflix). Hence, parameter $k$ is set as 12, 17, 22, 27 and 32 respectively for MNIST, correspondingly, parameter $k$ is set as 16, 24, 32, 40, 48 for Netflix. Based on the value of $\textit{Dev}^{\mathcal{A},\mathcal{B}}_{s,e}$ , $\theta$ is set value between 0 and 10.

To choose suitable hyperparameters, some rating matrixs with different attacks are generated. Each rating matrix is considered as a sample, and its features are constructed by $I^{\mathcal{R}}_{s_{i},e_{i}}$ , with the output of Algorithm 2 denoted as its cluster label. Calinski-Harabasz Criterion is used to evaluate the effectiveness of the hyper-parameters. The grid search is applied to select the appropriate hyper-parameters. The experimental results are depicted in Fig. 5, in which darker red points with a larger area indicate a higher Calinski-Harabasz value. This implies a more proper determination of attack types by the hyper-parameters. It can be observerd that $\theta=1$ for MNIST yields a better Calinski-Harabasz value, and $k$ has little impact on the results. Therefore, $k$ is set as 32. On the other hand, for Netflix, a better Calinski-Harabasz value can be obtained when $\theta=3$ and $k=16$ . The main reason is that Netflix is sparser than MNIST, making it more sensitive to the values of $\theta$ and $k$ . Hence, for our method, according to the experimental results, the region [ $\mu-4\sigma$ , $\mu+4\sigma$ ] is considered and the whole region is separated into 32 and 16 partitions for MNIST and Netflix respectively, $\theta$ takes the values 1 and 3 accordingly.

Figure 5.

Accuracy in MovieLens-100k and Netflix with different hyper parameters in Algorithm 1.

The hyperparameters for Pop-SAD and SpDetector are solely determined by the number of intervals ( $k$ ) and the dimension of the spectral feature ( $K$ ) respectively. The original papers of these methods have extensively studied and recommended specific hyperparameter settings, suggesting that $k=6$ and $K=50$ . In our experiments, we have strictly adhered to the settings proposed in their original papers.

4.3 Comparison with SOTA methods

The experimental results between the proposed method and Pop-SAD, SpDetector and PCA-VarSelect under various attacks with 5% attack size and 5% filler size are presented in Tables 1 and 2. Despite our method requiring no prior knowledge of the attack and no label information, it still achieves the best detection accuracy and stability, i.e., the highest average and the lowest standard deviation of F-measure, among all the methods in standard behavior attacks. For obfuscated behavior attack PIA, the accuracy of our method (i.e., 99.0% and 99.5%) is insignificantly lower than 100% achieved by two supervised detection methods, Pop-SAD and SpDetector, for both datasets. The overall performance of SpDetector is close to our method. However, our method possesses the advantage of not necessitating any annotations during training. On the other hand, another unsupervised method, PCA-VarSelect, is less accurate than our method in detecting the standard behavior attack, even although PCA-VarSelect is also used in our method. These results explain that the approximation on the genuine users’ distribution by using 50% profiles of our method benefits the detection process. Obtaining more the prior knowledge of the attack is evidently helpful in detection. PCA-VarSelect only outperforms Pop-SAD in detecting the standard behavior attack in MovieLens-100k. In other cases, Pop-SAD performs much better than PCA-VarSelect. Netflix’s dataset is sparser compared to MovieLens, which impacts the effectiveness of certain attack detection methods. Specifically, AoP attacks target popular items, but due to the subtle difference in MPU values between legitimate and attack profiles within the Netflix dataset, using popularity distribution as a detection measure is less effective. Consequently, Pop-SAD shows diminished performance on the Netflix dataset. Conversely, with other standard behavior attacks where items are rated randomly, this increases the popularity of each item and the inherent sparsity of the dataset actually enhances the ability to detect attacks through popularity distribution analysis. As more complex hypergraph spectral features are extracted by SpDetector, it generally performs better than Pop-SAD and PCA-VarSelect.

Table 1
F-measure (%) of the compared results in MovieLens-100k with 5% attack size and 5% filler size

Detection method	Standard behavior attack					Obfuscated behavior attack
	Random	Average	Bandwagon	AvgTargetShift	AvgNoiseInject	AoP	PIA
Supervised
Pop-SAD	96.9 ${\pm}$ 3.6	93.7 ${\pm}$ 3.3	93.3 ${\pm}$ 4.2	94.8 ${\pm}$ 3.3	94.9 ${\pm}$ 4.6	99.4 ${\pm}$ 1.9	100 ${\pm}$ 0.0
SpDetector	99.9 ${\pm}$ 0.2	99.7 ${\pm}$ 0.3	99.9 ${\pm}$ 0.2	99.4 ${\pm}$ 0.6	99.7 ${\pm}$ 0.4	99.8 ${\pm}$ 0.3	100 ${\pm}$ 0.0
Unsupervised
PCA-VarSelect	97.0 ${\pm}$ 1.1	96.0 ${\pm}$ 1.6	95.5 ${\pm}$ 1.2	96.6 ${\pm}$ 1.5	96.2 ${\pm}$ 1.7	19.8 ${\pm}$ 3.0	0.0 ${\pm}$ 0.0
ProMethod	100 ${\pm}$ 0.0	100 ${\pm}$ 0.0	99.9 ${\pm}$ 0.3	100 ${\pm}$ 0.0	100 ${\pm}$ 0.0	99.0 ${\pm}$ 0.3	99.0 ${\pm}$ 0.3

Table 2

F-measure (%) of the compared results in Netflix with 5% attack size and 5% filler size

Detection method	Standard behavior attack					Obfuscated behavior attack
	Random	Average	Bandwagon	AvgTargetShift	AvgNoiseInject	AoP	PIA
Supervised
Pop-SAD	98.5 $\pm$ 1.3	99.0 $\pm$ 1.3	98.5 $\pm$ 1.3	98.5 $\pm$ 1.3	99.0 $\pm$ 1.3	88.9 $\pm$ 4.3	100 $\pm$ 0.0
SpDetector	99.9 $\pm$ 0.2	99.9 $\pm$ 0.1	99.9 $\pm$ 0.2	99.9 $\pm$ 0.1	99.9 $\pm$ 0.2	99.6 $\pm$ 0.3	99.9 $\pm$ 0.1
Unsupervised
PCA-VarSelect	98.5 $\pm$ 0.7	97.2 $\pm$ 0.8	97.0 $\pm$ 1.2	97.4 $\pm$ 1.0	97.0 $\pm$ 1.1	68.3 $\pm$ 1.9	0.0 $\pm$ 0.0
ProMethod	100 $\pm$ 0.0	100 $\pm$ 0.0	100 $\pm$ 0.0	100 $\pm$ 0.0	100 $\pm$ 0.0	99.5 $\pm$ 0.0	99.5 $\pm$ 0.0

4.4 Performance in different attack parameters

The detection performance on attacks with different attack size and average filler size in terms of average F-measure of MovieLens-100k and Netflix is shown in Fig. 6. The results suggest that the performance of all detection method generally increases with the increase of attack sizes, aligning with many studies, as more information about the attack is provided. There is no doubt that the proposed method and SpDetector perform the best among all methods, i.e., their F-measures are almost equal to 100% in all scenarios. As our method does not require any label information, it is more efficient than SpDetector. PCA-VarSelect cannot effectively handle obfuscated behavior attacks since the rated items, mostly selected from popular items, are highly correlated to genuine profiles. The F-measure of PCA-VarSelect decreases when filler size increases, as the difference between genuine profiles and attack profiles is reduced. The performance of Pop-SAD improves with the increase of filler size and attack size, Pop-SAD is able to recognize the attacks effectively when the attack characteristics are obvious. Hence, Pop-SAD performs well in the identification of obfuscated behavior attacks, as the accumulated MPU probability for some intervals of obfuscated behavior attack profiles is noticeably larger than genuine profiles.

Figure 6.

Average F-measure (%) in MovieLens-100k and Netflix with different attack and filler sizes.

Table 3

Running time (s) of the detection method in MovieLens-100k and Netflix with different attack sizes

DataSet	Detection method		Attack size
			3%	5%	10%	15%
MovieLens-100k	Supervised	Pop-SAD	0.718 $\pm$ 0.062	0.733 $\pm$ 0.042	0.741 $\pm$ 0.047	0.737 $\pm$ 0.055
		SpDetector	12.582 $\pm$ 1.184	12.982 $\pm$ 1.235	14.076 $\pm$ 1.367	14.818 $\pm$ 1.332
	Unsupervised	PCA-VarSelect	0.260 $\pm$ 0.094	0.279 $\pm$ 0.074	0.317 $\pm$ 0.076	0.326 $\pm$ 0.091
		ProMethod	0.299 $\pm$ 0.132	0.313 $\pm$ 0.134	0.342 $\pm$ 0.138	0.381 $\pm$ 0.157
Netflix	Supervised	Pop-SAD	0.495 $\pm$ 0.032	0.499 $\pm$ 0.027	0.511 $\pm$ 0.047	0.54 $\pm$ 0.036
		SpDetector	32.001 $\pm$ 2.177	33.011 $\pm$ 2.741	35.397 $\pm$ 2.727	38.127 $\pm$ 3.887
	Unsupervised	PCA-VarSelect	1.639 $\pm$ 0.239	1.698 $\pm$ 0.222	1.945 $\pm$ 0.392	2.28 $\pm$ 0.331
		ProMethod	1.541 $\pm$ 0.696	1.614 $\pm$ 0.73	1.806 $\pm$ 0.863	2.006 $\pm$ 0.935

4.5 Running time

The running time (in seconds) of our method and others in MovieLens-100k and Netflix with different attack sizes is illustrated in Table 3. Due to the larger size of Netflix comparing to MovieLens-100k, all methods except Pop-SAD require a longer detection time. This is because the probability is calculated much faster due to the sparsity of Netflix. SpDetector uses dramatically longer time than the other methods due to its complicated feature extraction from a deep neural network. As a result, although the accuracy achieved by our method and SpDetector is similar, as shown in previous sections, the time complexity of our method is much lower than SpDetector. Moreover, the running time of our method is similar to PCA-VarSelect and much shorter than Pop-SAD.

5. Conclusion

In this paper, we propose an unsupervised divide-and-conquer approach for detecting shilling attacks. Our model, requiring no prior knowledge of the attacks, begins by classifying contaminated profiles into Standard or Obfuscated Behavior Attacks. These profiles are then clustered using the OPTICS algorithm based on $\widehat{\textit{MPU}}$ and $\widehat{\textit{WDMA}}$ for the respective attack types. The suspect profile set is further refined through an analysis of the target items. Experimental results demonstrate that our method is robust against various common attack methods and parameters, achieving comparable performance to supervised methods like SpDetector, but with significantly shorter running times. Our study establishes a versatile framework for the detection of shilling attacks.

One possible future work is to address hybrid attack identification, detecting the amalgamation of suspect profiles within the same attack type but with different parameters. The challenge of identifying multiple attackers employing varying strategies will also be considered. Mixed-strategy attacks tend to obscure characteristic signatures, which may necessitate the use of data augmentation to reinforce these signatures and improve identification capabilities. Moreover, the issue of reduced characteristic clarity in such complex attack scenarios warrants further investigation, potentially leading to the development of more sophisticated detection methods.

Footnotes

https://grouplens.org/datasets/movielens/.

http://netflixprize.com/.

Acknowledgments

This paper is supported by the Henan Provincial Science and Technology Research Project (No.222102210258) and Key Platform, Research Project of Education Department of Guangdong Province (No.2020KTSCX132).

References

Jalili

Ahmadian

Izadi

Moradi

and Salehi

, Evaluating collaborative filtering recommender algorithms: A survey, IEEE Access 6 (2018), 74003–74024.

Mohamed

M.H.

Khafagy

M.H.

and Ibrahim

M.H.

, Recommender Systems Challenges and Solutions Survey, in: 2019 International Conference on Innovative Trends in Computer Engineering (ITCE), 2019, pp. 149–155.

Esmaeili

Mardani

A.H.G.

and Madar

Z.Z.

, A novel tourism recommender system in the context of social commerce, Expert Systems with Applications 149 (2020).

Zhang

Yao

Sun

and Tay

, Deep Learning Based Recommender System: A Survey and New Perspectives, ACM Computing Surveys, 2017.

Cai

and Zhang

, An unsupervised method for detecting shilling attacks in recommender systems by mining item relationship and identifying target items, The Computer Journal 62(4) (2019), 579–597.

and Li

, Shilling attacks against collaborative recommender systems: A review, Artificial Intelligence Review: An International Science and Engineering Journal 53(1) (2020), 291–319.

Mobasher

Burke

R.D.

Bhaumik

and Williams

, Toward trustworthy recommender systems: An analysis of attack models and algorithm robustness, ACM Transactions on Internet Technology (TOIT), 2007.

Chirita

P.A.

Nejdl

and Zamfir

, Preventing shilling attacks in online recommender systems, in: Seventh ACM International Workshop on Web Information and Data Management (WIDM 2005), 2005.

Burke

Mobasher

Williams

and Bhaumik

, Classification Features for Attack Detection in Collaborative Recommender Systems, in KDD ’06, 2006, pp. 542–547.

10.

Yang

and Cai

, Detecting abnormal profiles in collaborative filtering recommender systems, Journal of Intelligent Information Systems 48(3) (2017), 499–518.

11.

Yang

Cai

and Xu

, Re-scale AdaBoost for attack detection in collaborative filtering recommender systems, Knowledge-Based Systems, 2016, 74–88.

12.

Zhang

and Zhou

, HHTâ€“SVM: An online method for detecting profile injection attacks in collaborative recommender systems, Knowledge-Based Systems 65(JUL.) (2014), 96–105.

13.

Zhang

and Chen

, An ensemble method for detecting shilling attacks based on ordered item sequences, Security and Communication Networks 9(7) (2016), 680–696.

14.

Wen-Tao

Min

Hua

Qing-Yu

Jun-Hao

and Bin

, An shilling attack detection algorithm based on popularity degree features, Acta Automatica Sinica 41(9) (2015), 1563–1576.

15.

Yang

Sun

Liu

Yan

and Zhang

, Rating behavior evaluation and abnormality forensics analysis for injection attack detection, Journal of Intelligent Information Systems 59(1) (2022), 93–119.

16.

Hao

Zhang

and Zhang

, Multiview ensemble method for detecting shilling attacks in collaborative recommender systems, Security and Communication Networks 2018 (2018), 1–33.

17.

Gao

Huang

and Sha

, Shilling Attack Detection Scheme in Collaborative Filtering Recommendation System Based on Recurrent Neural Network, in: Advances in Information and Communication, 2020, pp. 634–644.

18.

Cai

and Zhang

, BS-SC: An Unsupervised Approach for Detecting Shilling Profiles in Collaborative Recommender Systems, IEEE Transactions on Knowledge and Data Engineering, 2019.

19.

Mehta

and Nejdl

, Unsupervised strategies for shilling detection and robust collaborative filtering, User Modeling and User-Adapted Interaction 19 (2009), 65–97.

20.

Zhang

and Wang

, UD-HMM: An unsupervised method for shilling attack detection based on hidden Markov model and hierarchical clustering, Knowledge-Based Systems 148 (2018), 146–166.

21.

Hao

and Zhang

, An unsupervised detection method for shilling attacks based on deep learning and community detection, Soft Computing, 2021, 477–494.

22.

Gao

Zeng

Xiong

and Hirokawa

, Shilling attack detection in recommender systems via selecting patterns analysis, IEICE Transactions on Information & Systems 99(10) (2016), 2600–2611.

23.

Zhou

M.G.F.

Wang

Fan

and Yang

, Fusing hypergraph spectral features for shilling attack detection, Journal of information security and applications, 2021, 103051:1–10.

24.

Huang

Gong

N.Z.

and Xu

, Data Poisoning Attacks to Deep Learning Based Recommender Systems, in: Proceedings 2021 Network and Distributed System Security Symposium, 2021.

25.

Rezaimehr

and Dadkhah

, A survey of attack detection approaches in collaborative filtering recommender systems, Artificial Intelligence Review: An International Science and Engineering Journal 54 (2021), 2011–2066.

26.

Chen

Chan

P.P.K.

Zhang

and Li

, Shilling attack based on item popularity and rated item correlation against collaborative filtering, International Journal of Machine Learning and Cybernetics 10(7) (2019), 1833–1845.

27.

Turk

A.M.

and Bilge

, Robustness analysis of multi-criteria collaborative filtering algorithms against shilling attacks, Expert Systems with Application 115(JAN.) (2019), 386–402.

28.

Williams

Mobasher

Burke

Sandvig

and Bhaumik

, Detection of obfuscated attacks in collaborative recommender systems, in: Proceedings of the 17th European Conference on Artificial Intelligence, 2006, pp. 19–23.

29.

Lin

Chen

Xiao

and Yang

, Attacking Recommender Systems with Augmented User Profiles, in: Conference on Information and Knowledge Management (CIKM), 2020, pp. 855–864.

30.

Wang

Gao

Zhang

and Zhong

, Gray-box shilling attack: An adversarial learning approach, ACM Transactions on Intelligent Systems and Technology (TIST) 13(5) (2022), 82:1–82:21.

31.

and Zhang

, Detecting shilling attacks in social recommender systems based on time series analysis and trust features, Knowledge-Based Systems 178 (2019), 25–47.

32.

Hao

Zhang

Wang

Zhao

and Cao

, Detecting Shilling Attacks with Automatic Features from Multiple Views, Security and Communication Networks, 2019, 1–13.

33.

Zhou

Wen

Xiong

Gao

and Zeng

, SVM-TIA a shilling attack detection method based on SVM and target item analysis in recommender systems, Neurocomputing 210(Oct.19) (2016), 197–205.

34.

Cai

and Zhu

, Trustworthy and profit: A new value-based neighbor selection method in recommender systems under shilling attacks, Decision Support Systems 124(Sep.) (2019), 113112.1–113112.15.

35.

Huawei

Zaiseng

Fuzhi

Xiaohui

and Yajun

, Robust recommendation algorithm based on kernel principal component analysis and fuzzy c-means clustering, Wuhan University Journal of Natural Sciences 23(2) (2018), 111–119.

36.

Cai

and Zhang

, Detecting shilling attacks in recommender systems based on analysis of user rating behavior, Knowledge Based Systems 177(AUG.1) (2019), 22–43.

37.

Zhou

and Duan

, Semi-supervised recommendation attack detection based on Co-Forest, Computers & Security, 2021, 102390.

38.

Cao

and Tao

, HySAD: a semi-supervised hybrid shilling attack detector for trustworthy product recommendation, in: Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2012, pp. 985–993.

39.

Bhattacharjee

and Mitra

, A survey of density based clustering algorithms, Frontiers of Computer Science 15(1) (2021), 151308.

Unsupervised contaminated user profile identification against shilling attack in recommender system

Abstract

Keywords

1. Introduction

2. Related work

2.1 Shilling attack model

2.2 Shilling attack detection

3. Proposed method

3.2.1 Identification on profiles with standard behavior attack

4.1 Experimental settings

4.2 Parameter settings

Table 1 F-measure (%) of the compared results in MovieLens-100k with 5% attack size and 5% filler size

5. Conclusion

Footnotes

Acknowledgments

References

Table 1
F-measure (%) of the compared results in MovieLens-100k with 5% attack size and 5% filler size