Privacy preserving defect prediction using generalization and entropy-based data reduction

Abstract

The software engineering community produces data that can be analyzed to enhance the quality of future software products, and data regarding software defects can be used by data scientists to create defect predictors. However, sharing such data raises privacy concerns, since sensitive software features are usually considered as business assets that should be protected in accordance with the law. Early research efforts on protecting the privacy of software data found that applying conventional data anonymization to mask sensitive attributes of software features degrades the quality of the shared data. In addition, data produced by such approaches is not immune to attacks such as inference and background knowledge attacks. This research proposes a new approach to share protected release of software defects data that can still be used in data science algorithms. We created a generalization (clustering)-based approach to anonymize sensitive software attributes. Tomek link and AllNN data reduction approaches were used to discard noisy records that may affect the usefulness of the shared data. The proposed approach considers diversity of sensitive attributes as an important factor to avoid inference and background knowledge attacks on the anonymized data, therefore data discarded is removed from both defective and non-defective records. We conducted experiments conducted on several benchmark software defect datasets, using both data quality and privacy measures to evaluate the proposed approach. Our findings showed that the proposed approach outperforms existing well-known techniques using accuracy and privacy measures.

1. Introduction

1.1 General overview

In the real world, it is quite important to identify software defects, or potential causes of such defects using data from previous software projects. However, information about such projects is usually “classified information” that is not shared with researchers or the public, especially for mission critical applications where defected files, modules, or even the number of lines of codes in such applications cannot be shared with the public. In this case, if shared it could potentially provide enough information for adversaries to attack those systems. In addition, there are business reasons for not sharing such data with the public.

Researchers usually create defect prediction models based on many public datasets. However, such datasets contain flaws and are not domain specific, meaning that they do not usually work well when applied to predict or detect defects in real-world systems. For instance, it is not useful to predict errors in manned spacecraft software using a data mining model that was prepared for a completely different type of system.

Another solution is to test defect prediction models on synthetic datasets, which is insufficient compared to testing on real systems. There is a new trend to disclose software datasets after scrambling specific sensitive attributes from those datasets. Yet again, masking only direct identifiers is not enough to protect data, as indirect identifiers are as significant as direct identifiers for adversaries wishing to initiate attacks. As an example, such datasets usually contain sensitive attributes such as the line of codes (LOC), which is usually associated with the effort needed to discover bugs in specific modules. When released, it could affect the business of an organization, as many third parties could discover various important metrics such as its coding schemes, potential vulnerabilities in the code, etc.

1.2 Motivation example

An example is shown in Tables 1 and 2. Data in Table 1 is considered private and sharing it would disclose sensitive attributes such as the LOC. Table 2 provides an anonymized version of the data in Table 1. When creating an anonymized release of the data, we remove the direct identifiers; however, doing so does not always guarantee that the release of this data will not be a target for specific types of attacks by adversaries or third party, including background knowledge attacks. For instance, even if we hide attributes such as file ID or name when releasing Table 2, some adversaries may still be able to infer the values of sensitive attributes, if they have background information about specific files. For example, if an attacker knows the number of children or weighted methods per class metric for file C_4, then he can be certain that the number of LOCs in that file is between 260 and 280 by having access to the data in the encrypted Table 2. One reason for this is that the attribute LOC is not diverse enough to prevent such types of attack.

Table 1
Private dataset

File ID	Weighted methods per class	Depth of inheritance tree	Number of children	Coupling between objects	Response for a class	Lack of cohesion in methods	Afferent couplings	Efferent couplings	Lines of code
C_1	14	8	1	15	40	30	2	12	280
C_2	10	2	2	7	30	50	3	15	258
C_3	11	3	0	8	32	31	1	14	260
C_4	5	3	8	1	8	5	5	11	300
C_5	5	2	9	3	20	1	8	4	135
C_6	6	1	6	2	9	11	10	1	213
C_7	4	2	1	4	11	20	0	2	214

This is just one example of how an attacker can identify the values of sensitive attributes. Tables 1 and 2 contain several types of attribute. For example, file ID is considered a direct identifier that needs to be removed when scrambling this dataset. Indirect identifiers, also called quasi identifiers, are another example of auxiliary information used by adversaries to reveal the values of sensitive attributes. Such identifiers are usually used in combination with external information to infer the values of sensitive attributes such as LOC.

While data scrambling or anonymization enhances the privacy of datasets when shared, it also leads to information loss. This trade-off between privacy and utility is not usually considered when creating anonymization algorithms. The existing works are mostly randomizing data; as such, the resulting anonymized data loses its statistical distribution.

Table 2

Released (anonymized data)

File ID	Weighted methods per class	Depth of inheritance tree	Number of children	Coupling between objects	Response for a class	Lack of cohesion in methods	Afferent couplings	Efferent couplings	Lines of code
${}^{**}$	10–14	$<$ 9	$\leqslant$ 6	7–15	30–40	30–50	${}^{*}$	12–20	280
${}^{**}$	10–14	$<$ 9	$\leqslant$ 6	7–15	30–40	30–50	${}^{*}$	12–20	258
${}^{**}$	10–14	$<$ 9	$\leqslant$ 6	7–15	30–40	30–50	${}^{*}$	12–20	260
${}^{**}$	$<$ 7	$<$ 9	$>$ 6	1–4	8–20	1–10	5–8	4–11	58
${}^{**}$	$<$ 7	$<$ 9	$>$ 6	1–4	8–20	1–10	5–8	4–11	58
${}^{**}$	$<$ 7	$<$ 9	$\leqslant$ 6	1–4	8–20	11–20	${}^{*}$	${}^{*}$	100
${}^{**}$	$<$ 7	$<$ 9	$\leqslant$ 6	1–4	8–20	11–20	${}^{*}$	${}^{*}$	116

While sharing data is also a major problem due to privacy regulations, including the General Data Protection Regulation (GDPR), regulations such as GDPR have recognized privacy-enhancing technologies. Data that is properly anonymized, thereby anonymous, does not fall under many GDPR restrictions [52].

This study introduces a novel privacy-preserving defect prediction approach which anonymizes data using a generalization technique that also maintains the diversity of the anonymized defect datasets. Furthermore, the approach reduces the size of the original dataset by employing an entropy-preserving measure to remove noisy records after anonymization. We found no evidence of an existing method that uses generalization techniques with data reduction to increase data usefulness after anonymization, where the latter is a major limitation of many of the existing techniques.

The aim of this research is to create a new generalization approach to share defect related metrics for software projects, while at the same time preserving sensitive attributes present in datasets containing those metrics. The approach has several practical implications in real life. Each company has specific privacy requirements, so our approach calibrates this by allowing users to identify the required level of privacy. Companies that have high restriction on their data can anonymize at a more generalized level, allowing for better privacy assurance. Furthermore, the quality of the shared data can be enhanced through the proposed record reduction mechanism which aims to discard less important records that mislead classification algorithms.

This research examines the use of generalization techniques with a novel data reduction approach that relies on information theory, specifically entropy, to anonymize defect-related data and use the resulting data in tasks such as classification. The approach tests both the privacy and the utility of the proposed anonymization. Additionally, since the data reduced is diverse, the resulting data needs to be immune to specific types of privacy attacks such as background knowledge attacks. Our approach will answer the following research question. Can we share anonymized release of software datasets using our diversity preserving technique while at the same time preserving both privacy and utility of the released data?

This research is significant because it will enable companies to share their defect metrics data without privacy concerns; therefore they will be able to share their data with other companies even to create cross-company defect prediction (CCDP) models. It has been shown that defect prediction models are as effective as within-company defect prediction [29, 30, 51].

1.3 Innovation

The proposed solution is innovative in the following aspects: 1) a generalization approach that provides both strong privacy guarantee and better preservation of data properties useful for subsequent analysis; 2) an intelligent tuning mechanism that makes this approach adaptive to different contexts including the privacy risks of a and its importance in analysis tasks; 3) no extra computation overhead for the data analyzer, i.e., the data analyzer can conduct data analysis directly on the anonymized data without the need of decryption. The second innovation of our approach is that it uses an intelligent tuning mechanism to make our solution adaptive to different contexts and domains including the privacy risk of a field its importance in analysis tasks. This can be done by tuning the privacy and utility budget based on the domain

1.4 Research background

Generally, the data owner stores data in table form, which mainly consists of different types of feature that can be classified into: explicit features, which explicitly identify a record in the table such as social security number (SSN), and sensitive attributes which indicate private specific information about a record such as salary; and quasi identifier attributes (QID), which is group of attributes that might identify a specific record. This classification of features is also applied to software data where some features are considered very sensitive such as the LOC and the effort related metrics.

Anonymization techniques fall into various categories, including generalization, suppression, anatomization, permutation, and perturbation. Generalization is the opposite of suppression and replaces QID or sensitive attributes with values that are more general; for example, it may replace engineer with job. On the other hand, rather than replacing the values of anatomization and permutation attributes, they concentrate on de-associating the relationship between QID attributes and sensitive attributes. Perturbation may reform the data by swapping values, adding noise, or generating synthetic data based on some statistical information of the raw data.

The anonymization process means hiding sensitive attributes that could be used to identify objects. Many techniques and tools can be used to anonymize data, but one of the main issues that needs to be considered when anonymizing data is the preservation of the record values and the contribution of features, in order to carry out a productive analysis on the data (e.g. creating defect prediction models). Therefore, the analytical results gained from the anonymized data will be similar to the result gained from un-anonymized data. It is essential, but not straightforward, to achieve such a tradeoff between privacy and utility so that the anonymized data can still be analyzed.

Anonymization algorithms differ with the noise level added and the task to which the anonymized data can be applied. While some algorithms such as Black Marker deteriorate the data and make it useless to researchers, others can partially preserve the structure and utility of the data. Most anonymization algorithms depend mainly on anonymizing identification attributes, which is not always enough to avoid all types of privacy attacks. The attacker may infer confidential information from external or auxiliary information, and therefore absolute privacy cannot be achieved in the presence of auxiliary information.

As mentioned above, it is vital to preserve data utility after anonymization, so it is quite important to ask researchers about their intentions, needs and tasks when studying software defect related matters. This is needed to release data that meets their needs and the tasks to which the data can be applied and create new anonymization techniques that preserve the statistical characteristics of the original data. For instance, when anonymizing data it can be applied for specific mining algorithms (e.g. classification), but not for others (e.g. clustering).

Research on data anonymization has been conducted in many domains including medical, social and financial. There is also a difference between anonymization and encryption. In encryption the main objective is to hide information, whereas in anonymization the objective is to create a data release that can be used by researchers, while at the same time maintaining its privacy. Applying privacy preserving techniques to software engineering has been used not only in defect prediction but also in many other objectives such preserving the privacy of personal data when collecting requirements and developing systems [8].

Anonymization, in the context of defect prediction, means hiding attributes that can be used to identify objects in defect datasets or organizations that generate the data. Beckers and Heisel [8] propose an approach to preserve the privacy of individual data collected during the system development process. Their approach creates privacy patterns to express and analyze different privacy goals such as anonymity, pseudonymity, unlinkability and unobservability by applying specific privacy mechanisms to the software engineering process to identify incomplete privacy requirements.

In many countries, privacy laws do not allow the use of actual data to test software products; therefore many projects use fake data to test their code. However, using fake data does not generally cover the operational conditions that arise when using the developed software in real-life environments. Specifically, good test case coverage cannot be achieved using fake data. Grechanik et al. [24] propose an approach to share realistic data that can be used by software testers to test their code. They applied a $k$ -anonymity approach to anonymize data, then used it to test software products while at the same time retaining a high level of test coverage. The authors found that using a small $k$ value, e.g. $k<$ 7, it was still possible to achieve 70% of test coverage. However, increasing the value of $k$ to more than 7 yielded coverage of less than 30%. One potential limitation of this approach was the difficulty in retaining privacy while achieving a high level of test coverage.

In a related work by Li et al. [31] an approach called TestIng taSks (PISTIS)/protecting and minimizing databases for software was introduced. This approach relies on a weight-based data clustering algorithm that clusters data point to achieve the objective of reducing the test space while covering as many test paths as possible. Each cluster is represented by a cluster centroid, which characterizes the general population. In addition, it becomes difficult for an adversary to infer useful information when only a cluster centroid is represented instead of the entire cluster. Experimental results have shown that it is still possible to achieve the required level of test coverage on the anonymized data.

Privacy preserving software testing has been utilized in other testing approaches, such as regression testing, which uses a test selection (RTS) technique with control-flow graphs (CFGs) to model web service interactions. To test these techniques, service providers need to share their CFGs with participants. The latter includes sensitive implementation details that cannot be always shared with participants. Ruth’s [44] study proposes a technique to sanitize individual nodes and alter the shape of the CFG to protect its sensitive details while at the same time keeping it effective for conducting testing.

Senarath and Arachchilage [46] address the issue of applying privacy requirements by developers and why they cannot embed privacy in software systems, and conclude that most developers are not even aware of personal privacy issues when implementing software systems. They provide several guidelines that should be applied by organisations to enforce privacy requirements and recommend that privacy guidelines should be made as simple and clear as possible. They also suggest that formal training procedures should be applied and used as guidelines in creating software systems. In the software engineering community, developers’ lack of knowledge regarding privacy affects their opinion which interferes in the way they embed privacy into designs. They also state that privacy guidelines should provide formal mechanisms for evaluation. Finally, they advise that the way in which privacy in software systems should be implemented at the technical level must be explicitly specified.

Peters and Menzies [40] proposed a privacy preserving defect prediction approach, MORPH, an anonymization-based technique which anonymizes data through a mutation function that moves data points based on random distance considering class boundaries. It was tested on several classification algorithms such as Random Forests, Bayes Net, and Logistic Regression and it was found that its main limitation is the difficulty in achieving a balance between preserving class boundary when noise is added and the required level of privacy; this solely depends on the distribution of the defect prediction dataset.

1.5 Limitations of the existing techniques and our contributions

The existing techniques, such as MORPH, have a number of limitations. In general, they do not guarantee that an instance does not move across the boundary between the original instance and instances of another class, which means that it may be difficult to keep the MORPHed value close to the original to maintain the utility of the dataset. Our approach, on the other hand, has the following contributions:

1.
This research examines using generalization techniques with a novel data reduction diversity preserving approach that relies on information theory to anonymize defects related data.
2.
The proposed approach tests both the privacy and the utility of the proposed anonymization.
3.
Our results show that we can share anonymized release of the datasets using our diversity preserving data reduction technique while at the same time preserving both the privacy and the accuracy of the released data.

The rest of this paper is organized as follows: Section 2 discusses the related work; Section 3 presents the methodology of our approach; the experiment and evaluation is presented in Section 4; the discussion of the results are shown in Section 5; threats to the validity are presented in Section 6; and Section 7 concludes the work and discusses the potential of future work.
2. Literature review

2.1 Privacy preserving analytics in software engineering

Data analytics in software engineering has been applied over the years for many objectives such as defect prediction, but one of the main challenges has been data privacy. Moparthi and Geethanjali [37] proposed a multi-party ensemble-based defect prediction approach on multiple associated products which privately discovers and uses defect patterns for prediction. They encoded discovered patterns using string encoding functions by dividing data analysis sites into masters and slaves, then a public key function was used for transmitting data between sites. Once the sensitive patterns were discovered in the master site, they were stored in a matrix and then sent randomly to slave sites. They repeated the process until all instances were processed and tested using classification accuracy measure; however, there was no privacy-based evaluation of the approach. In addition, the authors did not describe the encoding functions and public key infrastructure used to implement the system.

Peters and Menzies [40] propose a privacy preserving defect prediction approach-MORPH. MORPH is a Anonymization-based technique which anonymizes data through a mutation function that moves data points based on a random distance considering class boundaries. MORPH was tested on several classification algorithms such as Random Forests, Bayes Net, and Logistic Regression. The main limitation for MORPH is the difficulty of achieving a balance between preserving class boundary when the noise is added and the required level of privacy; this solely depends on the distribution of the defect prediction dataset.

Peters et al. [41] continued this line of research by proposing a new algorithm, CLIFF, which is an instance pruner that finds attribute subranges in the data that are more present in one class than in other classes. They propose that after anonymization; a dataset can be pruned using CLIFF in order to mitigate inference attacks that benefit from large data size. CLIFF utilizes a power function proposed by Jalali et al. [28] to prune less useful attributes. As such, the power of each attribute’s value needs to be calculated for each class. The power values depend mainly on the frequency of attribute values with a specific class. This can be done at the row level, then rows with the highest power are kept in the dataset, while rows with less power are pruned from the dataset. Experimental results have shown that combining MORPH with CLIFF leads to significantly better privacy compared with only using MORPH. While MORPH can work well on large datasets, it cannot be applied effectively to small datasets. LACE2 was proposed by Peters et al. [42] to reduce the amount of data shared when multi-party data sharing is needed. Data can be incrementally added by parties to the repository, when it is not similar to the current data in the repository, thereby avoiding data redundancy. The obfuscation approach used was quite similar to MORPH; however the authors mention that their new approach yielded better privacy than their previous approaches without affecting data utility.

Li et al. [33] proposed a multiple sources and privacy preservation approach for heterogeneous defect prediction. Their approach selects the most similar projects to the one under testing by joining the target project and a well-selected set of other sources in a discriminative common subspace, where the target and sources have similar data distributions and favorable classification ability. A privacy preserving algorithm was designed based on double obfuscation (SRDO). The authors found that SRDO achieved better privacy and utility than CLIFF and MORPH.

Fu et al. [21] proposed a new approach called pattern preserving tree that perturbed the original software engineering data considering the patterns existing in the original tree. Unlike the existing approaches, the new approach perturbs data while paying attention to the patterns that exist in the original tree. It generates a new regression tree from the perturbated data, which can then be shared with researchers.

A privacy model for behavior preserving testing and debugging of software systems was introduced by Budi et al. [13]. Their approach handled one of the main limitations of $k$ -anonymity when applied to behavior testing, proposing, which not only preserves privacy but also ensures that the replaced data exhibits the same program behavior as the original data; therefore, the original data may be still used for testing and debugging. Thus, test cases that lead to failures on the original datasets will lead to the same behavior on the new dataset to maintain the same level of utility.

A number of similar studies have been carried out regarding privacy preserving testing of database applications by generating a mock dataset that retains the statistical character of the original dataset [53]. The objective of creating a mock dataset is to have an almost identical structure as the original data in which table schema description and view indices are retained and can be used to estimate the performance of the original dataset. The authors showed that a mock database generated from this can simulate the live environment for testing purpose, while maintaining the privacy of data in the original database.

Wang et al. [53] went on to propose a model for quantifying the privacy leakage and application performance metrics differences when generating synthetic datasets. The analysis conducted in their paper shows that designing an efficient privacy preserving approach to measure performance issues is a challenging problem.

Recently a number of works have been conducted utilizing differential privacy to conduct privacy preserving analytics of software systems including Chen et al. [15] who propose an approach called DP-Share, which applies the traditional data mining process to software engineering datasets, such as sampling and classification. Once the model is created, Laplacian noise is added to satisfy the requirements of differential privacy. Nine software projects were used during the experiments and several performance measures were used to evaluate the privacy and utility of the proposed technique. The results showed that after privacy and utility analysis DP-Share can achieve better performance than a baseline method.

Other related works focus on using privacy preserving techniques in effort estimation. For example, Qi et al. [43] show the importance of preserving the privacy features of effort estimation. They highlight the many indicators that reflect sensitive features in the dataset, such as the properties of the product or the image of a team, e.g. function points count (FP), line of code (LOC) and number of function point (size). The authors conclude that privacy threats can target their effort data; therefore they propose an interval covering based subclass division (ICSD) strategy to obfuscate data. ICSD can divide the target data into several subclasses by extracting a new attribute (i.e. class label) from the effort data. The obtained class label is beneficial for maintaining the distribution of the target data after obfuscation. The authors proposed a manifold learning based bi-directional data obfuscation (MLBDO) algorithm, which uses two nearest neighbors, selected respectively from the previous and next subclasses by utilizing the manifold learning based nearest neighbor selector, as the disturbances to obfuscate the target sample. The approach was called ICSD&MLBDO. Experimental results on seven public effort datasets showed that: 1) ICSD&MLBDO enhanced the privacy and maintained the utility of obfuscated data, and 2) ICSD&MLBDO can achieve better privacy and utility than the compared privacy-preserving methods.

2.2 Privacy preserving data analytics

Privacy preserving methods and techniques focus on modifying data to hide direct identifiers and make it difficult to identify individual entities by inferring relationships between indirect identifiers and external information [1]. It has been shown that removing direct identifiers such as name and email only is not effective in avoiding all privacy related attacks [7]. There is another category of attributes called indirect identifiers which are not explicit identifiers but can still be combined with other external attributes to identify individuals. This type of attack is known as a record linkage attack (Benjamin et al., 2010). An example of such attributes is the QID set [5-digit ZIP, gender, date of birth]. It has been shown that about 87% of the residents of the United States can be identified when this set of attributes is publicly available [23].

Privacy preserving data mining (PPDM) focuses on transforming data to create a private version of it, but preserving its statistical attributes. Mendes and Vilela [36] surveyed existing privacy preserving data mining techniques and classified them into data collection, data publishing, data distribution and output-based approaches. Data collection techniques include additive [39] and multiplicative noise methods [34].

Data anonymization techniques are divided into many categories including generalization, perturbation, etc. In generalization the values of quasi identifiers are replaced with more general values to avoid inference and record linkage attacks. Introducing similarity between records through k-anonymity is a very common approach to releasing generalized anonymized datasets [50]. In $k$ -anonymity, data attributes are divided into sensitive attributes, identifiers, and quasi identifiers. Direct identifiers are removed from the dataset before any release. On the other hand, quasi identifiers are generalized such that each record appears $k$ times in the dataset with $n$ records. This way, $n/k$ clusters are created. Sometimes all sensitive attributes have the same value for all records in a cluster; therefore, diversity in the values of sensitive attributes is needed. Machanavajjhala et al. [35] proposed $l$ -diversity, an improved variety of $k$ -anonymity. In $l$ -diversity each cluster must contain a minimum number of values of sensitive attributes. One limitation of $l$ -diversity is that it does not take into consideration the distribution of classes in the original dataset when creating diverse clusters; therefore T-closeness was proposed to resolve this problem [32]. Other data sharing techniques include personalized privacy preserving solutions [22], and differential privacy [19]. The latter is one of the most widely used in preserving the privacy of statistical queries, where the objective is to maximize the accuracy of query output, while at the same time having a low impact on individuals’ privacy. A Laplace noise is added with $\theta(1)$ variance to each entry.

Multiple data sharing techniques target distributed data storage systems. For instance, in secure sum, secure product, and secure set union the objective is to calculate some mathematical functions over two or more sites, without violating the privacy of individual sites [16]. There are several metrics for measuring privacy and utility after applying privacy preserving algorithms. Privacy measures include confidence level [5], average conditional entropy [4], variance [39], and hidden failure [38]. Utility measures include minimal distortion [45], loss metrics [27], information loss [54], misses cost [39], and misclassification error [39].

Several other approaches included in the literature inherit the characteristics of the previous techniques to create a privatized version of the datasets. Randomization approaches have been classified into numerical and itemset randomization [11] and do not need prior knowledge about all records and correlations patterns inherent in the data. Randomization approaches treat all records irrespective of their local density; therefore outlier records are more susceptible to adversarial attacks than records in more dense regions [1]. Evfimievski et al. [20] utilize a randomization-based privacy preserving approach for identifying support and variance in association rules. Approaches such as noise addition, data swapping, and synthetic data generation can be classified under randomization-based anonymization. Soria-Comas et al. [49] introduce a micro-aggregation privacy model. Unlike the traditional $k$ -anonymity approach, micro-aggregation provides protection against attribute disclosure, an attack which occurs if the variability of the sensitive values in a group of $k$ subjects is too small. Micro-aggregation perturbs the data by increasing data granularity and reducing the impact of outliers. Both micro-aggregation and $k$ -anonymity were used by Casino et al. [14] for collaborative filtering. To provide estimated recommendations from perturbed data while guaranteeing user anonymity. Yuan et al. [57] utilized $l$ -diversity for anonymizing social network data. Their model, $k$ -degree- $l$ -diversity, adds new nodes as noise to the original graphs instead of editing their edges. Soria-Comas et al. [49] introduce stochastic $t$ -closeness function, which takes advantage of micro-aggregation algorithms to attain higher utility. Cormode et al. [18] compared modern privacy models such as $k$ -anonymity, $l$ -diversity, $t$ -closeness, with differential privacy based on privacy and utility levels and found that differential privacy achieved higher privacy than other techniques. They recommend adopting privacy models according to data recipients. For example, if the data is to be released to an external organization, it is preferable to adopt differential privacy, whereas $l$ -diversity and $t$ -closeness are enough if data is released to users in the same organization.

Differential privacy is a privacy model that provides worst-case privacy guarantee regardless of what attackers know [19]. It protects statistical aggregate queries to prevent attackers from inferring sensitive information about individual records based on subsequent queries. The assumption is that if an adversary has full or partial background knowledge about a group of individuals in a dataset, with the exception of a single record, he cannot re-identify that record. Privacy is achieved by adding enough noise that follows a specific distribution called Laplace or double exponential. In differential privacy, the level of privacy can be controlled by a data publisher who decides the amount of noise (i.e., privacy budget) to be introduced to the dataset using a $\varepsilon$ parameter. The randomized function provides $\varepsilon$ -differential privacy, if the results of the function calculated on a dataset are indistinguishable within a certain privacy metrics. Differential privacy has been applied for different purposes such as perturbing search logs [26], releasing histogram queries [55], and matrix factorization in recommender systems [10]. Identifying $\varepsilon$ – values is a major challenge when anonymizing data using differential privacy. In addition, the conventional differential privacy cannot be applied to software data while preserving the original semantics of features. It has been shown that differential privacy works better when applied to big data. A good discussion on this can be found in [17], regarding the advantages of differential privacy in the field of big data even when an attacker has strong background knowledge about the data.

3. Research methodology

3.1 Overall research design

Figure 1 shows the architecture of our proposed approach. In a real environment our method works by obtaining an original defects dataset with features describing the data. We first applied a generalization-based anonymization technique to the dataset to generate an anonymized release of it using a per-class generalization technique. Singular value decomposition (SVD) was applied to each generated cluster to create statistically sound data similar to the original dataset.

Figure 1.

The architecture of the proposed approach.

Importance of features of the anonymized data was used to reduce the size of data while preserving data privacy and diversity. This approach generates anonymized datasets with defects and no-defects records which can then be shared with data recipients, who may use it to perform defect prediction. Defect prediction was applied to the original data and a comparison was conducted before and after anonymization. The methodology is evaluated based on several utility and privacy measures as shown in our experiments.

3.2 Research phases

3.2.1 Software defect data

Software data with defective software modules, components, or files is utilized by companies to manage the software testing process using valuable historical data. However, sometimes such data does not exist or is not sufficient, in which case an organization can use data from other organizations. Table 3 shows some of the characteristics or quasi identifiers that may be found in software data sets.

Table 3
Sample quasi identifiers in software defect data

Attribute symbol	Description	Attribute symbol	Description
AMC	Average method complexity described in terms of Java byte code	LCOM	Lack of cohesion in method which is measured in terms of not sharing references to instance variables
AVG_CC	Cyclomatic complexity	LOC	Volume of code
CA	Affert coupling between classes and the class	MAX_CC	Maximum Cyclomatic complexity
CAM	Cohesion among classes which depends on the types of method parameters in the class	MFA	Measure of number of methods inherited by a class
CBM	Coupling between methods which depend on the number of redefined methods to which all inherited methods are coupled	MOA	Aggregation measures with user defined classes
CBO	Coupling between object, which increases when methods of classes serve each other	NOC	Number of descendants of a class
CE	Number of classes used inside a specific class	NPM	Measure of public methods in a class
DAM	Attributes access level measured in terms of private or protected attributes in the class	RFC	Method invoked as response to invocations by objects
DIT	Depth of the inheritance tree and the location of the class in the inheritance tree	WMC	Measure the number of methods in the class
IC	Inheritance coupling, which is measure of coupling of a class with parent classes	DEFECTS	Number of defects per class

In our experiment we used a data repository which contains different datasets generated by NASA [12] and is renowned in test research. All the datasets include attributes related to code.

To explain our approach, we will use the following motivation example. Table 4 shows a sample of an original dataset that contains defects. The example will be used to drive our description of the methodology steps and our contributions.

Table 4

Private software defect dataset

File ID	Weighted methods per class	Depth of inheritance tree	Number of children	Coupling between objects	Response for a class	Lack of cohesion in methods	Afferent couplings	Efferent couplings	Lines of code	DF/NDF
C_1	14	8	1	15	40	30	2	12	400	DF
C_2	10	2	2	7	30	50	3	15	258	NDF
C_3	11	3	0	8	32	31	1	14	50	DF
C_4	5	8	8	1	8	5	5	11	58	DF
C_5	5	8	9	3	20	1	8	4	58	DF
C_6	6	1	6	2	9	11	10	1	60	NDF
C_7	4	2	1	4	11	20	0	2	116	NDF

Table 5

Clustering based anonymization (min 2 instances per cluster)

File ID	Weighted methods per class	Depth of inheritance tree	Number of children	Coupling between objects	Response for a class	Lack of cohesion in methods	Afferent couplings	Efferent coupling	Lines of code	DF/NDF
${}^{**}$	10–14	$<$ 9	$\leqslant$ 6	7–15	30–40	30–50	${}^{*}$	12–20	400	DF
${}^{**}$	10–14	$<$ 9	$\leqslant$ 6	7–15	30–40	30–50	${}^{*}$	12–20	258	NDF
${}^{**}$	10–14	$<$ 9	$\leqslant$ 6	7–15	30–40	30–50	${}^{*}$	12–20	50	DF
${}^{**}$	$<$ 7	$<$ 9	$>$ 6	1–4	8–20	1–10	5–8	4–11	58	DF
${}^{**}$	$<$ 7	$<$ 9	$>$ 6	1–4	8–20	1–10	5–8	4–11	58	DF
${}^{**}$	$<$ 7	$<$ 9	$\leqslant$ 6	1–4	8–20	11–20	${}^{*}$	${}^{*}$	60	NDF
${}^{**}$	$<$ 7	$<$ 9	$\leqslant$ 6	1–4	8–20	11–20	${}^{*}$	${}^{*}$	116	NDF

3.2.2 Formal definition of clustering based anonymization

The algorithm then runs an insensitive micro aggregation method to divide data in each $D i$ into clusters with size at least K [1]. Suppose there are two data sets D and D’ that differ by at most one record, incentive microaggregation ensures that each resulting cluster from D and D’ will only differ by at most one record as well. The method computes two boundary points P and P’ where P takes the minimal value on each field and P’ takes the maximal value on each field. The method then forms a cluster around P with K points closest to P. A similar cluster is formed around P’ by K points closest to P’. These points are removed from the data set and the process continues until there are fewer than 2K points left. These remaining points form a cluster. To prove our algorithm is differential private, it is sufficient to show that the addition of a record z will have limited impact on the result.

Our example in Table 5 shows the resulting dataset when anonymizing using a clustering-based anonymization and the taxonomy shown in Fig. 3. Here LOC is considered a sensitive feature to be protected. The last column is the class attribute, and all other features are considered quasi identifiers therefore anonymized.

Figure 2.

Generalization example in which the original LOC counts are clustered (left), then generalized based on cluster mean (right).

In reality, clustering-based anonymization is carried out using clustering algorithms such as $k$ -means since it is not easy to create the taxonomy shown above. An objective function which measures the tightness within clusters should be minimized (e.g. the intra-cluster distance, average centroid radius of the clusters). We need to partition a dataset $D$ , into $S=N/K$ groups $C1\ldots Cs$ of at least $k$ data points each.

Figure 3.

Using a domain taxonomy to anonymize defect data.

3.2.3 Diversity preserving data reduction

We relied on information theory, in which entropy is a measure of the uncertainty associated with a random variable, to accomplish the data reduction step. This term usually refers to Shannon entropy, which quantifies the expected value of the information contained in a message, usually in units such as bits [47]. Entropy has been utilized in areas such as natural language processing. It is a measure of the average information content which is missing when the value of a random variable is not known. Overall, entropy, when a defected code is conditioned on one or (more) features, reveals the degree of uncertainty about the occurrence of a defect when such feature(s) are observed. This form of entropy is called conditional entropy. The conditional entropy $(H)$ for a set of classes, $Y$ , given a feature, $F$ , is the sum of entropy when $N$ are conditioned on $F$ and it is calculated over all possible $k$ values of $F$

$\displaystyle H({Y{|}F})=-\mathop{\sum}\limits_{j=1}^{k}\mathop{\sum}\limits_{% i=1}^{|Y|}P({y_{i},f_{j}})\log_{2}P({y_{i}{|}f_{j}})$ (1)

It has been reported that conditional entropy is one of the best preforming entropy measures in signal compression algorithms [48]. In general, the more non-uniform occurrence of a feature, $f$ , with a class $n_{i}$ the lower the value of $H({n_{i}{|}f})$ .

Figure 4.

Points in the space representing defective and non-defective records.

For example, Fig. 4 shows many defective (green circles) and benign code records represented as points in the space. Records belonging to different classes, but close to each other, yield the highest entropy, since the values of features of those records have almost uniform distribution. Data scientists call these instances Tomek links, which are pairs of very close instances, but of opposite classes. Removing such instances increases the space between the two classes, facilitating the classification process. Additionally, it preserves the diversity of the original dataset by removing records from both classes. Formally, a Tomek link is defined between records $n_{i}$ , $n_{j}$ of different classes ( $c_{i},c_{j}$ ), such that there is no example $n_{z}$ for which

$\displaystyle d(n_{i},n_{z})<d(n_{i},n_{j})\text{ or }d(n_{j},n_{z})<d(n_{i},n% _{j})$

Thus, $d()$ is the distance between the two samples. In other words, Tomek links exist when there are nearest neighbors of different classes as shown in Fig. 5.

Figure 5.

Using Tomek links for data reduction.

Algorithm 1. Tomek’s links-AllKNN data reduction algorithm (TLAll-KNN)
1. EDRTM (Dataset $D$ )
2. Let $n$ be the number of records in the dataset $D$
3. Let $Y$ be the set of classes in $D\|y\subseteq Y$
4. For each Class $y$ in $D$ do
5. For each record $n_{c_{j}}$ in $c_{j}\|n_{c_{j}}\to y$ do
6. if $n_{c_{j}}\in$ Tomek link then
7. Add it to the setTL
8. End if
9. End for
10. End for
11. Select all elements in TL
12. D’= D $\backslash$ TL
13. For each Class $y$ in ${D}^{\prime}$ do
14. For each record $n_{c_{j}}$ in $c_{j}\|n_{c_{j}}\to y$ do
15. Discard $n_{c_{j}}$ from $D^{\prime}$ if it is misclassified using the k-NN rule
16. End for
17. End for
18. End

Removing Tomek links is not enough since there will be low ranked records within the boundaries of the class itself, therefore we utilized another reduction technique, which was proposed by Alejo et al. [6]. Each instance $n_{i}$ is removed if its class does not agree with most of its nearest $k$ neighbors (with $k=$ 3, typically). This edits out noisy instances as well as close border cases, leaving smoother decision boundaries. Algorithmically, our reduction approach works as described in Algorithm 1.

In a related work, Peters et al. [42], utilized the near-miss approach for data obfuscation which is shown in Fig. 6. Specifically, this approach selects samples from the majority class (non-defective components) for which the average distance of the $N$ closest samples of a minority class (defective records) is smallest. Then they use the selected sample to add noise to the closest instances.

Table 6

Entropy-based ranking for the private dataset

File ID	Weighted methods per class	Depth of inheritance tree	Number of children	Coupling between objects	Response for a class	Lack of cohesion in methods	Afferent couplings	Efferent couplings	Lines of code	DF/NDF	TL
C_1	14	8	1	15	40	30	2	12	400	DF	N
C_2	10	2	2	7	30	50	3	15	258	NDF	N
C_3	11	3	0	8	32	31	1	14	50	DF	N
C_4	5	8	8	1	8	11	5	11	58	DF	N
C_5	5	1	9	3	20	12	8	4	58	DF	Y
C_6	6	1	7	2	20	11	10	1	60	NDF	Y
C_7	4	2	8	4	11	20	0	2	116	NDF	N

This approach utilizes the following obfuscation function

$\displaystyle n_{i}=n_{j}\mp({n_{i}-n_{z}})\ast r$

nearest neighbor of $n_{j}$ whose class is different from $n_{j}$ . $r$ is a random number between 0.15 and 0.35. The main limitation of this approach was discussed by Li et al. (2017); although the MORPHed value can ensure that the privatized dataset is private enough, it does not always guarantee that an instance does not move across the boundary between the original instance and instances of another class, which means that it may be difficult to keep the MORPHed value close enough to the original value to maintain the utility of the dataset. Using our example, if the Tomek links are identified, the result will be as shown in Table 6 where the column TL $=$ Y means that this a Tomek link record.

If the clustering step is conducted, one potential clustering result is shown in Table 7. This table contains three clusters, with $k=$ 2.

Table 7

Discarding records with the lowest entropy rank (highest entropy)

File ID	Weighted methods per class	Depth of inheritance tree	Number of children	Coupling between objects	Response for a class	Lack of cohesion in methods	Afferent couplings	Efferent couplings	Lines of code	DF/NDF	TL
${}^{**}$	10–14	$<$ 9	$\leqslant$ 6	7–15	30–40	30–50	${}^{*}$	12–20	400	DF	N
${}^{**}$	10–14	$<$ 9	$\leqslant$ 6	7–15	30–40	30–50	${}^{*}$	12–20	258	NDF	N
${}^{**}$	10–14	$<$ 9	$\leqslant$ 6	7–15	30–40	30–50	${}^{*}$	12–20	50	DF	N
${}^{**}$	$<$ 7	$<$ 9	$>$ 6	1–4	8–20	11–20	5–8	4–11	58	DF	N
${}^{**}$	$<$ 7	$<$ 9	$>$ 6	1–4	8–20	11–20	5–8	4–11	58	DF	Y
${}^{**}$	$<$ 7	$<$ 9	$>$ 6	1–4	8–20	11–20	${}^{*}$	${}^{*}$	60	NDF	Y
${}^{**}$	$<$ 7	$<$ 9	$>$ 6	1–4	8–20	11–20	${}^{*}$	${}^{*}$	116	NDF	N

Figure 6.

Near-miss data reduction approach.

3.2.4 Data reduction

Once the data is anonymized it will be shared with data recipients. Our approach will calibrate this by allowing users to identify the required level of generalization, where higher $k$ means better privacy. Each organization has specific privacy requirements, so companies that have high restriction on their data can anonymize their data at a much-generalized level, allowing for more privacy guarantee. The effectiveness of the anonymized data is compared with the original data in terms of how accurate the anonymized data is in identifying defects. Based on our working example, the data in Table 8 is the resulting anonymized data after removing TLs. This data maintains both diversity and privacy, since each cluster contains records from each class after generalization.

Table 8
Resulting data maintaining both diversity and privacy (2 clusters $s=$ 2) each)

File ID	Weighted methods per class	Depth of inheritance tree	Number of children	Coupling between objects	Response for a class	Lack of cohesion in methods	Afferent couplings	Efferent couplings	Lines of code	DF/NDF
${}^{**}$	10–14	$<$ 9	$\leqslant$ 6	7–15	30–40	30–50	${}^{*}$	12–20	400	DF
${}^{**}$	10–14	$<$ 9	$\leqslant$ 6	7–15	30–40	30–50	${}^{*}$	12–20	258	NDF
${}^{**}$	10–14	$<$ 9	$\leqslant$ 6	7–15	30–40	30–50	${}^{*}$	12–20	50	DF
${}^{**}$	$<$ 7	$<$ 9	$>$ 6	1–4	8–20	11–20	5–8	${}^{*}$	58	DF
${}^{**}$	$<$ 7	$<$ 9	$>$ 6	1–4	8–20	11–20	${}^{*}$	${}^{*}$	116	NDF

3.2.5 Proving the anonymization approach

Suppose a point z is added, and its distance to P is in between xi and $x_{i+1}$ . Without loss of generality, let us consider points in clusters closer to P. Suppose these points are sorted by their distance to P as $x_{1},x_{2},\ldots,x_{n}^{\prime}$ . Clearly the first cluster around P is formed by $x_{1},\ldots x_{k}$ , and the next cluster is formed by $x_{k+1},\ldots,x_{2k}$ and so on. Only clusters formed after xi will be affected by addition of z and the difference between such a cluster before and after addition of z will be at most one point. This allows us to bind the sensitivity (i.e., the maximal impact) of adding z on cluster mean by Amax/K where Amax is the maximal value of a field and K is the minimum cluster size. There are at most n/2K clusters affected as the other half of points will be assigned to clusters closer to P’. As a result a noise with mean zero and standard deviation $\frac{A_{\max}n}{2k^{2}\varepsilon}$ can be added to ensure privacy. To make the standard deviation of noise smaller, we just need to set the minimum cluster size $k>\sqrt{n/2}$ .

Pseudo data per cluster is generated using singular value decomposition (SVD). SVD is applied to the anonymized clustered data set. It decomposes a matrix $D$ with $n\times m$ into three matrices as:

$\displaystyle D=U\sum V^{T}$ (2)

Whereas $U$ is $n\times n$ orthonormal matrix (matrix with eigenvectors $DD^{T}$ , $\sum$ contains singular values in diagonal matrix, and $V^{T}$ is $m\times m$ orthonormal matrix (a matrix with eigenvectors $D^{T}D$ ). SVD is mainly used as a dimensionality reduction technique, through which new transformed matrix $D^{r}$ is produced, which is a resemblance of $D$ in a reduced $r$ dimensional space, such that the largest $r$ singular values approximated in new matrix $D^{r}$ . Whereas:

$\displaystyle D^{r}=U_{r}\sum_{r}V_{r}^{T}$ (3)

The number of first $r$ columns of $U$ are represented by $U_{r}$ , and the number of first row in $\sum$ , and $V$ are represented by $\sum_{r},V_{r}^{T}$ respectively [25, 56].

4. Experiments and evaluation

4.1 Data analysis and interpretation

In theory, the ideal situation for any privacy algorithm is to achieve the highest level of privacy and utility. However, the objective of such algorithms has been always to reach an acceptable tradeoff between privacy and utility as shown in Fig. 7.

4.1.1 Privacy

To measure the privacy level of anonymized datasets, we use conditional privacy [3], This measure depends on the notion of mutual information between the raw and anonymized records at $c$ % confidence level, while information loss is related to the amount of mismatch between them. Conditional privacy is based on differential entropy of a random variable. The differential entropy of $A$ given $B=b$ is

$\displaystyle h({A{|}B})=\int_{{\Omega}_{A,B}}f_{A,B}({a,b})\log_{2}f_{A|B=b}(% a)da\,db$ (4)

Where $A$ is a random variable that describes the data, and $B$ is a variable that gives information on $A$ . ${\Omega}_{A,B}$ identifies the domain of $A$ and $B$ . Therefore, the average conditional privacy of $A$ given $B$ is

$\displaystyle\prod({A{|}B})=2^{h(A|B)}$ (5)

If $D_{i}$ is the attribute value of the original data and $D^{\prime}_{i}$ is the value after anonymization, the conditional privacy value for anonymizing such an attribute is $2^{h(D_{i}|{D^{\prime}_{i}})}$ where $h({D_{i}{|}D^{\prime}_{i}})$ is the conditional entropy of the original data given the anonymized data. Conditional privacy is calculated and then averaged over all attributes.

Figure 7.

Relationship between privacy and accuracy.

4.1.2 Utility

We determine whether the anonymization techniques produce satisfactory accuracy values in terms of differentiating between files that contains defects and those that are defect free. Thus, if we apply a classification model on the original dataset, the results should not be very different if the same model is applied to the anonymized dataset. Accuracy is measured based on the following formula

$\displaystyle\textit{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}$ (6)

$T P$ , $T N$ , $F P$ , and $F N$ represent the true positives, true negatives, false positives, and false negatives respectively. $T N$ A $T P$ represents a defected code that is correctly identified as having defects. $T N$ represents a defect free code (benign code) that is classified as defect free. An $F P$ arises when a defect-free code under evaluation is incorrectly recognized as having defects. An $F N$ arises when a defected code is incorrectly recognized as having no defect. We used $k$ -nearest neighbor (K-NN) as the classification algorithm to test our approach. We performed our experiments on a testbed environment that included a prototype implementation of our approach on a 64-bit Windows with Intel Pentium D Dual Core 3.4 GHZ CPU and 32 GB RAM.

In addition to accuracy, we used other measures to evaluate the effectiveness of our approach. Both recall ( $R$ ) and precision ( $P$ ) were calculated based on the values of true positive (TP), false positive (FP), and false negative (FN) which were calculated using the testing data. The value TP represents the number of defective modules that are predicted as defective. The FP represents the number of benign modules that are predicted as defective. An FN represents the number of defective modules that are predicted as defective. Equations (7)–(9) are used to calculate the values of $P$ , $R$ and $F$ -score.

$\displaystyle P=\frac{TP}{TP+FP}$ (7) $\displaystyle R=\frac{TP}{TP+FN}$ (8) $\displaystyle\textit{F-score}=\frac{({1+\beta^{2}})\times P\times R}{\beta^{2}% \times({P+R})}\beta^{2}=1$ (9)

4.2 Types of experiment

To run our experiments, we implemented several components of our approach in MATLAB and Python, with the anonymization approach implemented in MATLAB and the data reduction approach implemented in Python. We also used the implementation of MORPH, CLIFF and LACE2 in Python to conduct a comparison with other approaches. The experiments were conducted on 11 datasets using the PROMISE dataset repository which contains different datasets generated by NASA (Boetticher, 2007). As mentioned previously, this is a well-known repository that has been used to test research on software engineering. Table 9 shows the number of records, features, and labels and the percentage of bugs in each dataset. These datasets have been used recently by Peters et al. [4] to test their approach on privacy preserving defect prediction.

Table 9
Datasets used in the experiments

Dataset	Features	Labels	Percentage of bugs	No of defective records	No of non-defective records
CM1	22	2	10%	49	449
KC1	22	2	15.5%	326	1783
KC2	22	2	20.5%	105	415
KC3	40	2	19%	36	158
PC1	22	2	7%	77	1032
PC2	37	2	2%	16	729
PC3	38	2	12%	134	943
PC4	38	2	12%	178	1280
MC1	38	2	3%	46	1942
MC2	38	2	35%	44	81
MW1	38	2	11%	27	226

Several types of experiment were conducted on each dataset as follows:

The first experiment aimed to measure the contribution of each feature to the overall privacy level. The objective of this experiment was to examine if some features contribute greater than others to overall privacy. This experiment was conducted by varying the number of instances per cluster and then measuring the overall privacy level.

In the second experiment we measured the effect of data reduction on the privacy values. This experiment was conducted by varying the number of instances per cluster. The objective of this experiment is to examine if removing some instances from the dataset makes the data more or less vulnerable to privacy threats.

The third experiment measured the utility of the resulting data in terms of different metrics. This experiment had two objectives: first to examine how perturbing data affects the values of accuracy when the resulting data is tested and second, to examine if data reduction can enhance the classification accuracy when the resulting data is used for classification.

The final experiment compared our approach with the existing techniques. We conducted this experiment by measuring both privacy and utility of the resulting data and comparing it with the original data.

4.2.1 Privacy results

The first experiment measured privacy per feature for each dataset, by varying the number of instances per cluster, then measuring the privacy values per feature, and finally calculating the overall privacy values. The results of this experiment are shown in Fig. 8.

Figure 8.

Privacy values for all the datasets used.

From Fig. 8 we can summarize the following:

The CM1 dataset consisted of 21 features, some of which were used to measure the empirical characteristics of the code, predicating its susceptibility to errors using features such as the lines of codes and the number of branches. First, it is noted that privacy values increased when the number of instances per cluster were increased. The first implication of this is that organizations that have higher privacy requirements may need to increase the generalization level. The second implication of the above results is that increasing the number of instances reduces the probability of background knowledge attacks as demonstrated by the results. Since CM1 is a small size dataset, the largest number of instances that is used to run this experiment is 14.

The second dataset (KC1) contained the same set of features as the first dataset, however the number of instances was significantly different. This experiment was conducted under the same constraints as those on the first data set. Since the number of the instances is high, we increased the number of instances per cluster to examine if we could attain higher privacy values. We observed similar results on this dataset; however, as it contained more instances per cluster, higher privacy values were attained. We also noticed that the increase in privacy values was lower when the number of instances was increased to more than 150.

The third experiment was conducted on the KC2 dataset, which had some similarities with KC1, however it had a smaller number of instances than KC1. It is noted that applying our method to KC2, yielded less privacy than KC1. In addition, the increase in the values of conditional privacy followed a regular pattern. It is important to note that most of the features contribute equally to the privacy values, which indicates that our approach is not affected by the weights of features when calculating privacy values.

The KC3 dataset contained more features of complexity metrics in the code. Adversaries may benefit from such data by trying to understand how many errors a code may contain, coding patterns and other coding characteristics. The KC3 data contained a smaller number of records than KC1 and KC2. The first experiment measured the privacy values when the number of instances per cluster was changed. The results show the benefits of having more features on the privacy values, indicating that higher privacy values are achieved when the dataset contains diverse features. Specifically, adversaries need to spend more time trying to map the correlation between features in order to recover the original data.

The MC1 dataset had similar characteristics to KC3, however it was less balanced, containing fewer instances of the minority class. The results clearly indicate that the privacy results of our approach are not affected by the imbalance induced in our dataset.

The MC2 dataset was small compared to the other datasets. On the other hand, it was more balanced than the previous datasets. Although it was small, its privacy values are shown to still be high, which highlights the importance of attribute diversity in achieving higher privacy values.

The MW1 dataset had similar features to the previous datasets. It contained a smaller number of instances from the minority class. The privacy-based analysis was conducted based on the same constraints as the previous datasets and the values look similar when compared to the results of the previous experiments. However, an important observation regarding this dataset is the variation in the values of privacy when the number of instances per cluster was changed. For instance, the feature Num_Operators yielded lower privacy than other features when the number of instances per cluster was increased. As there was small number of instances for one of the classes, increasing the number of instances per cluster led to clusters that contained a small number of records from the minority class compared to the majority class, which eventually lowered the privacy values.

This experiment was conducted on the PC1 dataset, which contained features related to code statistical characteristics, such as lines of code, code comments, number of unique operands in the code, etc. The data set contained more non defective instances than defective instances. Increasing the number of instances, led to a gradual increase in the privacy values of; although not as significant as it was as in the case of the previous datasets. One reason is related to the number and diversity of the features in this dataset. Most features in this dataset are related to code attributes, thus it lacks the diversity needed to create generalized clusters that results when diverse features exist in the dataset.

The PC2 dataset contained more features than the PC1 dataset. Although both datasets shared a similar distribution in terms of the number of instances, the PC2 dataset contained more features on the complexity characteristics of the code. The first experiment measured privacy by varying the number of instances in the clusters when creating generalized instances. As opposed to the PC1 dataset, the PC2 dataset yields more privacy when generalized. Once again, this is related mainly to the diversity of the generalized features. Due to the high level of diversity in terms of the values of attributes, the achieved level of privacy was high, i.e. $>$ 3, as shown when the number of instances was increased to 15 per cluster.

The PC3 dataset had the same set of features as the PC2 dataset, however it was noticed that PC3 had more instances than PC2. Increasing the number of instances per cluster led to higher privacy values; however, it was also observed that increasing the number of instances did not significantly increase the privacy values.

The characteristics of the PC4 dataset were similar to those of the previous datasets, but with except more instances. An increase in the privacy values was noted when the number of instances per cluster was increased. The highest value of achieved conditional privacy was 2.649. The lowest value was 1.36 and it was achieved when the number of instances was 5.

Figure 9.

Number of instances before and after reduction for all datasets used.

Figure 10.

Privacy values before and after reduction for all datasets used.

4.2.2 The impact of data reduction on privacy

The second experiment measured the effects of reducing the size of the training dataset and measuring how such a reduction affects the total privacy. For the reduction step, and given the small number of defective instances, we used Tomek link to remove the instances of the majority class only. AllNN was also used to remove the instances of the minority class, and to reduce the number of removed instances we used the $k=$ 5 to only remove very noisy instances from the minority class. The number of instances before and after reduction for all the datasets is shown in Fig. 9.

Privacy was measured before and after data reduction as shown in Fig. 10. This experiment also measured privacy per number of instances.

It was noted that the CM1 dataset attained higher privacy values when the number of instances was increased. While data reduction led to a minor decrease in the values of privacy when a small number of instances was used for generalization, this difference diminished when the number of instances was increased. The implication of these results is that to balance the effect of data reduction, we need to increase the number of instances per cluster. Data reduction preserves the level of diversity of the resulting data, but we may still need to increase the number of instances per cluster to lower the possibility of background knowledge attacks. This is demonstrated by higher privacy values when the number of instances per cluster is increased.

For the KC1 dataset, since the number of instances is higher from both classes we applied both Tomek link and AllNN to both classes resulting in a higher reduction percentage, as shown in Fig. 10. We then measured the privacy values before and after reduction. The results are shown in Fig. 11. It is noticed that privacy values increased as we increased the number of instances per cluster. Additionally, increasing the number of instances lowered the effect of data reduction. At $k=$ 150 and higher values, privacy after data reduction was even higher than privacy values without reduction.

Since the KC2 dataset contained a smaller number of instances, the rate of reduction was lower, as demonstrated in Fig. 9. Using both approaches, the resulting dataset was reduced by approximately 18% of the original dataset. When measuring the privacy values after reduction, we did not notice significant differences between before and after reduction, as shown in Fig. 10. However, the privacy values for the reduced data were slightly higher when the number of instances per cluster was increased.

We noticed that the KC3 dataset had a more balanced distribution of both classes. Overall, about 18–20% of the instances were removed, as shown in Fig. 9. Privacy analysis to compare between before and after data reduction is shown in Fig. 10. It is noticed that although the number of instances was small, the after-data reduction achieved higher privacy after the number of instances per cluster was increased to more than 15. The diversity of features compensated the loss of privacy caused by reducing the number of records.

Since the number of defects in the MC1 dataset was in the minor class, we only applied Tomek link to the majority class, which was the non-defective class. Then we applied AllNN to both classes, which led to a 17% reduction in the size of the minority class and 26% reduction in the size of the majority class, as shown in Fig. 9. The results of privacy analysis after data reduction are shown in Fig. 10. Although the privacy values for the “before reduction” case were higher when small number of instances were selected for each group, higher values of instances led to similar privacy results.

For MC2 we applied both data reduction techniques on the original dataset to produce the reduced dataset, as shown in Fig. 9. Using both reduction techniques reduced the size of the dataset about 20%, as shown in Fig. 9. Privacy results after reduction show that features diversity is an important factor in balancing the effects of data reduction, as demonstrated in Fig. 10. We noticed an increase in the privacy values after the number of instances was increased to more than 20.

For MW1 we had a small number of instances for the minority class, so we ran this experiment using Tomek links to reduce the size of the majority class and only AllNN to reduce the size of the minority class. The results of data reduction are shown in Fig. 9 and the privacy results after data reduction are shown in Fig. 10. The results show that there are no significant differences between privacy results before and after reduction although there were relatively higher privacy values before reduction; one reason for this being the data imbalance issue. The privacy values before and after reduction are still shown to be close to each other.

Since the PC1 dataset does not contain a high number of the positive class, we performed the reduction step using the Tomek link approach. The results after reduction are shown in Fig. 9, while the results of privacy analysis are shown in Fig. 10. There were no significant differences between the privacy values before and after data reduction. Although this dataset lacks the attribute diversity, it was noticed that privacy values are not affected by the reduction steps.

For the PC2 dataset we applied the Tomek link approach to the majority class only and AllNN to the minority class, and then we reported the reduced dataset in Fig. 9. We then measured privacy before and after data reduction and these results are shown in Fig. 10. At the beginning of the generalization process, privacy for “after reduction” exhibited a minor decrease, then the differences between the before and after data reduction diminished.

We reduced the size of the PC3 dataset using both techniques AllNN and Tomek links. We used both techniques because this dataset had more instances of the minority class than the previous datasets. The results of the reduction step are shown in Fig. 9, while privacy analysis for the after-data reduction is shown in Fig. 10. In this dataset, a smaller number of instances were discarded than in other datasets, which either indicates that the datasets contain less noise or that the applied data reduction approaches were unable to identify the noisy records. It was observed that the data reduction step had no significant effect on the resulting conditional privacy results.

Since the PC4 dataset was not balanced, we only applied the Tomek link approach to remove the instances of the majority class, and then AllNN was applied on both classes. The results of the reduction step are shown in Fig. 9, while privacy values before and after data reduction are shown in Fig. 10. Although before data reduction had higher privacy at the beginning, increasing the number of instances per cluster increased the privacy values which narrowed the differences between the two values.

4.2.3 Utility results

The final experiment on this dataset measured utility under different scenarios. The original dataset was divided into training data and testing data, then three training datasets were used to create three classification models, using the original dataset, the dataset before reduction and the dataset after reduction. The same testing data was used to test each of those three models. The subsections below describe the results for each dataset.

4.2.3.1. CM1 dataset

Figure 11 measures classification accuracy of the original dataset, the anonymized dataset before reduction, and the anonymized dataset after reduction for the CM1 dataset.

Figure 11.

Utility metrics (CM1 dataset).

We can make several observations from the above results. First, our approach preserved the utility of the original dataset, whose overall recall was 89%. When the same testing dataset was utilized on the model created on the anonymized data, without any reduction, we attained about 81% rcall. However, when the reduction step was implemented and the model retrained using the reduced data, we observed an increase in the accuracy value as well as other classification metrics. This included higher TP rate, higher precision, recall, and $f$ -measure values.

4.2.3.2. KC1 dataset

Figure 12 shows the classification accuracy of the original dataset, the anonymized dataset before reduction, and the anonymized dataset after reduction for the KC1 dataset. Although there is some similarity with the results of the previous dataset, it is observed that “after data reduction” had more advantage regarding the accuracy values than before reduction. The reduction step also reduced the FP rate for the positive class (the defective class).

Figure 12.

Utility metrics (KC1 dataset).

Figure 13.

Utility metrics (KC2 dataset).

4.2.3.3. KC2 dataset

The following confusion matrices show the results of classification metrics when the same testing data was used as input to classification models created based on the original data, data before reduction and data after reduction. The results are reported in Fig. 13.

We can summarize the results shown in Fig. 13 as follows: First, there was a significant difference in the values of classification metrics between the data before and after anonymization. That is, the classification results were affected when the data were anonymized. However, this happened only before data reduction. After data reduction the values of classification metrics increased relatively. This experiment proved the advantage of data reduction on the classification results. Second, the false positive values were smaller after data reduction, which also shows that reduction also improved the classification results for the minority class.

4.2.3.4. KC3 dataset

Figure 14 measures the classification accuracy of the original dataset, the anonymized dataset before reduction, and the anonymized dataset after reduction for the KC3 dataset. Reduction of the dataset led to better results in terms of many classification metrics. Nevertheless, we noticed a minor increase in misclassifying non-defective records as defective as shown when comparing 17. One potential reason for this is that after data reduction, fewer similar instances are produced which happens with other experiments, however, since the features were diverse in this dataset, such an effect became clearer and eventually led to an increasing of the miscalculation rate of negative examples.

Figure 14.

Utility metrics (KC3 dataset).

Figure 15.

Utility metrics (MC1 dataset).

4.2.3.5. MC1 dataset

Figure 15 highlights the classification accuracy of the original dataset, the anonymized dataset before reduction, and the anonymized dataset after reduction for the MC1 dataset.

Figure 15 demonstrates several interesting patterns. First, in terms of accuracy utility for the after-data reduction case achieved results close to the original dataset. Although the results for the after-data reduction were better than the before data reduction, the false positive rate for the minority class as slightly higher after data reduction. However, this came at the cost of a lower true positive rate for the majority class.

4.2.3.6. MC2

Figure 16 indicates the classification accuracy of the original dataset, the anonymized dataset before reduction, and the anonymized dataset after reduction for the MC2 dataset. The results for this dataset clearly indicate the benefits of our data reduction approach, particularly when the dataset is of a small size but with a diverse number of sensitive features. The recall was around 67% compared to 60% when no reduction was applied. This dataset yielded lower accuracy even compared to the original dataset. Although the dataset was more balanced than the previous datasets, it gained less classification accuracy. However, the main objective of our approach was not to gain higher accuracy, but to decrease the differences in accuracy between the original and anonymized data.

Figure 16.

Utility metrics (MC2 dataset).

Figure 17.

Utility metrics (MW1 dataset).

4.2.3.7. MW1

Figure 17 shows the classification accuracy of the original dataset, the anonymized dataset before reduction, and the anonymized dataset after reduction for the MW1 dataset. The overall results show the benefits of the data reduction step in increasing the values of several classification metrics. Compared to before data reduction this yielded 78% recall, this value increased after data reduction, reaching around 81%.

4.2.3.8. PC1

Figure 18 demonstrates the classification accuracy of the original dataset, the anonymized dataset before reduction, and the anonymized dataset after reduction for the PC1 dataset. There were no significant differences between classification metrics for before and after anonymization. In fact, it is observed that there were very minor differences between the values of accuracy before and after anonymization. However, there is an increase in the values of false positives after anonymization than before anonymization, which is explained by the very few instances of the minority class.

Figure 18.

Utility metrics (PC1 dataset).

Figure 19.

Utility metrics (PC2 dataset).

4.2.3.9. PC2

The final experiment measured the utility of the resulting data in terms of several classification metrics as shown in Fig. 19. There were no significant differences between the accuracy values of the original dataset and the anonymized datasets. In addition, data reduction yielded better results than before reduction. It is also noted that the after-data reduction yielded a lower FP rate than the original datasets.

4.2.3.10. PC3

Figure 20 displays the classification accuracy of the original dataset, the anonymized dataset before reduction, and the anonymized dataset after reduction for the PC3 dataset. Two main observations can be drawn from the results of this aspect of the experiments, as shown in Fig. 20. First, the result clearly indicates the benefits of reducing the size of the dataset, although the difference is not as significant as in the previous experiments.

Figure 20.

Utility metrics (PC3 dataset).

4.2.3.11. PC4

Figure 21 measures the classification accuracy of the original dataset, the anonymized dataset before reduction, and the anonymized dataset after reduction for the CM1 dataset. The recall of the original dataset was around 90% compared to 88% when the reduced dataset was used and 87% when the non-reduced dataset was used. This shows that there are no significant differences between before and after data anonymization. However, this is associated with higher values of accuracy for the after-data reduction case.

4.3 Utility and privacy comparison with existing techniques

We compared our approach with three existing techniques. All such techniques obfuscate the data by morphing its values via a randomization function between-class instance. Although the MORPHed value can ensure the dataset is sufficiently privatized, it does not always guarantee that an instance does not move across the boundary between the original instance and instances of another class, which means that it may be difficult to keep the MORPHed value close enough to the original to maintain the utility of the dataset.

Table 10
Comparison with existing approaches – accuracy-based comparison

Dataset	Accuracy on original data	Our approach	MORPH $+$ CLIFF	CLIFF	LACE2
CM1	0.89	0.83	0.74	0.69	0.80
MC1	0.95	0.88	0.79	0.77	0.82
MC2	0.72	0.67	0.63	0.61	0.65
KC1	0.98	0.85	0.69	0.63	0.81
KC2	0.89	0.86	0.78	0.69	0.79
KC3	0.81	0.79	0.77	0.65	0.79
PC1	0.99	0.96	0.90	0.86	0.92
PC2	0.96	0.95	0.89	0.83	0.90
PC3	0.87	0.84	0.79	0.71	0.81
PC4	0.90	0.88	0.81	0.76	0.83
MW1	0.88	0.85	0.79	0.66	0.85

Figure 21.

Utility metrics (PC4 dataset).

We compared the results achieved by the MORPH technique and its successors with our approach. To conduct a fair comparison, the training part of each dataset was anonymized using those techniques and our technique, and then the same testing dataset was applied to each approach. The results are reported in Table 10. The accuracy values clearly show that our approach achieves significantly higher accuracy values. Of the existing approaches LACE2 achieves better results. LACE 2 also applies a data reduction technique that removes the noisy instances.

The final experiment also compared our approach with other existing techniques such as CLIFF, LACE 2, and MORPH, in terms of privacy. The experiment was conducted by varying the number of instances per cluster. It should be noted that this value varies from one dataset to another as shown in Table 11. The results shown in Table 11 clearly indicate that our approach yields better privacy for most of the datasets, even on small number of instances per cluster. This result is noticed for our approaches before and after data reduction as demonstrated in the table. It is important to mention that while our approach inherits some of the characteristics of the K-anonymity approach, applying SVD per cluster enhanced the level of privacy for each of the resulting clusters and eventually the privacy values of each dataset.

Table 11

Privacy-based comparison with the existing techniques

Dataset	Technique/privacy	Number of instances per cluster
		3	6	10	14	20
CM1	Our approach-before reduction	0.94	0.95	1.12	1.39	1.36
	Our approach-after reduction	0.79	0.77	1.10	1.35	1.33
	Lace 2	1.11	1.11	1.11	1.11	1.11
	Morph	0.92	0.92	0.92	0.92	0.92
	Cliff	1	1	1	1	1
		5	10	15	20	30
MC1	Our approach-before reduction	2.08	2.14	2.26	2.81	2.88
	Our approach-after reduction	1.66	1.91	2.3	2.8	2.9
	Lace 2	0.89	0.89	0.89	0.89	0.89
	Morph	0.91	0.91	0.91	0.91	0.91
	Cliff	0.9	0.9	0.9	0.9	0.9
		5	10	15	20	30
MC2	Our approach-before reduction	1.917	2.02	2.07	2.18	2.57
	Our approach-after reduction	1.56	1.7	1.9	2.3	2.9
	Lace 2	1.2	1.2	1.2	1.2	1.2
	Morph	1.03	1.03	1.03	1.03	1.03
	Cliff	0.87	0.87	0.87	0.87	0.87
		10	30	50	80	120
KC1	Our approach-before reduction	0.45	0.59	0.66	1.23	1.66
	Our approach-after reduction	0.29	0.31	0.36	0.7	1
	Lace 2	0.5	0.5	0.5	0.5	0.5
	Morph	0.57	0.57	0.57	0.57	0.57
	Cliff	0.63	0.63	0.63	0.63	0.63
		10	20	30	40	50
KC2	Our approach-before reduction	0.55	0.58	0.58	0.59	0.78
	Our approach-after reduction	0.389	0.525	0.615	0.621	0.803
	Lace 2	0.76	0.76	0.76	0.76	0.76
	Morph	0.71	0.71	0.71	0.71	0.71
	Cliff	0.75	0.75	0.75	0.75	0.75
		5	10	20	30	40
PC1	Our approach-before reduction	0.52	0.66	0.62	0.66	0.70
	Our approach-after reduction	0.49	0.61	0.63	0.68	0.7
	Lace 2	0.69	0.69	0.69	0.69	0.69
	Morph	0.71	0.71	0.71	0.71	0.71
	Cliff	0.66	0.66	0.66	0.66	0.66
		5	10	20	30	50
PC3	Our approach-before reduction	1.61	1.84	1.88	1.97	2.12
	Our approach-after reduction	1.65	1.75	1.92	1.95	2.1
	Lace 2	0.79	0.79	0.79	0.79	0.79
	Morph	0.92	0.92	0.92	0.92	0.92
	Cliff	1.02	1.02	1.02	1.02	1.02
		5	10	15	20	25
PC4	Our approach-before reduction	1.36	1.39	1.39	1.45	1.46
	Our approach-after reduction	1.42	1.52	1.51	1.59	1.57
	Lace 2	0.91	0.91	0.91	0.91	0.91
	Morph	0.86	0.86	0.86	0.86	0.86
	Cliff	0.9	0.9	0.9	0.9	0.9
		5	10	15	20	25
MW1	Our approach-before reduction	1.80	2.31	2.69	2.72	2.88
	Our approach-after reduction	1.71	2.2	2.38	2.66	2.89
	Lace 2	1.2	1.2	1.2	1.2	1.2
	Morph	1.1	1.1	1.1	1.1	1.1
	Cliff	1.6	1.6	1.6	1.6	1.6

5. Time efficiency

In this experiment, we tested the execution time of our anonymization approach and compare it to other approaches. On the first three datasets, the results can be generalized to the rest of datasets. The results clearly indicate that our approach does not add an extra overhead to the anonymization process due to the clustering step. It can be noticed that our approach is competitive to other approaches. It achieves comparable or close results to other approaches as demonstrated in Fig. 22.

Figure 22.

Efficiency of the anonymization process.

When measuring the time in second, our approach anonymizes all three datasets in less than 0.35 seconds. The results of other approaches are close to ours, with minor advantage to CLIFF approach. However this comes at the privacy and accuracy budget.

6. Discussion and conclusions

6.1 Results discussion

In this research we proposed, implemented, and tested an approach for preserving the privacy of software data when sharing it publicly. The work contributes to the field from two perspectives. First, we show that shared data can still be used to identify defects with an accuracy that is quite close to that achieved from the original data. Second, we introduced an approach for reducing noisy records that may affect the utility of the shared data. We also showed that removing such records significantly improves accuracy values after anonymization. In addition, we proved that our reduction approach has no effect on the achieved privacy. We compared our approach with several well-known approaches such as LACE2, CLIFF and MORPH and showed that our approach outperforms those techniques in terms of both privacy and utility.

Unlike the other techniques, our approach does not add random noise using the features of nearest neighbor instances of the opposite class. In fact, this significantly reduced the accuracy when the anonymized data was tested using approaches such as LACE and CLIFF. We believe that this increases the similarity between instances of different classes after anonymization. Accordingly, approaches such as CLIFF, LACE and MORPH result in more noise being introduced to the data.

In addition, approaches such as LACE remove instances that are similar to the existing ones. This negatively affects the utility of the resulting data, since it introduces similarity of instances from different classes, which misleads the classifier used to test the anonymized data. Privacy-wise, this also increases the number of unique instances that are not similar to any of the existing instances, which may boost the number and types of background knowledge attacks. Instead, our approach removes similar instances per class that mislead the classifiers and most of the time misclassified by many classifiers. Then to avoid background knowledge and mapping attacks, we increased the similarity after anonymization using a generalization-based anonymization approach and perturbed data using a well-known SVD to add noise per cluster and per class.

We used NASA public datasets to test our approach. We conducted several types of experiment, with the first set of experiments testing our approach from the privacy perspective. The following are the major findings related to privacy:

1.
The data reduction approach does not affect the values of privacy. Whereas one may argue that discarding some instances may increase the probability of data breach, we showed that increasing the number of instances per cluster can control this. In fact, when increasing the number of instances per cluster we were able to reach analogous privacy values compared to data before reduction.
2.
Data reduction has no effects on diversity since we removed instances from both classes. In addition, we tried to remove instances that negatively affect utility not privacy.
3.
While our results prove that privacy values are affected by size of dataset, we showed that it is still possible to control this on small datasets by increasing the $k$ values to remove a smaller number of instances.
4.
The results also indicate that there were no significant differences between attributes contribution to the overall privacy values.

The following are the major findings related to accuracy of the resulting data after anonymization

1.
We showed that accuracy values after anonymization are comparable to those values before anonymization.
2.
We showed that data reduction significantly increases the TP rate, reduced the FP rate and eventually increases the accuracy values.
3.
We showed that compared to other approaches, our approach leads to higher accuracy values. We demonstrated this by feeding the same testing data to our approach and other approaches. This result is explained by randomization technique used by those approaches which adds noise by taking the most similar instances of the opposite class and uses the features of that instance in adding a random noise. Researchers have criticized this and showed that it is not reasonable to control the amount of noise without reducing the quality of the resulting dataset.

7. Threat to validity

Algorithms implemented in this research can be mainly applied to data regarding software products’; therefore applying our approach to other domains is out of the scope of this work and could be considered for future work. We will focus on using our approach for defect prediction, which is considered a classification problem. Evaluating the proposed methods using other data mining algorithms such as clustering is beyond the scope of our work, which only focuses on privacy related issues.

8. Conclusions and future work

In this paper, we introduced an approach to perturb sensitive attributes of software defect data as it has been shown that such data is vulnerable to different types of attack when shared without anonymization. We showed that our anonymization approach produces data that inherits the statistical characteristics of the original data. Instead of applying simple randomization approaches, we applied a hybrid approach to reduce the size of the original dataset and then anonymize it. The main objective of this approach was to remove records that mostly affect the utility of the produced data. First, sensitive attributes were identified; we then identified then discarded records using two approaches, Tomek links and AllNN. Unlike the existing techniques, which only consider removing low ranked records, we focused on removing similar records that were close to the decision boundary and had different classes. Those records are most likely misclassified after anonymization. We showed that privacy values are not affected after anonymization. In addition, we showed a significant improvement in the accuracy value after data reduction. Finally, we compared the results of our experiments with several existing techniques and showed that our approach outperformed many of those approaches.

The proposed approach could be extended to address its limitations and enhance its effectiveness. First, it could cover not only numerical but also textual attributes. It has been shown that it is not only the code that is vulnerable to attacks but also other important documents such as requirement documents, thus it is important to extend our approach to consider textual features, for instance information about platforms, operating environments, and server types are all sensitive attributes that needs to be anonymized. However, as opposed to numerical features, textual features require approaches that preserve the semantic-based characteristics of the proposed approach.

Moreover, we plan to test our approach on newer datasets, other than the NASA dataset. We also plan to apply other techniques such as differential privacy for anonymization of software engineering data.

References

Aggarwal

C.C.

and Philip

S.Y.

, A general survey of privacy-preserving data mining models and algorithms, in: Privacy-Preserving Data Mining: Advances in Database Systems, Springer, Vol. 34, 2008, pp. 11–52.

Aggarwal

C.C.

and Yu

P.S.

, On static and dynamic methods for condensation-based privacy-preserving data mining, ACM Transactions on Database Systems (TODS) 33(1) (2008), 2.

Agrawal

and Aggarwal

C.C.

, On the design and quantification of privacy preserving data mining algorithms, in: Paper Presented at the Proceedings of the Twentieth ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, 2001, pp. 247–255.

Agrawal

and Aggarwal

C.C.

Agrawal

and Srikant

, Privacy-preserving data mining, Paper presented at the ACM Sigmod, Vol. 29, No. 2, 2000, pp. 439–450.

Alejo

Sotoca

J.M.

Valdovinos

R.M.

and Toribio

, Edited nearest neighbor rule for improving neural networks classifications, in: Paper Rresented at the International Symposium on Neural Networks, 2010.

Baker

Knoppers

B.M.

Phillips

van Enckevort

Kaufmann

Lochmuller

and Taruscio

, Privacy-preserving linkage of genomic and clinical data sets, IEEE/ACM Transactions on Computational Biology and Bioinformatics 16(4) (2018), 1342–1348.

Beckers

and Heisel

, A foundation for requirements analysis of privacy preserving software, in: International Cross-Domain Conference and Workshop on Availability, Reliability, and Security (CD-ARES), Aug 2012, Prague, Czech Republic, 2012, pp. 93–107.

Benjamin

Fung

Wang

Chen

and Yu

, Privacy-preserving data publishing: A survey of recent developments, ACM Computing Surveys 42(4) (2010), 1–53.

10.

Berlioz

Friedman

Kaafar

M.A.

Boreli

and Berkovsky

, Applying differential privacy to matrix factorization, in: RecSys ’15: Proceedings of the 9th ACM Conference on Recommender Systems, September 2015, 2015, pp. 107–114.

11.

Bertino

Lin

and Jiang

, A survey of quantification of privacy preserving data mining algorithms, in: Privacy-Preserving Data Mining, Springer, 2008, pp. 183–205.

12.

Boetticher

, The PROMISE repository of empirical software engineering data. Available ate: http://promisedata.org/repository, 2007.

13.

Budi

and Jiang

, KB-anonymity: a model for anonymized behaviour-preserving test and debugging data, Paper presented at the ACM SIGPLAN Notices, Vol. 46, No. 6, 2011, pp. 447–457.

14.

Casino

Domingo-Ferrer

Patsakis

Puig

and Solanas

, A k-anonymous approach to privacy preserving collaborative filtering, Journal of Computer and System Sciences 81(6) (2015), 1000–1011.

15.

Chen

Zhang

Cui

Z.-Q.

and Ju

X.-L.

, DP-share: Privacy-preserving software defect prediction model sharing through differential privacy, Journal of Computer Science and Technology 34(5) (2019), 1020–1038.

16.

Clifton

Kantarcioglu

Vaidya

Lin

and Zhu

M.Y.

, Tools for privacy preserving distributed data mining, ACM SIGKDD Explorations Newsletter 4(2) (2002), 28–34.

17.

Clifton

and Tassa

, On syntactic anonymity and differential privacy, in: Paper Presented at the IEEE 29th International Conference on Data Engineering Workshops (ICDEW’13), 2013.

18.

Cormode

Procopiuc

C.M.

Shen

Srivastava

and Yu

, Empirical privacy and empirical utility of anonymized data, in: Paper Presented at the 29th IEEE International Conference on Data Engineering Workshops (ICDEW’13), 2013.

19.

Dwork

, Differential privacy, Encyclopedia of Cryptography and Security, Springer, Boston, MA, 2011, 338–340.

20.

Evfimievski

Srikant

Agrawal

and Gehrke

, Privacy preserving mining of association rules, Information Systems 29(4) (2004), 343–364.

21.

Koru

A.G.

Chen

and El Emam

, A tree-based approach to preserve the privacy of software engineering data and predictive models, in: PROMISE ’09: Proceedings of the 5th International Conference on Predictor Models in Software Engineering, May 2009 Article No: 3 Pages 1–12, 2009.

22.

Gal

T.S.

Tucker

T.C.

Gangopadhyay

and Chen

, A data recipient centered de-identification method to retain statistical attributes, Journal of Biomedical Informatics 50 (2014), 32–45.

23.

Golle

, Revisiting the uniqueness of simple demographics in the US population, in: Paper Presented at the Proceedings of the 5th ACM Workshop on Privacy in Electronic Society, 2006, pp. 77–80.

24.

Grechanik

Csallner

and Xie

, Is data privacy always good for software testing? in: Paper Presented at the 2010 IEEE 21st International Symposium on Software Reliability Engineering, 2010, pp. 368–377.

25.

Han

W.K.

and Philip

S.Y.

, Privacy-preserving singular value decomposition, in: Paper Presented at the 2009 IEEE 25th International Conference on Data Engineering, 2009, pp. 1267–1270.

26.

Hong

Vaidya

Karras

and Goel

, Collaborative search log sanitization: Toward differential privacy and boosted utility, IEEE Transactions on Dependable and Secure Computing 12(5) (2015), 504–518.

27.

Iyengar

V.S.

, Transforming data to satisfy privacy constraints, Collaborative search log sanitization: Toward differential privacy and boosted utility, IEEE Transactions on Dependable and Secure Computing 12(5) (2002), 504–518.

28.

Jalali

Menzies

and Feather

, Optimizing requirements decisions with keys, in: Paper Presented at the Proceedings of the 4th International Workshop on Predictor Models in Software Engineering, 2008, pp. 79–86.

29.

Kocaguneli

Gay

Menzies

Yang

and Keung

J.W.

, When to use data from other projects for effort estimation, in: Paper Presented at the Proceedings of the IEEE/ACM International Conference on Automated Software Engineering, 2010, pp. 321–324.

30.

Kocaguneli

and Menzies

, How to find relevant data for effort estimation? in: Paper presented at the 2011 International Symposium on Empirical Software Engineering and Measurement, 2011, pp. 255–264.

31.

Grechanik

and Poshyvanyk

, Sanitizing and minimizing databases for software application test outsourcing, in: Paper Presented at the 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation, 2014, pp. 233–242.

32.

and Venkatasubramanian

, t-closeness: Privacy beyond k-anonymity and l-diversity, in: Paper Presented at the IEEE 23rd International Conference on Data Engineering, 2007, pp. 106–115.

33.

Jing

X.-Y.

Zhu

Zhang

and Ying

, On the multiple sources and privacy preservation issues for heterogeneous defect prediction, in IEEE Transactions on Software Engineering, Vol. 45, No. 4, 2017, pp. 391–41.

34.

Liu

Kargupta

and Ryan

, Random projection-based multiplicative data perturbation for privacy preserving distributed data mining, IEEE Transactions on knowledge and Data Engineering 18(1) (2006), 92–106.

35.

Machanavajjhala

Gehrke

Kifer

and Venkitasubramaniam

, l l-diversity: Privacy beyond k-anonymity, in: Paper Presented at the 22nd International Conference on Data Engineering (ICDE’06), 2006, pp. 24–24.

36.

Mendes

and Vilela

J.P.

, Privacy-preserving data mining: Methods, metrics, and applications, IEEE Access 5 (2017), 10562–10582.

37.

Moparthi

N.R.

and Geethanjali

, A novel privacy preserving based ensemble cross defect prediction model for decision making, Perspectives in Science 8 (2016), 76–78.

38.

Oliveira

S.R.

and Zaiane

O.R.

, Privacy preserving frequent itemset mining, in: Paper Presented at the Proceedings of the IEEE International Conference on Privacy, Security and Data Mining, Vol. 14, 2002, pp. 43–54.

39.

Oliveira

S.R.

and Zaiane

O.R.

, Privacy preserving clustering by data transformation, Journal of Information and Data Management 1(1) (2010), 37–51.

40.

Peters

and Menzies

, Privacy and utility for defect prediction: Experiments with morph, in: Paper Presented at the 34th International Conference on Software Engineering (ICSE), 2012, pp. 189–199.

41.

Peters

Menzies

Gong

and Zhang

, Balancing privacy and utility in cross-company defect prediction, IEEE Transactions on Software Engineering 39(8) (2013), 1054–1068.

42.

Peters

Menzies

and Layman

, LACE2: Better privacy-preserving data sharing for cross project defect prediction, in: Paper Presented at the Proceedings of the 37th International Conference on Software Engineering, Vol. 1, 2015, pp. 801–811.

43.

Jing

X.-Y.

Zhu

and Cheng

, Privacy preserving via interval covering based subclass division and manifold learning based bi-directional obfuscation for effort estimation, in: 31st IEEE/ACM International Conference on Automated Software Engineering (ASE), Singapore, 2016, pp. 75–86.

44.

Ruth

M.E.

, Employing privacy-preserving techniques to protect control-flow graphs in a decentralized, end-to-end regression test selection framework for web services, in: Paper Presented at the 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops, 2011, pp. 139–148.

45.

Samarati

, Protecting respondents identities in microdata release, IEEE Transactions on knowledge and Data Engineering 13(6) (2001), 1010–1027.

46.

Senarath

and Arachchilage

N.A.

, Why developers cannot embed privacy into software systems?: An empirical investigation, in: Paper presented at the Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering, 2018, pp. 211–216.

47.

Shannon

C.E.

Weaver

Blahut

R.E.

and Hajek

, The mathematical theory of communication, The Bell System Technical Journal 27(1) (1949), 379–423.

48.

Song

Jermaine

and Ranka

, Conditional anomaly detection, IEEE Trans. on Knowl. and Data Eng. 19(5) (2007), 631–645.

49.

Soria-Comas

Domingo-Ferrer

Sánchez

and Martinez

, t-closeness through microaggregation: Strict privacy with enhanced utility preservation, IEEE Transactions on Knowledge and Data Engineering 27(11) (2015), 3098–3110.

50.

Sweeney

, k-anonymity: k-anonymity: A model for protecting privacy, International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 10(5) (2002), 557–570.

51.

Turhan

Menzies

Bener

A.B.

and Di Stefano

, n the relative value of cross-company and within-company data for defect prediction, Empirical Software Engineering 14(5) (2009), 540–578.

52.

Vollmer

, Recital 26 EU GDPR. Retrieved from https://www.privacy-regulation.eu/en/recital-26-GDPR.htm, 2018.

53.

Wang

and Zheng

, Privacy preserving data generation for database application performance testing, TrustBus 2004, Lecture Notes in Computer Science, vol 3184. Springer, Berlin, Heidelberg, 2004.

54.

Xiao

and Tao

, Personalized privacy preservation, in: Paper Presented at the Proceedings of the ACM SIGMOD International Conference on Management of Data, 2006, pp. 229–240.

55.

Zhang

Xiao

Yang

and M. Differentially private histogram publication, The VLDB Journal – The International Journal on Very Large Data Bases 22(6) (2013), 797–822.

56.

Zhang

Han

and Wang

, Singular value decomposition based data distortion strategy for privacy protection, Knowledge and Information Systems 10(3) (2006), 383–397.

57.

Yuan

Chen

Philip

S.Y.

and Yu

, Protecting sensitive labels in social network data anonymization, IEEE Transactions on Knowledge and Data Engineering 25(3) (2013), 633–647.

Privacy preserving defect prediction using generalization and entropy-based data reduction

Abstract

1. Introduction

1.1 General overview

1.2 Motivation example

Table 1 Private dataset

1.4 Research background

1.5 Limitations of the existing techniques and our contributions

2.1 Privacy preserving analytics in software engineering

2.2 Privacy preserving data analytics

3. Research methodology

3.1 Overall research design

3.2.1 Software defect data

Table 3 Sample quasi identifiers in software defect data

Table 8 Resulting data maintaining both diversity and privacy (2 clusters s = 2) each)

4.1 Data analysis and interpretation

4.1.1 Privacy

Table 9 Datasets used in the experiments

4.2.3 Utility results

4.2.3.1. CM1 dataset

4.2.3.2. KC1 dataset

4.2.3.3. KC2 dataset

4.2.3.4. KC3 dataset

4.2.3.5. MC1 dataset

4.2.3.6. MC2

4.2.3.7. MW1

4.2.3.8. PC1

4.2.3.9. PC2

4.2.3.10. PC3

4.2.3.11. PC4

Table 10 Comparison with existing approaches – accuracy-based comparison

6.1 Results discussion

8. Conclusions and future work

References

Table 1
Private dataset

Table 3
Sample quasi identifiers in software defect data

Table 8
Resulting data maintaining both diversity and privacy (2 clusters $s=$ 2) each)

Table 9
Datasets used in the experiments

Table 10
Comparison with existing approaches – accuracy-based comparison