Software fault prediction using machine learning techniques with metric thresholds

2.1 Fault prediction in literature

There is a plethora of research on the relationship between faults and software metrics [23, 26, 37, 45, 51]. Software metrics are utilized for predicting faulty parts in current and future releases of software. Different sets of software metrics have been validated as predictors of fault-proneness including [25, 30, 45, 53]. However, The Chidamber and Kemerer metrics were the most studied among all [38, 39]. In a systematic review, Malhotra has found C&K useful metrics for predicting fault-proneness. Therefore, this research focuses on the use of C&K metrics only for fault prediction.

Previous works on C&K metrics [45] have found an empirical association with fault-proneness [23], fault counts and fault categories [44]. Researchers have built empirical fault prediction models using several methodologies, including statistical models [23, 25, 40], and machine learning models (for example, neural networks, and classification trees) [35]. Such studies on fault-proneness categorized software classes into several groups based on the number of faults in a class. Usually, classes are divided into two groups: faulty classes that had one or more faults in the current release under investigation, and not faulty classes that did not have any faults [35].

Some other researchers used the severity of faults to create several categories (low, medium, and high) [40]. These researchers used the bug repository to extract the information on the severity of bugs as proposed by the developers and the testers of open-source systems. The authors could build sound and plausible models using the severity levels

2.2 Fault data quality

The fault prediction models are used for binary variables in most published work because of the data imbalance and the low variability in the number of faults uncovered in classes. Data imbalance degrades fault prediction performance [18]. Folleco et al. also reported that noise has a large effect on classifiers’ performance [1]. Both studies in [1, 18] have shown that noise in the minority class (faulty instances) has a more detrimental effect on classification performance than noise in the majority class (non-faulty). Gray et al. noted that NASA metrics data needs significant preprocessing before inclusion in machine learning algorithms [9]. They suggested many preprocessing techniques to eliminate parts of the data such as removal of repeated attributes, removal of repeated and inconsistent instances, and enforcing metrics integrity. Gao et al. conducted many sampling techniques followed by feature selection techniques to improve the effectiveness of the software fault prediction [21]. Dallal studied the effect of special methods (such as constructors, destructors, and access methods) on measuring cohesion of classes [16]. Dallal conducted an empirical study to find the effect of including/excluding these special methods for refactoring and predicting faulty classes [16]. The results showed significant differences in cohesion measurements and significant effects on refactoring prediction but no significant effects on fault prediction. Recently, Song et al studied the data imbalance using extensive techniques and found significant improvements in fault-prediction models [38]. Therefore, researchers have used techniques such as oversampling and undersampling to improve the performance of the predictions [7, 38]. Other techniques were also employed including semi-supervised learning and feature selection [2]. Data imbalance could happen because of class noise which can be attributed to either data entry errors or insufficient information used to label instances [17]. In this work, we aim to reduce the data imbalance by using threshold values to increase the instances in the minority class. Thresholds are used to label instances as faulty if they exceed particular threshold values and if the prediction models confirm the outcome.

2.3 Thresholds techniques

The work on fault prediction was extended to include the derivation of threshold values on software. Threshold values are usually used to separate instances into two groups, faulty or not. Threshold values separate between good quality and low quality types in the system. Some researchers have used advanced techniques such as the logistic regression to derive thresholds [39, 42, 44]. The logistic regression technique requires having the fault data in advance to build fault prediction models and then thresholds are derived from the parameters of the logistic regression. The logistic regression was not successful in all studies. Thresholds were identified for some metrics in [42] and no thresholds were identified in [44]. The logistic-based thresholds were also derived again in [39] for object-oriented metrics and thresholds were used in prediction models.

Shatnawi et al. have proposed to derive thresholds using the Receiver Operating Characteristic curve (ROC) [43]. Catal et al. have employed the ROC technique to detect outliers and labeling non-faulty classes as faulty to improve fault predictionperformance [6]. The ROC technique requires knowing which classes are faulty and which are not in advance. The use of thresholds resulting from ROC analysis are affected by fault data which are also used to train and test the prediction models. These models may have a confounding effect as the same data are used to derive thresholds and build the prediction models.

Some researchers have used the statistical distribution of metrics such as frequencies and percentiles to rank classes into several levels [20, 36]. Ferreira et al. have classified classes into three levels: high frequency values, low frequency values, and regular values [20]. Oliveira et al. on the other hand have used percentiles to identify relative threshold values [36]. Shatnawi has proposed to use the log transformation on metrics to reduce skewness in measurement [41]. Shatnawi has proposed to use one standard deviation after transformation for threshold derivation [41].

Statistical methods were successful in finding thresholds for all metrics in the C&K suite. In addition, statistical methods depend on static analysis only and do not require information about faults in systems. Therefore, we select three statistical methods (Shatnawi, Alves, and Vale) that have reported thresholds for the C&K metrics. These methods are proposed to find thresholds and then thresholds are used to find which unlabeled instances are faulty. The results of threshold application are used to improve the quality of the datasets and therefore to improve fault prediction models.

There is plenty of software metrics to measure the internal quality of software metrics. The Chidamber and Kemerer (C&K) [45] are among these metrics. However, the C&K metrics have gained more attention from researchers than other types of metrics because they measure the most important aspects of object-oriented software. The C&K suite measures software properties including size, complexity, aggregation, generalization, coupling and cohesion. C&K has defined the metrics as follows:

•

Coupling between Objects (CBO): the CBO counts the number of couplings of a class to other classes in the system. Coupling increases interconnectedness and interdependence between components of the system, which causes side effects when a class changes. Hence, large values of CBO are considered riskier than low values. When a threshold is identified, a CBO value that exceeds the threshold value is considered more fault-prone than otherwise.

•

Response for Class (RFC): the RFC counts the responsibility of a class by counting local and remote methods involved in the activities of a class. Large responsibility causes more side effects on the quality of the software. RFC values that are larger than a threshold are faultier than otherwise and require more quality investigation.

•

Weighted Methods per Class (WMC): the WMC counts the complexity in a class by counting the number of methods. A large number of methods are indicated as god classes. Such classes are problematic and difficult to maintain and test. Classes that have WMC values are larger than a particular threshold require more attention and more quality investigation.

•

Depth of Inheritance Hierarchy (DIT): the DIT counts the number of direct ancestors of a class to the root class in the inheritance hierarchy. DIT is an indicator of the extent of inheritance. Classes that have DIT values larger than a threshold require more effort to maintain and test.

•

Number of Child Classes (NOC): the NOC metric counts the number of direct children of a class. Large values of NOC indicate the number of specializations of a class and therefore more complicated behavior in the system.

•

Lack of Cohesion of Methods (LCOM): the LCOM metric measures interrelationships between methods and data fields in a class. If the relationship is strong then the class is coherent and represents similar functionalities. Low cohesive classes require more refactoring and maintenance. The LCOM is calculated from two parameters (P and Q). P counts method pairs using no data fields in common, whereas Q counts method pairs that have shared data fields. Therefore, the LCOM is calculated as follows:

$\text{LCOM}=(P>Q)\mathrm{?}(P-Q):0.$(1)

Large values of LCOM denote lack of cohesion and therefore such classes require more maintenance and testing.

3.2 Data sets

In this research, the study is conducted on fault data from the Promise repository that is publicly available for reuse by researchers. The use of this data is easier for more experimentation and replication. Fault data were collected from the repositories of systems under investigation as shown in the work of [26, 27]. The tool has used regular expressions to analyze logs of every software and has associated the bugs found in the repository with relevant classes. Whenever a bug is associated with a class then the bug count is incremented by one for the class. The procedure is applied for all systems under study. The bytecode of systems was analyzed using a specialized static analysis tool called Ckjm for java applications only. Ckjm is an open-source tool and can be validated or improved.

Five large open-source systems and five large commercial systems are used in this research. All open-source systems are built under the license of Apache systems. The study is conducted on the following five open-source systems.

•

Ant: A Java Building tool (http://ant.apache.org)

•

Camel: A versatile open-source integration framework (http://camel.apache.org)

•

Ivy: A dependency manager focusing on flexibility and simplicity (http://ant.apache.org/ivy/)

•

Jedit: A Java IDE and editor (http://jedit.org/)

•

Tomcat: an implementation of the Java Servlet, JavaServer Pages, Java Expression Language and Java WebSocket technologies (http://tomcat. apache.org/)

The five commercial software systems are large applications that were developed for insurance businesses. The metrics and fault data were collected similarly as for the open-source systems. These systems are given anonymous names.

Data distribution affects the quality of the machine learning models and therefore it is necessary to understand the fault distribution. Table 1 shows the imbalance in fault classes in most systems. Faulty classes are the minority whereas most classes are labeled non-faulty. However, this behavior is not necessarily correct as more faults may be uncovered in future releases of software.

OPEN IN VIEWER

Table 1

The fault distribution for ten systems

Data set	#Not faulty	#Faulty	#Classes
Ant	579	166	745
Camel	777	188	965
Ivy	312	40	352
Jedit	319	48	367
Tomcat	781	77	858
Prop1	2557	268	2625
Prop2	3513	85	3598
Prop3	2078	229	2307
Prop4	2030	365	2395
Prop5	2641	213	2854

3.3 Threshold identification techniques

Threshold values are reference points that can be used to investigate when classes can be at risk when exceeding a particular value. In this work, three threshold identification techniques are used for validation The three techniques were selected for the following reasons:

•

All techniques should use statistical distribution parameters. Shatnawi’s work uses the standard deviation and the mean after applying the log transformation. Vale’s work uses the percentiles to select classes as bad smells on a large benchmark. Alves’s work selects the top 90% of classes after applying a weighted ratio.

•

All techniques should be validated for quality prediction. Shatnawi’s thresholds were found correlated with fault-proneness. Vales’ and Alves’ thresholds were found correlated with bad smells, e.g., large classes.

•

All techniques should be configured to characterize a metric value into several labels. However, in this work we have selected two labels to be consistent with fault-prediction models.

•

All techniques should be easy to implement and repeat on future systems.

3.3.1 Shatnawi thresholds

In this research, Shatnawi proposed to use the mean and standard deviation for each metric to derive thresholds. The log transformation is used to reduce data skewness. This methodology was originally proposed and validated in [41]. Thresholds are calculated as in Eq. (2).

T ′ = μ + σ(2)

Where μ denotes the mean and σ denotes the standard deviation. T’ is calculated for each metric after the log transformation. Threshold values ( T ′ ) are reversed back using the exponential function as in Eq. (3).

T = Exp ⁢ ( T ′ )(3)

The derived thresholds are calculated from the data distribution and since each system has different parameters then we do not expect to have the same threshold values for a particular metric among different systems. In [41], threshold values of C&K were derived (WMC, CBO, RFC, LCOM DIT and NOC).

This threshold derivation technique is easy to implement and interpret. The selected instances are outliers at the right-hand side of the distribution. The statistical distribution is skewed to the right and depending on statistics without log transformation may produce biased parameters for threshold derivation. In addition, this technique reports thresholds for individual systems. Although this technique cannot be generalized to other systems, the statistical distribution gives information about the data quality.

Alves et al. proposed a benchmark-based approach for threshold derivation [50]. Alves method consists of six steps. First, software metrics are calculated for each system. Metrics are separated for each system in a separate file. Second, a weighted ratio is calculated for each class in each system. The weighted ratio is calculated by dividing the LOC of each class by the total LOC of all classes. For example, if a class has 100 LOC and the system is composed of 10000 LOC, then the weighted ratio is 100/10000. In the third and fourth steps, the weights of each class at each system are aggregated. In the fifth and sixth steps, the classes are ranked by weight and the top 90% are used to derive thresholds. A threshold at 90% means that this threshold represents 90% of the overall code. In other words, this threshold selects 10% of the code as low-quality parts that require more quality assurance. The authors proposed thresholds at four levels: low (between 0–70%), moderate (70–80%), high (80–90%), and very high ( $>$ 90%). In this research, we are using thresholds to improve the fault prediction model. Therefore, we propose to use the top thresholds at 90% only. Alves threshold values are shown in Table 2. These thresholds are derived from 100 systems in the work of [14].

Vale has divided his methodology into six steps. The first step starts with metrics collection from a large benchmark of systems and metrics are collected for each class. In the second step, the percentages are computed for each class with respect to the total number of classes in a benchmark. In the third step, metric values are ordered ascendingly in a way equivalent to calculating a density function. In the fourth step, the percentages are aggregated for all classes per metric value. In the fifth step, thresholds are derived for the top 90% and 95% of classes for each metric. In the sixth step, thresholds are derived from the lower bounds below 3% and 15% for small classes. In our research, the top percentages are only relevant, hence step six is irrelevant to this work. Vale thresholds are shown in Table 2. These thresholds are derived from a benchmark of 100 systems.

Threshold values in Table 2 were previously published in [41, 14]. Vale has initially conducted his methodology on software product lines. He extended his work to other types of systems for generalizability. The authors selected a benchmark that consists of more than 100 systems from Qualitas Corpus [11]. The benchmark was used to find thresholds using the Alves’ method as well. Thresholds derived using Alves and Vale methodologies are used to improve fault-prediction models.

4.1 Traditional fault-prediction models

The models without thresholds are experimented using five classifiers. For validation, we run the experiment using the 10-fold cross-validation for all systems. The MCC results are shown in Table 4. MCC measures the correlation between the predicted and the actual classes. MCC values that are significantly larger than zero are favored for classification.

The MCC values, shown in Table 4, are weak for most classifiers (close to 0.0) whereas few have performance that can reach a value 0.40. The MCC values in Table 4 are closer to a random prediction than a perfect prediction for all classifiers. Some classifiers for the Ant system have MCC values that is less than zero, which is an indicator of weak classifiers. The MCC values for all classifiers show weak prediction results. Therefore, RQ1 is answered and the data imbalance affects the fault prediction performance greatly.

4.2 Data imbalance effect

Data imbalance affects fault-prediction. Therefore, we use thresholds to find whether data imbalance can be reduced using threshold values. Threshold values are used to select the instances that were marked not fault-prone but that exceed threshold values. Instances that are larger than one of the thresholds and non-faulty are kept unlabeled and separated into the testing dataset. The rest of instances are used as training dataset. This procedure is repeated for the three threshold identification techniques for each system.

The three threshold identification techniques (Shatnawi, Alves and Vale) have selected a different number of instances (N) as shown in Tables 5–7. These instances are selected as fault-prone using thresholds. However, we need to make sure that the threshold identification techniques are consistent in predicting the fault-pone instances. These instances (N) are unlabeled and separated into a testing dataset whereas the rest are used as training dataset.

The five prediction models are trained and the testing datasets are provided to the models. For brevity, the number of instances that are predicted as faulty are counted and provided in Tables 5–7 for Shatnawi, Alves and Vale techniques, respectively. For each classifier, the total number of instances that are classified as faulty are reported in the columns NB, LR, J48, KNN, JRip. For example, NB has classified 169 out of 177 classes as faulty.

There is a variation in instance classification among the five classifiers. To get reliable results we have considered a class faulty if it was predicted as faulty using all the five classifiers together as shown in the last column. For example, 100 instances were considered as faulty out of 177 in Ant when Shatnawi thresholds are used whereas only 28 out of 64 when Alves thresholds are used. The rest of instances that are not predicted as faulty are then considered as non-faulty again as were originally reported in the datasets. This process guarantees that the imbalance in data is reduced after using threshold values with confirmation by five machine learners.

OPEN IN VIEWER

OPEN IN VIEWER

To answer RQ2, we notice that the number of instances selected to be added to the minority group (faulty) is increased for all systems using the three threshold techniques as shown in Table 8. Table 8 shows the percentages of faulty instances after using thresholds and their validation using the five machine learning techniques. Therefore, we can conclude that threshold values help reduce data imbalance. To summarize this finding we have reported the boxplots (Figs 2–4) that show the percentages of classes that are selected as fault-prone using thresholds. The use of Shatnawi and Vale thresholds shows that classifiers have selected about 50% of classes as indeed faultprone using the single classifiers as reported in Figs 2 and 4. However, the five classifiers together have selected about 38% of classes as fault-prone. Therefore, we can conclude that RQ2 is answered by the experiment. Threshold values can be used to reduce imbalance in fault data as shown clearly in Table 8. For all systems, Shatnawi’s thresholds reduced data imbalance more than Vales’ and Aleves’ thresholds.

OPEN IN VIEWER

Datasets are rebuilt according to the classification of the unlabeled instances. The prediction models are then trained and tested using the 10-fold cross validation on the updated datasets. We report the results of each classifier to compare the updated datasets with the original datasets. The results of the five classifiers are shown in the Tables 9–13. The MCC values of each classifier are shown for the three threshold techniques.

The results of classifiers’ performance on the original data are shown in the second column in the Tables 9–13. The classifiers’ performance on the original data (i.e., without the use of thresholds) is weak on all classifiers, i.e. a small MCC value that is close to zero means a random correlation between the actual and the predicted values. The results for the classifiers after using thresholds are also shown in the tables.

To answer RQ3, we need a more formal analysis to find the significance of the differences between two models, the traditional and the modified models. We conduct a statistical comparison using a non-parametric test, cliff’s δ test [33]. Cliff’s δ measures the magnitude of difference between two groups [38]. Cliff’s δ in this research estimates the likelihood of how often a predictor is becoming better after using thresholds to reduce data imbalance. Song et al have used the following scales: trivial ( 0.147), small (0.147 0.33), moderate (0.33 0.474) or large ( $\delta >$ 0.474) [38]. We compare the differences in MCC values between the use of each threshold technique and the traditional machine learners. The results of Cliff’s δ analysis are shown in Table 14. We conclude that the use of thresholds has a large effect on improving the traditional fault-prediction models, i.e., all values are large (i.e., $\delta >$ 0.474). Therefore, this experiment answers RQ3 and provides evidence of using thresholds to improve fault prediction in imbalanced data.

OPEN IN VIEWER

For comparison purposes, the boxplots for all the classifiers’ performance (MCC values) are reported in Fig. 5. The boxplots show the MCC values for four datasets: original, and modified using Alves, Shatnawi and Vale techniques. The boxplots in Fig. 5 show better prediction performance after using thresholds than otherwise. The performance of using classifiers with Shatnawi and Vale thresholds are better than with Alves thresholds.

For the five classifiers, we observe similar patterns, i.e., the original data has the least performance whereas using Shatnawi thresholds produces the best performance, i.e., the median value is the largest. The MCC values are larger than random in models when using thresholds. These results show reliable prediction models. Generally speaking, using thresholds improves fault prediction performance. The improved classifiers have shown a great improvement on average when compared with models on original data. Table 15 shows the percentage of improvements on the average values of the performance for each model. For example, the performance of models with Shatnawi’s thresholds shows the most improvements for J48 (358%). All thresholds have shown improvements on average. The Alves’ thresholds are showing the least improvement, however, there improvements are significantly large.

Threshold values shown in Table 1 can be ordered by the number of instances that are selected for inclusion in faulty classes, i.e., Alves, Vale, and Shatnawi. Shatnawi’s thresholds select more instances for further investigation. Threshold values can be used separately to mark classes as fault-prone or can be used in combination with fault prediction models to improve the prediction performance. This is the reason why Shatnawi’s thresholds have better improvement on the modified datasets than the other threshold techniques. As noticed in Tables 5–7, using thresholds has selected a large number of instances as faulty and we found at least 50% of these instances were classified as such.

Using classifiers such as naïve Bayes, nearest neighbors and JRip show better results than LR and C4.5. However, LR and C4.5 performances were significantly larger than random (MCC $>$ 0). The NB classifiers with Alves’ thresholds show large MCC values between 0.82 and 1. The KNN classifiers with Vale thresholds show large MCC values between 0.75 and 1. The JRip classifiers show values between 0.68 and 0.99. These results suggest that thresholds are reliable to find faulty classes for at least five well-known classifiers. We can conclude that thresholds are reliable in classifying classes as either faulty or non-faulty and the use of thresholds has a consistent effect on different classifiers as shown in the boxplots in Fig. 5.