Improved Biological Network Reconstruction Using Graph Laplacian Regularization

Abstract

Biological networks reconstruction is a crucial step towards the functional characterization and elucidation of living cells. Computational methods for inferring the structure of these networks are of paramount importance since they provide valuable information regarding organization and behavior of the cell at a system level and also enable careful design of wet-lab experiments. Despite many recent advances, according to the scientific literature, there is room for improvements from both the efficiency and the accuracy point of view in link prediction algorithms. In this article, we propose a new method for the inference of biological networks that makes use of a notion of similarity between graph vertices within the framework of graph regularization for ranking the links to be predicted. The proposed approach results in more accurate classification rates in a wide range of experiments, while the computational complexity is reduced by two orders of magnitude with respect to many current state-of-the-art algorithms.

1. Introduction

Reconstruction of biological networks (e.g., metabolic, protein-protein, and regulatory networks) from heterogeneous -omic data sources is a cornerstone of computational systems biology. This inference process is aimed at understanding how cellular components interact with each other in order to shed light on the underlying mechanisms that regulate the functioning of the cell. Moreover, accurately predicting whether there is an interaction between pairs of molecules allows careful designing of subsequent (time- and resource-costly) experiments. For instance, prioritization of candidate pairs determines which entities (e.g., proteins) could putatively interact with higher probability, hence suggesting the priority of possible further experimental screenings.

The network reconstruction problem is commonly recast as a pattern recognition problem on graphs whose nodes represent a (sub)set of molecules (e.g., proteins, genes, enzymes) of a given organism and whose edges (either directed or undirected) are representative of particular biological properties of the system (e.g., protein physical interaction, gene regulation, reaction catalysis). The reconstruction problem, therefore, can be reduced to the inference of interactions among vertices of these graphs, given a partial (and often affected by experimental errors) knowledge of the network. The whole process could in principle benefit from the inclusion of heterogeneous biological information from a wide spectrum of data sources: genomic sequence similarity, gene expression levels, and cellular localization (as well as other features) can be integrated into the network under study, with the aim of getting more accurate predictions (Bleakley et al., 2007; Yip and Gerstein, 2009; Kashima et al., 2009b).

1.1. Related work

In the rich literature concerning network inference, a major line of research has been devoted to the development of machine learning supervised or semi-supervised methods. Supervised learning (SL) is a pattern recognition technique that aims at learning a classification function from the labeled (known) data given as input to the classifier during the training phase. Semi-supervised learning (SSL) is a novel learning paradigm that takes into account not only labeled data, but also unlabeled data, during the learning phase. The aim is to complement the information given as input to the classification algorithms with the reciprocal relationships of the whole input data in order to the enrich the learning process. This framework is particularly useful if labels are scarce and difficult to obtain. Its usefulness and effectiveness has been demonstrated in a wide and heterogeneous number of experimental settings (Chapelle et al., 2006).

Among the class of SL algorithms applied to the problem of biological network inference, many approaches have exploited support vector machines (which are considered state of the art statistical learning algorithms). For instance, Ben-Hur and Noble (2005) proposed to frame the network reconstruction problem into a binary classification problem by taking protein pairs as input of the classifier algorithm and by using ad hoc developed kernels named pairwise kernels (P-SVM) for representing similarities between proteins. Bleakley et al. (2007) proposed a divide and conquer strategy to reduce space and time complexity with respect to SVM-based algorithms (i.e., learning subnetworks around each given vertex of the graph). This approach has been used for inferring metabolic (Bleakley et al., 2007) and regulatory networks (Mordelet and Vert, 2008).

For the semi-supervised approach to the problem, we recall here the work of Yip and Gerstein (2009), which refined the local model approach by expansion of the training set of each vertex. The basic assumption the authors made is that accurate predictions can be used to widen the training set in order to ensure a robust set of examples to the classifier algorithm.

Another SSL algorithm that has been recently proposed is a link mining method named LinkPropagation (Kashima et al., 2009a,b), which exploits the so-called label propagation method (Zhu et al., 2003; Chapelle et al., 2006) to predict the existence of a given link in a graph, given the partial knowledge of the network and the similarities among the nodes. Informally, label propagation consists of an application of the guilty by association principle: two vertices of a graph that are similar (according to a given measure) are considered to belong to the same class (e.g., functional category). Hence, if the class label for one of the two nodes is known, the label can be propagated and the annotation transferred.

Although many approaches have been proposed during the last years, many challenging questions are still considered to be open. Mainly, the inherent noisy nature of datasets and their size prompt the need for both accurate and efficient algorithms design. On the one hand, data provided as input to learning algorithms is incomplete and noisy (false positive rates for yeast, worm, and fly data are estimated to range from 25% to 45% while false negatives range from 75% to 90%, according to Huang et al., 2007), hence justifying the search for noise-resilient approaches. On the other hand, high-throughput techniques produce data at increasing rates, thus making of primary importance the development of efficient solutions.

In this work, we investigate a new approach for the network reconstruction problem based on regularization theory on graphs via the Laplace operator. In particular, we aim at taking advantage of the so-called Regularized Laplacian (Smola and Kondor, 2003).

The Regularized Laplacian (hereafter also denoted as RL) is a kernel matrix with several interesting theoretical properties (Smola and Kondor, 2003), whose usefulness has been demonstrated in graph mining when applied to citations of research articles (Shimbo et al., 2007). Interestingly, this type of regularization on graphs is intertwined with a graph-based similarity measure known as Matrix-Forest similarity metrics (hereafter also denoted as MFSM). MFSM has been proposed as an alternative way of measuring the affinity between objects represented as nodes of a graph (Chebotarev and Shamis, 1997, 1998). It integrates indirect paths between vertices of the network and is related to the Laplacian of the adjacency matrix of the underlying graph, from which it can be computed (Fouss et al., 2007). The concept of MFSM has been recently exploited in collaborative recommendation systems, with interesting results in terms of accuracy of the retrieval system under study (Fouss et al., 2007).

By means of the regularization operator, we aim at deriving a proximity measure between each pair nodes of the graphs under study. This type of measure can be naturally used as the score to rank links to be inferred. The rationale behind our approach is to exploit the inherent capability of the regularization approach of handling noisy and missing information, in order to correctly recover the structure of the underlying networks. The proposed method (hereafter denoted as RL-BNI, Regularized Laplacian Biological Network Inference1) has been tested on the same datasets used by Kashima et al. (2009b) for cross-validating their algorithm (i.e., the metabolic networks of Caenorhabditis elegans, Helicobacter pylori, and Saccaromyces cerevisiae). The experimental results show the capability of the proposed method to reconstruct the correct structure of these networks under different experimental conditions (i.e., different degree of knowledge of the network under study). A remarkable feature of our approach is its computational complexity when compared with current state-of-the-art algorithms. In fact, RL-BNI needs much fewer computational resources (both in terms of CPU time and memory usage) with respect to other supervised and semi-supervised learning algorithms (e.g., P-SVM and LinkPropagation) while retaining the same (or even a greater) accuracy.

This article is organized as follows: in Section 2.1, we describe our approach to the problem, while in Section 3, we introduce the experimental protocol we used for cross-validation. Then, in Section 4, we present and discuss experimental results, and in Section 5, we conclude with some final considerations and sketch some possible future research directions.

2. Methods

2.1. The proposed approach

In order to detail the proposed algorithm, we first recall some preliminary definitions. Let A be an m × m adjacency matrix of a given graph \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland,xspace}\usepackage{amsmath,amsxtra}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document}$${ \cal G}$$\end{document} . We define D as an m × m diagonal matrix with D_ii = ∑_jA_ij. The Laplacian L and the normalized Laplacian \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland,xspace}\usepackage{amsmath,amsxtra}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document}$$\tilde{ \bf L}$$\end{document} of \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland,xspace}\usepackage{amsmath,amsxtra}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document}$${ \cal G}$$\end{document} are, respectively, defined as L = D − A and \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland,xspace}\usepackage{amsmath,amsxtra}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document}$$\tilde {\bf L} = {\bf D} ^ {- \frac {1} {2}} {\bf LD} ^ {- \frac {1} {2}}$$\end{document} . The Regularized-Laplacian (RL) kernel matrix K_rl of \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland,xspace}\usepackage{amsmath,amsxtra}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document}$${ \cal G}$$\end{document} can be computed, according to Smola and Kondor (2003), as K_rl = (I + σ²L)⁻¹, where σ ≥ 0 is a real-value parameter to be properly chosen. Of course, the choice of σ is crucial for the inference accuracy. It is worthwhile to mention that this issue must be addressed when other kernel methods are used also. In order to assess the impact of σ on the accuracy of the network reconstruction, we sampled the parameter space assigning different values to σ, as detailed in Section 3.2. Interestingly, in our experiments, the best performance is almost always achieved with σ = 1. Other methods/kernels could be proposed for the computation of a proximity measure between nodes of a graph. While an exhaustive comparison between these methods is beyond the scope of this work, we compared the proposed approach with another reconstruction method based on a widely used kernel, i.e. the diffusion kernel (Kondor and Lafferty, 2002; Smola and Kondor, 2003). Moreover, as described in Section 3.1, we tested the capability of RL-BNI by comparing it with a random walk method and with LinkPropagation (considered a state-of-the-art method). The experimental results provide evidence of the validity of the Regularized Laplacian approach, which resulted into higher precision levels in the reconstruction tasks, outperforming the competing methods in a wide range of experiments. We also notice here that, since the MFSM matrix of a graph \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland,xspace}\usepackage{amsmath,amsxtra}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document}$${\cal G}$$\end{document} (let it be denoted A_mf) can be computed as A_mf = (I + L)⁻¹ (Fouss et al., 2007), the choice of σ = 1 implies the coincidence of RL with MFSM, intriguingly bridging a gap between the theory of regularization and graph topological analysis (which could be of independent interest).

With the help of the above introduced definitions, we may now sketch the RL-BNI algorithm. We are given in input an m × m matrix A which represents the network to be reconstructed. In particular, A(i, j) = 1 if a link exists between nodes i and j; A(i, j) = −1 if a link doesn't exist between i and j; and A(i, j) = 0 if the link status is unknown. We first pre-process A by setting: \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland,xspace}\usepackage{amsmath,amsxtra}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*}A (i , j) = \begin{cases} \mid E \mid / \mid E^ + \mid \qquad \hbox{if a link exists between} \ i \ {\rm and} \ j \\ - \mid E \mid / \mid E^ - \mid \ \hbox{if a link doesn't exist between} \ i \ {\rm and} \ j\end{cases}\end{align*} \end{document}

where ∣E∣ is the number of training examples (i.e., known edges), ∣E⁺∣ is the number of positive training examples, and ∣E⁻∣ is the number of negative training examples. This pre-processing allows us to set target values according to Fisher discriminant following Kashima et al. (2009a). A is then properly normalized in order to have all entries in the [0, 1] interval, and its normalized laplacian \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland,xspace}\usepackage{amsmath,amsxtra}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document}$$\tilde{\bf L}_A$$\end{document} is computed. The last step of RL-BNI entails the computation of Regularized Laplacian kernel A_rl as specified by equation 1: \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland,xspace}\usepackage{amsmath,amsxtra}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*}{\bf A}_{rl} = ({\bf I} + \sigma^2 \tilde{\bf L}_A) ^{- 1} \tag{1}\end{align*} \end{document}

A_rl can be used to rank the links to be predicted: the higher the value of A_rl(i, j), the higher the confidence that a link exists between nodes i and j.

Notwithstanding its simplicity, the performance of RL-BNI is interestingly effective, as we demonstrate below.

2.2. Computational complexity

The computation of the normalized Laplacian has time complexity O(m²), while the computation of the Regularized Laplacian kernel is O(m³) (since it entails the inversion of a m × m matrix). Hence, the resulting time complexity of RL-BNI is O(m³)

Regarding the space complexity, the computation of the normalized Laplacian costs O(m²). Since computing the RL-kernel has memory requirement O(m²), we can state that the overall space complexity of the proposed approach is O(m²).

Table 1 summarizes the time and space asymptotic complexities of RL-BNI compared to the time and space complexities of LinkPropagation and P-SVM (which are considered two state-of-the-art algorithms). The time complexity of RL-BNI is two orders of magnitude lower than the complexity of LinkPropagation and three orders of magnitude lower than the complexity of P-SVM, while the memory requirement is the same for RL-BNI and LinkPropagation (i.e., two orders of magnitude lower than the space complexity required by P-SVM).

Table 1.

Time and Space Complexities for LinkPropagation (LP), P-SVM, and RL-BNI

Algorithm	Time complexity	Space complexity
LP	O (m⁵)	O (m²)
P-SVM	O (m⁶)	O (m⁴)
RL-BNI	O (m³)	O (m²)

3. Experimental Protocol And Data

3.1. Competing methods

In order to carry out a comprehensive evaluation of our method, we compared it with a state-of-the-art semi-supervised algorithm, namely LinkPropagation (Kashima et al., 2009b), with a random-walk with restart algorithm (RWR) (Tong et al., 2006; Gallagher et al., 2008) and with a diffusion-based kernel (DK) (Kondor and Lafferty, 2002; Smola and Kondor, 2003).

3.1.1. LinkPropagation

The LinkPropagation algorithm extends the guilty-by-association principle to the problem of link mining by considering pairs of vertices instead of isolated nodes. Within this setting, two pairs of nodes can be considered to have similar link weights if they are (enough) similar to each other. This latter method has been renamed, by analogy, link propagation principle. The main feature of LinkPropagation is its capability of handling multiple associations among nodes, hence allowing one to embed in the link strength the information regarding two node pairs in two different networks. LinkPropagation has been used for simultaneously predicting the networks of multiple species (Kashima et al., 2009b).

3.1.2. Random walk with restart

We also chose to implement, for comparison purpose, another link prediction method that we selected as representative of pure topology-based algorithms. In particular, we opted for a random-walk with restart algorithm. Approaches that make use of random-walk similarity among nodes of a graph are routinely used in many information retrieval, machine learning, and computational biology applications (Page et al., 1998; Gallagher et al., 2008; Kohler et al., 2008; Morrison et al., 2005), hence making it a natural choice for network analysis.

The RWR algorithm aims at deriving the proximity between two nodes i and j by simulating a random walk process: the random walker starts from node i and, at each iteration, moves to one of its neighbors with probability proportional to the weight of the arc that join them (Gallagher et al., 2008; Tong et al., 2006; Morrison et al., 2005). At each iteration, the walker has also a probability equal to (1 − d) of being teleported back to starting node i (where 0 ≤ d ≤ 1 is a parameter known as restart probability or damping factor). The proximity score between nodes i and j is then defined as the steady-state probability that the random walker will finish its walk at node j. Given a graph \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland,xspace}\usepackage{amsmath,amsxtra}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document}$${\cal G}$$\end{document} , whose adjacency matrix is A and given a diagonal matrix D (D_ii = ∑_jA_ij), this score, which can be demonstrated to be an invariant measure, can be computed according to equation 2 (Gallagher et al., 2008; Tong et al., 2006): \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland,xspace}\usepackage{amsmath,amsxtra}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*}{\bf A}_{rwr} = ({\bf D} - d{\bf A} ^T) ^{- 1} \tag{2}\end{align*} \end{document}

Matrix A_rwr can be used to rank the links to be predicted: as for RL-BNI, the higher the value of A_rwr(i, j), the higher the confidence that a links exists between nodes i and j.

3.1.3. Diffusion kernel

The diffusion kernel has been introduced by Kondor and Lafferty (2002) and has been demonstrated to be a useful method for representing the structure of pairwise similarities among points of a data space in many applications. In this context, we make use of the regularization framework that includes DK as special case, as defined by Smola and Kondor (2003). In particular, given a graph with adjacency matrix A, whose normalized Laplacian is \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland,xspace}\usepackage{amsmath,amsxtra}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document}$$\tilde{\bf L}$$\end{document} , we may compute the diffusion kernel matrix as follows: \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland,xspace}\usepackage{amsmath,amsxtra}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*}{\bf A}_{dk} = exp (- \sigma^2 / 2 \tilde{\bf L}) \tag{3}\end{align*} \end{document}

This kernel has an interesting interpretation in terms of diffusion processes: if we hypothetically inject a given quantity at node i and let it diffuse through the graph, the result of this quantity flow measured at node j (at steady state) is A_dk(i, j). Once again, we may use A_dk(i, j) as affinity measure to predict the existence of links.

3.2. Data and experimental settings

To evaluate our approach, we adopted the same experimental setting used by Kashima et al. (2009b) to validate LinkPropagation. In particular, we used the metabolic networks of C. elegans, H. pylori, and S. cerevisiae as gold standard. The nodes of each network are enzymes. Links connect two nodes if the pair of enzymes associated to the nodes catalyze successive reactions in a given metabolic pathway (Kanehisa et al., 2008; Kashima et al., 2009b). Table 2 summarizes the number of nodes and edges for each of the investigated networks. We also used the same cross-species similarities (i.e., the normalized Smith-Waterman scores of pairwise local protein alignments) and the intra-species similarities (i.e., the gene expression data) used by Kashima et al. (2009b) to construct the nine similarity matrices used as input for inference by LinkPropagation.2

Table 2.

Metabolic Networks Used in Experiments

	C. elegans	H. pylori	S. cerevisiae
No. of vertices	532	291	722
No. of edges	2892	492	2323

We compared the proposed method with the LinkPropagation algorithm, with the random-walk based algorithm and with a diffusion kernel algorithm, to evaluate its predictive accuracy. For RWR, we tried different values of the restart probability (d). Results refer to d = 0.85, which provided the higher accuracy levels. For DK and RL-BNI, we tested different values to be assigned to parameter σ, taken, respectively, from the set {0.01, 0.1, 0.5, 1, 5, 10, 20, 30, 50, 80, 10}. The metabolic networks have been randomly sampled to obtain training and test data. We chose three different ratios of the training set: particularly, the 25%, 50%, and 75% of all the node pairs have been used as training data. The results have been evaluated by taking into consideration two types of metrics: the AUC (Area under the ROC curve) (Gribskov and Robinson, 1996) and the AUF (Area under the FDR curve) (Bleakley et al., 2007). The AUC summarizes the ROC curve which plots the true positives as a function of false positives when a threshold used for predicting interactions from the ranking scores is varied. The AUF summarizes the FDR curves which represents the ratio of false positives among all positive predictions. The AUC is a widely adopted measure to evaluate classification algorithms (in fact it has been used as a metric to benchmark LinkPropagation), but we decided also to take into account the AUF since biological network reconstruction is a type of problem that presents many more negative than positive examples. This prompts the need for a metric (such as the FDR) able to capture the fact that, among the first-ranked predictions, there should be enough true positives (Bleakley et al., 2007). We also recall here that a perfect classifier would present AUC equal to 1 and AUF equal to 0, while a random classifier would obtain an AUC of 0.5 and an AUF equal to the ratio of not connected nodes between all pairs (which is very close to 1). Motivated by the above considerations we run, for each ratio of training set, five experiment trials and we computed average values for either AUC and AUF.

4. Results And Discussion

The results are reported in Tables 3 –5 (for the AUC metrics) and in Tables 6 –8 (for the AUF metrics). Each table reports, for any of the three metabolic networks, the AUC and the AUF of LinkPropagation with individual (LP^ind) and simultaneous (LP^ind) inference, RWR, DK, and RL-BNI. For DK and RL-BNI, we decided to report, for the sake of conciseness, only the first two best results corresponding to given σ values. In particular, we found that the best accuracies are achieved by DK with σ = 5 and σ = 10, and by RL-BNI with σ = 1 and σ = 5. Results are reported for different values of the training set ratio: 25% for Tables 3 and 6, 50% for Tables 4 and 7, and 75% for Tables 5 and 8.

Table 3.

AUC Results: Average and Standard Deviation AUC Values for LinkPropagation With Individual (LP^ind) and Simultaneous (LP^ind) Inference, Random Walk Algorithm (RWR), Diffusion Kernel (DK, σ = 5, 10) and Regularized Laplacian Reconstruction Method (RL-BNI, σ = 5, 1). Ratio of Training Data: 25%

	C. elegans	H. pylori	S. cerevisiae	Total
LP^ind	0.702 ± 0.004	0.600 ± 0.007	0.749 ± 0.005	0.701 ± 0.005
LP^sim	0.747 ± 0.005	0.616 ± 0.007	0.782 ± 0.006	0.740 ± 0.004
RWR	0.623 ± 0.015	0.521 ± 0.023	0.585 ± 0.011	0.496 ± 0.005
DK^σ = 5	0.747 ± 0.003	0.632 ± 0.009	0.731 ± 0.004	0.712 ± 0.002
DK^σ = 10	0.745 ± 0.003	0.630 ± 0.010	0.727 ± 0.004	0.710 ± 0.002
RL-BNI^σ = 5	0.774 ± 0.004	0.640 ± 0.009	0.753 ± 0.003	0.733 ± 0.002
RL-BNI^σ = 1	0.818 ± 0.006	0.578 ± 0.025	0.740 ± 0.006	0.726 ± 0.003