In this paper, we propose CryptMed, a system framework that enables medical service providers to offer secure, lightweight, and accurate medical diagnostic service to their customers via an execution of neural network inference in the ciphertext domain. CryptMed ensures the privacy of both parties with cryptographic guarantees. Our technical contributions include: 1) presenting a secret sharing based inference protocol that can well cope with the commonly-used linear and non-linear NN layers; 2) devising optimized secure comparison function that can efficiently support comparison-based activation functions in NN architectures; 3) constructing a suite of secure smooth functions built on precise approximation approaches for accurate medical diagnoses. We evaluate CryptMed on 6 neural network architectures across a wide range of non-linear activation functions over two benchmark and four real-world medical datasets. We comprehensively compare our system with prior art in terms of end-to-end service workload and prediction accuracy. Our empirical results demonstrate that CryptMed achieves up to respectively , , and bandwidth savings for MNIST, CIFAR-10, and medical applications compared with prior art. For the smooth activation based inference, the best choice of our proposed approximations preserve the precision of original functions, with less than 1.2% accuracy loss and could enhance the precision due to the newly introduced activation function family.
Recent thriving deep learning techniques have been fueling a wide spectrum of medical endeavors, ranging from radiotherapy [27], clinical trial and research [60], to medical imaging diagnostics [55]. Enterprises capitalize on neural networks (NNs) to offer medical diagnostic services, facilitating hospitals and researchers to produce faster and more accurate decisions over their medical data. With the growth in such offerings comes rapidly growing awareness of daunting privacy concerns. The medical data is of sensitive nature and must be always kept confidential [1,23,49,50]. Meanwhile, NN models used in these services are seen as lucrative intellectual properties and encode knowledge of private training data [24].
Addressing these privacy concerns in the above deep learning powered service scenario generally fits within paradigm of secure multi-party computation (MPC). A rich body of work [25,37,43,47,56] proposes hand-tuning MPC protocols for secure inference, whereby the service provider and the customer interact to produce an inference over encrypted/secret-shared NN model and individual data. We note that prior designs are still facing obstacles regarding inference efficiency and accuracy in the ciphertext domain, and do not appear to be able to fulfill practical requirements of real-world medical diagnostic scenarios.
Firstly, they all require customers to conduct heavy cryptographic computations like homomorphic encryption (HE) and garbled circuits (GC), imposing intensive computational and communication overheads during inference. These overheads are further exacerbated when, e.g., the service is deployed to a hospital with resource-constrained devices (like portable medical imaging scanners [34]).
Secondly, existing protocols are either not compatible with non-linear (activation) functions [25,37] or only focus on the simple ReLU function [43,56,65], causing limitations of applicability for medical diagnoses. There are limited works investigating other essential (smooth) activation functions, like sigmoid. Among them, most of existing designs use high-degree polynomials [4,40] or ad-hoc piecewise polynomials [47,57,58,61] to approximate those activation functions. However, the former approach incurs heavy costs when evaluating repeated multiplications. The latter one relies on intervention and expertise on models and training datasets for fine-tuning [47], or encounters the severe ‘vanishing gradient problem’ making the NNs imprecise [57,58,61].
To address the above challenges, we design, implement, and evaluate CryptMed, a lightweight and secure NN inference system tailored for medical diagnostic services. CryptMed proceeds by having the hospital and the medical service engage in a tailored secure inference protocol over their secret-shared inputs. Only the hospital learns the diagnostic result; and the privacy of the medical data and model is ensured against each other. In particular, we combine insights from cryptography, digital circuit design, and deep learning literature, fostering an efficient, low-interaction, and accurate deep learning service suitable for realistic medical scenarios. Our contributions are summarized as follows.
We propose a new secure NN inference system framework CryptMed relying only on the lightweight secret sharing techniques, which requires neither heavy cryptographic computation nor large-size ciphertext transmission.
We present a hybrid protocol design that consists of a preprocessing phase and an online phase where the preprocessing phase conducts as much computation as possible to ease the online phase. Moreover, the preprocessing only involves lightweight computation in the secret sharing domain.
We devise an efficient and communication-optimized secure comparison function harnessing the insights from cryptography and the field of digital circuit design. Our proposed secure comparison function can efficiently support the widely adopted comparison-based non-linear functions, including ReLU, ReLU6, Leaky ReLU, Binary activation, and MaxPool/MinPool. Compared to the commonly-used GC solutions, CryptMed’s secure ReLU is faster and requires less communication, and the secure MaxPool is faster and uses less communication.
We devise secure smooth activation functions (i.e., tanh, sigmoid, ELU) from newly proposed precise and cryptography-friendly approximations in the field of digital circuit design and deep learning literature. Our introduced approximations are non-linear and low-degree piecewise polynomial approximations with quantitative performance and demonstrate promising accuracy through comprehensive empirical. They are not only approximations but also new activation function family that are natural replacements of the smooth functions. With such approximations, CryptMed reformulates the challenging support for secure smooth activation functions into the comparison-based construction that can be efficiently and accurately evaluated in secure domain.
We conduct formal security analysis. We implement a prototype of CryptMed. We extensively train varying neural network models based on 6 architectures across a wide range of non-linear activation functions. We conduct comprehensive experiments over two benchmarking datasets and four real-world medical datasets. Our experiment results show that CryptMed requires the least network resources compared to prior works with up to , and bandwidth savings for MNIST, CIFAR-10, and the medical applications, respectively.
The rest of this paper is organized as follows. Section 2 investigates the related work. Section 3 introduces the necessary preliminaries. After an overview of our system in Section 4, we present our proposed construction in Section 5. Section 6 presents our empirical evaluation on microbenchmarks and end-to-end secure inference service. Finally, Section 7 concludes this paper.
Related works
Secure neural network inference. Secure neural network inference has drawn much attention in the emerging field of privacy-preserving machine learning. Our design is closely related to a line of studies on secure NN inference. These studies [12,25,37,43,47,56,81,83] mostly propose an interactive protocol for secure inference running between the service provider and the customer. Among others, there are designs [48,58,65] that consider an outsourced scenario, where two non-colluding cloud servers collaboratively perform NN inference over the encrypted/secret-shared model and data. Apart from different system models, they commonly rely on heavy cryptographic techniques (like HE and GC) during the latency-sensitive online inference procedure. Very recently, Delphi [56] proposes a hybrid and interactive inference protocol, which preprocesses some cryptographic operations to accelerate the online inference execution. However, this work still demands intensive workloads on the customer to conduct heavy cryptographic computations during preprocessing, and relies on expensive GC based approach to evaluate the basic ReLU function. CryptMed adopts a similar hybrid setting yet only involves the lightweight secret sharing techniques during the entire secure inference procedure, which has an prominent advantage of rather simplified implementation for easy real-world deployment, compared to the SOTA which requires heavy optimization in GC and homomorphic encryption implementation.
We emphasize that most prior works only support the basic ReLU activation [43,56,65]. Other essential activation functions commonly-used in deep learning based medical diagnoses are unexplored, such as ReLU6, Leaky ReLU, and the exponential linear unit (ELU) [15]. Even worse, some works can not fully cope with the non-linearity [25,37]. Instead, they use the square function for approximation, resulting in an imprecise prediction [42,52] that could cause impactful consequences in the medical diagnostic applications.
In the literature, only limited works explore the smooth sigmoid activation function via polynomial approximations. These works either resort to high-degree polynomials [4,40] or ad-hoc piecewise polynomials [47,57,58,61]. The first approach suffers from substantial costs to evaluate a large amount of secure multiplications. The second approach heavily relies on intervention from developers to fine-tune the piecewise polynomials (coefficients and segments) [47], or runs into the severe vanishing gradient problem making the NNs imprecise [57,58,61]. All those solutions do not appear to be competent for practical deep learning based medical diagnoses deployment. A detailed comparison between our work and prior works is summarised in Table 1.
Secure machine learning MPC frameworks. A number of generic MPC frameworks are designed and implemented for complicated computation tasks like machine learning. Noteworthy examples include the two-party framework with a trusted party to generate correlated randomness proposed in Chameleon [65], the framework proposed by Reza Sadeghi et al. [68], TAPAS [70], FHE DiNN [11], the framework proposed by Dalskov et al. [19], MP2ML [9]; the three-party frameworks proposed in ABY3 [57], SecureNN [75], CryptTFlow [41]; the four-party frameworks in Trident [62], FLASH [13]; and multi-party frameworks in TFEncrypted [18], PySyft [67,89], FALCON [76]. Note that a handful of latest privacy-preserving machine learning systems opt for specialized and optimized designs rather than direct application of generic MPC frameworks [2,56,64,84] for performance consideration. Our design also follows such trend.
Limitations of prior secure neural network inference systems and comparison with our systems
1These systems requires NN model architecture modification, such as polynomial approximation of ReLU activation. This setting could downgrade the prediction accuracy.
2These systems are designed for quantized NNs. Quotient [2] uses Ternarized NN, and XONN [64] is designed for Binarized NN.
Privacy-preserving medical diagnosis. This work also relates to the designs of privacy-preserving medical diagnosis. There is a plethora of work proposed on privacy-preserving medical imaging based diagnostic applications. Some works strive to enhance the reliability of image-centric diagnoses via privacy-preserving image denoising [32,85,86,88]. These works resort to DNNs [86,88] or reference image patches [32,85] to devise image denoising protocols that can privately reduce the noises and deliver high-quality medical image content. Meanwhile, a line of work aims to explore privacy-preserving machine learning for various medical diagnostics, like medical image classification [3,38,39,48,51], tumor segmentation [44,45,54,71], and genomic data regression [10,21,59]. Some of them utilize cryptographic privacy-enhancing techniques (e.g., homomorphic encryption, secure multiparty computation) to protect the privacy of machine learning models and medical data [3,10,21,48,51,59]. Others resort to differential privacy techniques to train a private model that can be used to conduct private inference over medical images [38,39,44,45,54,71]. We emphasize that these works focus on different problems and their designs are highly different from ours. A few of them consider secure neural network inference based medical diagnosis, yet requiring to use heavy cryptography [3,48] or not focusing on supporting smooth activation functions [51]. Many others explore different problems, like the relatively simpler regression models [10,21,59], or federated learning [38,39,44,45,54,71]. Besides, those works relying on differential privacy techniques require to perturb the data with noise, and thus would downgrade the utility of models [38,39,44,45,54,71].
Apart from the machine learning based approaches, works adopt data mining techniques to securely analyze the medical data for diagnostics. Applications include similarity analysis of human genome sequences [6,69,78], genome-wide association studies [16,35,36,72], biometric identification [33], pharmacology and medicine [14,29], and medical time-series data analytics [7,49,50]. A common paradigm is to customize secure computation protocols to meet certain requirements for different medical diagnostic applications.
Differences from the conference version. Portions of this paper have been presented in [51]. We have revised the preliminary work [51] with substantial new contributions and improvements, as summarized below. Firstly, we have proposed a number of new efficient, lightweight, and accurate secure non-linear activation functions in Section 5, including the secure comparison-based activation functions ReLU6, Leaky ReLU, Binary activation, and the secure smooth activation functions tanh, sigmoid, and ELU with different approximation approaches. Secondly, we have made a full-fledged implementation of the new realizations of our security design and conducted comprehensive performance evaluation and comparisons. The overall experiment results have demonstrated the prominent performance advantage of our new design. Finally, we have refined the previous work significantly to reflect our new contributions and latest understanding on the topic, as well as improve the clarity.
Table of notation
Notation
Description
X, x
Input/feature tensor and element.
W, w
Weight tensor and element.
μ, δ, γ, β
Batch normalization parameters: the running mean, running variance, scale, and shift.
L
Number of neural network layers
ℓ, n
Bit length ℓ and vector length n.
i
Identifier of a party i, where .
,
Control bits
The most significant bit of element x
,
The k-th element of vector x; The k-th vector.
Additive secret shares of value x in ring held by the party i
Boolean shares of value x in ring held by the party i
Addition/subtraction over additive secret shares
Multiplication over additive secret shares
Bitwise XOR over Boolean shares
Bitwise AND over Boolean shares
Preliminaries
In this section, we introduce the core primitives and background used in CryptMed. We summarize the key notations used in this paper in Table 2.
Secret sharing. We now present the key cryptographic primitive used in our design: additive secret sharing. Additive secret sharing [5] protects an ℓ-bit value as two secret shares and such that , where is a ring and r is a random value from (). It perfectly hides x as each share is a random value and reveals no information of x. Given two parties and , each party holds corresponding shares of two secret values x and y. Additive secret sharing supports efficient local addition and subtraction over shares and scalar multiplication (η is a public value). They are calculated by each party () locally without interaction. Multiplication over two shares is realized by the secret-shared Beaver’s triple [8], i.e., holds in a way that . Such a multiplication operation with Beaver’s triple is a standard secure computation protocol, whereby obtains the shares of at the end. Note that Beaver’s triples are data independent and can be efficiently generated via one-off computation by a third party [65,87]. In addition, additive secret shares can readily support boolean operations over binary values. Given the bit length and the ring , a secret binary value x is shared as and . The bitwise XOR (⊕) and AND (∧) over shares are calculated in the same way as the above addition and multiplication, respectively.
Deep neural networks. A typical Deep Neural Network (DNN) comprises two types of layers in sequence: linear layers and non-linear layers. Linear layers include convolutional layers (CONV), fully-connected layers (FC), batch normalization (BN) layers, and average pooling layers (AvgPool). The functionalities of these layers in cleartext can be formulated as a bunch of additions, multiplications, and flattened operations over kernels (partial model weights for a certain function) and features (user inputs and intermediate results). Specifically, the underlying functionality of CONV and FC is the vector-wise dot product between a kernel vector and a feature vector within a sliding window . Given a kernel , CONV transforms an input feature into an output feature via
where , . That is, any data point in Y is produced by applying a sliding kernel tensor over the entire feature tensor X, and performing cross-channel operations repeatedly. Such a function indicates the FC layer when . Batch normalization is used to regularize the model. It is applied after CONV/FC layers and performs over the feature x on each neuron, where μ, δ, γ, and β are BN parameters: the running mean, the running variance, the scale, and the shift. Note that the BN parameters are part of the model weights and should be protected properly.
Typical non-linear layers in DNNs
Layer
Description
Function
Comparison-based activation function
Binary activation function
ReLU activation function
ReLU6 activation function
Leaky ReLU activation function
Smooth activation functions
Sigmoid function
Hyperbolic tangent (tanh) function
Exponential linear units (ELU) activation function
Pooling layers (window size n)
max pooling
min pooling
As summarized in Table 3, non-linear functions in DNNs can broadly be classified into comparison-based activation functions, smooth activation functions, and pooling layers. Comparison-based activation functions (ReLU, ReLU6, LeakyReLU, and Binary activation) have been demonstrated with superior performance in deep learning applications for rapid learning and high prediction accuracy. They are essential building blocks in neural networks to introduce non-linearity, particularly in image classifications. These functions alleviate the well-known ‘vanishing gradient problem’ (neural networks could not converge) encountered when the sigmoid and tanh functions are leveraged for training. The ReLU function is the most widely adopted activation function in CNNs. The ReLU6 activation function is a variant of the ReLU that clips weights between 0 and 6. The Leaky ReLU function adopts a linear function for negative features. The Binary activation function is usually adopted in quantized NNs.
Smooth activation functions make non-trivial usage in deep learning. Similar to comparison-based activation functions, smooth activation functions introduce non-linearity to NNs. Additionally, these functions have properties of continuity, smoothness, and monotonicity to empower NNs with complex capabilities [20,79]. CryptMed focuses on three widely-adopted smooth activation functions, i.e., sigmoid, hyperbolic tangent (tanh), and ELU. They are vital building blocks in a variety of machine learning and deep learning paradigms for medical diagnoses, like medical time series predictions [79], tumor segmentation, and medical imaging denoising [86]. The tanh function and the sigmoid function are preferable over logistic regression, LSTM, and RNN for sequential and time series data prediction. The ELU function [17] is a noise-robust and precise activation function compared to the conventional ReLU activation and its variants (e.g., Leaky ReLU, Parametrized ReLU). Besides, ELU effectively diminishes the ‘vanishing gradient problem’ by setting the identity for positive features.
Secure computation over fixed-point ring. Deep learning inference operates over real-valued numbers, i.e., DNN weights and user inputs. In cleartext, they are represented in floating-point numbers. To allow CryptMed to operate in the secret sharing domain, we leverage the fixed-point representation to project values to the underlying ring . Such fixed-point representation is a common paradigm adapted in prior work [47,56,58,61]. Specifically, given a floating-point number x, we first convert it to a signed fixed-point integer with a scaling factor s embedding the fractional part. Afterwards, we project such an integer to the ring via . To represent the sign, we leverage the two’s complement representation, where the most significant bit (MSB) represents the sign. In this way, non-negative values are mapped to the lower-half ring , while negative values are mapped to the upper-half ring . Then, the MSB will be ‘0’ for a non-negative value, and ‘1’ for a negative value. In fixed-point representation, repeated multiplications may lead to integer overflow due to the excess of fractional bits (from s to bits). A common treatment is to use a secure local truncation [56,58,61], where the least s bits are chopped off ahead of subsequent multiplications.
System overview
Architecture
Figure 1 illustrates the system architecture of CryptMed which enables secure deep learning based medical diagnostic service. CryptMed operates between two parties: the hospital and the medical service provider. On the one hand, we consider that the medical service as an enterprise which deploys an NN powered medical diagnoses service offering though a proprietary NN model. On the other hand, we consider that the hospital as a customer intends to take advantage of the deep learning service to facilitate an accurate medical conclusion, while protecting its own confidential medical records (e.g., CT image, physiological data). Note that the role of the hospital in the real world can be any healthcare institutes, such as clinics, biomedical research centers, or life-science institutes. To launch a secure medical diagnostic service, the above two parties execute CryptMed’s secure NN inference protocol over the secret-shared model and secret-shared medical record. At the end, only the hospital can obtain the secret shares prediction result, which is then recovered to get the cleartext diagnosis. CryptMed provides a cryptographic guarantee such that the hospital obtains the inference result only and nothing else, while the medical service learns no information about the hospital’s medical records.
System architecture.
Threat model
CryptMed designs two-party inference secure against semi-honest adversaries. In CryptMed, the hospital and the medical service will honestly follow the prescribed protocol, yet trying to deduce auxiliary information about each other’s private input beyond what revealed from the protocol result. It is noted that such an assumption is practical. Nowadays the behavior of hospital is widely enforced by ethics, law and privacy regulations [1,23]. In the meantime, the medical service is usually offered by well-established vendors (e.g., Microsoft Project InnerEye [55], Google DeepMind Health [27]), that do not have strong incentives to risk their business model and publicity for malicious behaviors [7]. Due to the above facts and observations, such a threat model is commonly adopted in prior secure NN inference work [47,56] as well. CryptMed strives to ensure the privacy of the hospital’s medical records and the NN model (values of trained weights). Consistent with prior art [47,48,56], CryptMed does not hide the data-independent model architecture, e.g., the model size and number of layers. Lastly, CryptMed deems thwarting adversarial machine learning attacks orthogonal, which attempt to exploit the inference procedure as a blackbox oracle to extract private information. Mitigation strategies can be differentially private learning [82].
Our proposed design
In this section, we introduce CryptMed’s secure NN inference protocol for medical diagnostic applications. At a high level, our design consists of two types of secure layer evaluations: secure linear layers and secure non-linear layers. CryptMed efficiently supports a suite of secure linear layers, including the secure convolutional layers, secure fully-connect layers, secure batch normalization, and secure average pooling layers. For secure non-linear layers, CryptMed efficiently realizes a series of comparison-based non-linear layers, i.e., ReLU, ReLU6, Leaky ReLU, Binary activation, MaxPool, and MinPool. Besides, CryptMed enables rich non-linear functionalities by supporting lightweight and accurate evaluations of secure smooth activation functions, including tanh, sigmoid, and ELU. All these layers are vital building blocks in deep learning based medical diagnostic services.
In CryptMed, each secure layer securely evaluates a certain cleartext functionality in the secret sharing domain, which proceeds by taking the secret-shared inputs (features and/or kernels) and producing the secret-shared outputs passed to the succeeding secure layer. Our overarching goal is to devise a lightweight protocol for secure neural network inference with optimized interactions, while empowering rich and accurate secure functionalities for deep learning based medical services. Atop such goal, we have three prominent design insights.
Supporting lightweight secure linear layers. We first split CryptMed’s protocol into a preprocessing phase and an online phase, so as to shift as much computation as possible to preprocessing phase. Inspired by [56], we preprocess the model as secret shares and deliver corresponding shares to the hospital before medical record becomes available. So, the online phase can directly work over secret shares without any heavy cryptographic techniques (like HE) or multi-round ciphertext transmissions. Yet we are aware that the protocol in [56] involves heavy HE during preprocessing to produce and send the model shares as ciphertexts, which may not be amiable for the resource-limited hospital, like COVID-19 pandemic screening centers with handheld medical imaging scanners [34]). Instead, our protocol delicately leverages the insight from Chameleon [65] and enables the preprocessing to be purely based on lightweight computation in the secret sharing domain. As a result, our entire protocol works only with small shares, which immediately gains improvement on preprocessing and on overall communication costs over [56].
Supporting lightweight non-linear layers. For secure evaluation of non-linear layers, prior works either resort to the heavy cryptographic techniques (i.e., garbled circuits) [47,56], or circumvent the non-linearities with the square function approximations [25,37]. Unfortunately, such methods may introduce high communication overheads or induce instabilities of NN when handling complex tasks [42,52]. In CryptMed, we make observations from the field of digital circuit design [28] and present a secure comparison function that can efficiently evaluate comparison-based non-linear layers, including ReLU, ReLU6, Leaky ReLU, and Binary activation functions. At the core, this function is fully based on lightweight secret sharing with optimized interactions between the hospital and the medical service. With these designs, our experiment demonstrates a bandwidth reduction compared with prior works.
Supporting accurate activation functions over secret sharing domain. Most existing works [43,56,65] focus only on the simple ReLU function. Other essential activation functions remain under exploration, including Leaky ReLU, ReLU6, and ELU. These activation are fundamental building blocks in modern NN architectures for medical applications, such as medical image classification [15], image denoising [86], and medical times series (physiological data) prediction [63,79]. The most challenging task is to accurately and efficiently evaluate the smooth activation functions (e.g., sigmoid, tanh, and ELU) in secure domain. Such functions involve exponentiation and division operations that are knowingly expensive to be computed over secret-shared data.
Prior art tackling such challenges mainly falls into two categories. Works in the first category resort to function approximations based on polynomials. Some of them use the high-degree polynomials [4,40,47], which require heavy computational and communication costs to evaluate repeated multiplications. Others leverage the ad-hoc piecewise polynomial approximation [47,57,58,61]. These works either heavily rely on intervention and expertise on DNN model and training dataset to fine-tune the coefficients [47] or encounter severe ‘vanishing gradient problem’ making the NNs quite imprecise [57,58,61]. The work [66] in the other category uses GC to evaluate the smooth functions in a blackbox manner, suffering from prohibitively performance overheads.
Instead of the strawman approximations, CryptMed proposes secure smooth activation functions that are accurate while keeping the cryptographic costs in mind. The core idea is to make use of advancements from the field of digital circuit design [46,73,74,86] and the machine learning literature [79,80] so as to propose more precise and cryptography-friendly approximations. These approximations are non-linear and low-degree piecewise polynomials that have quantitative performance demonstrating promising accuracy over comprehensive empirical evaluations. With such approximations, CryptMed reformulates the smooth activation functions into comparison-based constructions, and thus circumvents the obstacles coming from exponentiation and division. As a result, CryptMed empowers accurate and efficient secure realizations of smooth activation functions over secret sharing domain.
Secure linear layers
CryptMed’s secure inference protocol.
The subsequent section presents CryptMed’s secure inference protocol, which is comprised of the preprocessing phase and the online inference phase as shown in Fig. 2.
Preprocessing phase. During preprocessing, the hospital and the medical service pre-generate custom secret shares of the NN model in an appropriate form which are to be used during online inference. This is a one-off computation and conducted independent of the hospital’s medical record. Let L be number of layers. The hospital takes as input the L sets of randomnesses (in tensor form) , where . Similarly, the medical service takes as input the tensors of model weights for each layer and randomnesses tensors . Such randomness tensors are independent to any party’s input and can be pre-distributed to the parties. They satisfy the relationship: . Note that the dimension of each randomness tensor is in line with the dimension of each layer’s filter. Given these inputs, the two parties perform the following steps.
For each , the medical service computes over the weight tensors and sends to the hospital.
The hospital computes for each layer.
Let denote , and denote . The medical service thus holds , and the hospital holds , i.e., an additively secret-shared weight tensor .
Online inference phase. During online inference, the hospital takes as input the tensor of a medical record , the randomnesses , and weight shares . The medical service takes as input the weight tensors and the randomnesses . They then perform the secure layer function in pipeline as follows.
The first linear layer :
The hospital computes and sends to the medical service, and uses to denote .
The medical service computes .
At this point, the hospital and the medical service hold the additive secret shares (i.e., , ) of features2
Biases can be added to the medical service’s shares locally.
outputted from the first linear layer .
Remaining linear layers :
Similar to the first layer, the hospital computes over its share of activation produced from the secure ReLU evaluation (which we will detail later), and sends it to the medical service. Such a treatment can perfectly hide the hospital’s share, and protect the activation against the medical service. It then sets .
The medical service computes . Then it gets , ensuring both parties hold additive secret shares (i.e., , ) of layer result .
Non-linear layers: The shares form secure linear layer evaluation can be fed into the secure non-linear layer, which outputs shares , of activations to each party.
Output layer: The medical service sends to the hospital, who can then integrate for reconstruction of the final inference result .
Secure non-linear layers
CryptMed supports highly efficient evaluation of the secure non-linear layers in the secret sharing domain. We observed that most of the non-linear activation functions and pooling layers can be decomposed into a series of comparison operations along with some linear operations (i.e., addition and multiplication). Besides, as mentioned above, the smooth activation functions will be delicately reformulated to the comparison-based piecewise non-linear polynomials, and thus relying on the comparison as well. With such an observation in mind, we reformulate each comparison to the MSB extraction defined as follows. Suppose we have two features , . The MSB extraction is defined as
Then, finding the maximum (or the minimum ) is reformulated as
The sign of a feature x is calculated via
Note that similar philosophy has also been adopted in the preliminary version of our paper [51], yet prior solution requires two multiplications for each comparison (i.e., ), whereas our newly proposed reformulation involves only one multiplication. Such an improvement is non-trivial, since a typical neural network usually requires a million and even billion scale of the number of comparison operations. With the above reformulation, the most challenging computation is how to securely extract the MSB in the secret sharing domain. We resort to a communication-optimized construction of the secure extraction function. Details are presented in this subsequent section.
Communication-optimized secure MSB extraction. The secure extraction function is used to securely extract the MSB of an additive-shared data and generate a boolean-shared MSB , where ℓ is the bit length. The key idea is to efficiently extract the MSB in the secret sharing domain. CryptMed’s proposed design is built upon an practical and communication-optimized construction [51], which realizes the MSB extraction via carry look-ahead adder logic. By taking advantage of the parallel prefix adder [28] (PPA), such a construction can securely produce the MSB in logarithm round complexity in the secret sharing domain.
A concrete example of 8-bit parallel prefix adder and the corresponding binary operator.
The construction of the PPA-based MSB extraction is introduced as follows. The staring point is to view the two shares of an ℓ-bit value x as two inputs of the PPA. To do so, we decompose the secret shares , as two bit strings and , respectively. Afterwards, an ℓ-bit PPA is used to calculate via a series of binary additions and pop out the carry bits . In this way, the MSB can be produced as . We provide a concrete example of 8-bit PPA in Fig. 3 and introduce the details of an ℓ-bit PPA realization as follows.
The first step is to calculate the initial signal tuple (, ): the carry generate signal and the carry propagate signal in parallel via and .
The second step is to produce the rest signal tuples via a binary operator ⊙. Given , are the inputted two adjacent signal tuples, and is the outputted single tuple. Each binary operator is defined as , where and . Such a binary operation is recursively performed over the input tuples, and the outputted signal tuples is propagated to the next layer’s nodes as inputs, until reaching the layer.
The third step is to calculate the carry bits via .
Finally, the most significant carry bit can be generated via . The MSB is calculated as .
With the above PPA-based MSB construction in mind, we present details of the secure extraction function in what follows. On each neuron, it takes as input the arithmetic shared integer feature , and produces the boolean shared MSB as output. The hospital (denoted as ) and the medical service (denoted as ) jointly compute the function as follows:
Decompose into bit strings:
sets e as , and decomposes it to bit string ;
sets f as , and decomposes it to bit string .
For each , sets and ; sets and .
Compute signal tuples :, .
Compute PPA:
Round:
and set as a dummy node.
For each , let , . and compute .
Round:
For each , let , . and compute .
Compute MSB: and set , .
Secure B2A function. Recall that in CryptMed, our proposed secure comparison function is formulated via the above proposed secure extraction. For example, finding the minimum between two features , is formulated as . The secure realization of such a formula requires secure share conversion when securely multiplying with . Namely, the boolean-shared needs to be firstly converted into its corresponding additive shares . Afterwards, the additively-shared MSB can multiply with the additive shares .
CryptMed resorts to the standard secure boolean-to-additive shares conversion construction (i.e., ) [48,87]. The aim is to convert any boolean shares to its additive shares . Given two parties, the hospital (denoted as ) and the medical service (denoted as ), the secure function is computed as follow:
sets , , and sets , ;
and compute .
Secure comparison based activation functions
CryptMed targets on four popular comparison-based activation functions, i.e., ReLU and its variants ReLU6 and Leaky ReLU, and the conventional Binary activation function, as summarized in Table 3. CryptMed manages to convert their cleartext functionalities into the MSB extraction based constructions. Through careful customizing, we propose efficient and lightweight realizations of secure function, secure function, secure function, and secure function that are purely based on secret sharing techniques. In what follows, we present the details of their secure constructions.
Secure ReLU activation function
In CryptMed, we reformulate the ReLU activation to a simpler MSB extraction problem that can be efficiently evaluated in the secret sharing domain. Given the feature x on each neuron outputted from the preceding linear layer, it is reformulated into an MSB extraction based construction via
Such a reformulated construction comprises of four atomic steps: the secureextraction, the secure NOT (i.e., ¬ operation), the secure to convert the boolean-shared MSB into additive shares, and the secure multiplication. All these steps can be efficiently realized by CryptMed’s secure linear and non-linear functions. Without loss of generality, we demonstrate the secure function over single feature element x on each neuron. Given the shares of a single input feature , the hospital (denoted as ) and the medical service (denoted as ) jointly compute the secure function as follows:
and run to get .
computes the NOT operation .
and run to get the additively shared NOT MSB .
and produce the activation .
Secure ReLU6 activation function
The ReLU6 activation function is a variant of the ReLU that clips the weights between 0 and 6. Given the feature x on each neuron, the MSB extraction based ReLU6 is converted via
Similar to ReLU, such a construction can be efficiently realized in the secret sharing domain via CryptMed’s secure linear and non-linear functions. Given the additive shares of each neuron’s feature , the hospital and the medical service jointly compute the secure function as follows:
and run to get the MSB and .
and compute the control bits and .
and run to get the additively shared control bits and .
and produce the activation .
Secure leaky ReLU function
Given the feature x on each neuron, CryptMed reformulates the Leaky ReLU into the MSB extraction based construction via
Similar to other comparison-based activation functions, the reformulated Leaky ReLU can be realized via CryptMed’s secure functions. Specifically, given the additive shares of each neuron’s feature , the hospital and the medical service jointly compute the secure function as follows:
and run to get the MSB .
and run to get the additively shared MSB .
and produce the activation .
Secure binary activation function
Given the feature x on each neuron, CryptMed converts the Binary activation function into an MSB extraction based construction as follows
Such a construction can be efficiently realized via CryptMed’s secure linear and non-linear functions. Given the additive shares of each neuron’s feature , the hospital and the medical service jointly compute the secure function as follows:
and run to get .
and run to get the additively shared NOT MSB as the activation .
Secure smooth activation functions
The smooth activation functions make non-trivial usages in deep learning. As shown in Table 3, CryptMed focuses on three widely-adopted smooth activation functions, i.e., sigmoid, tanh, and ELU. They are vital building blocks in a variety of machine learning and deep learning paradigms for medical diagnoses, like medical time series predictions [79] and medical imaging denoising [86].
CryptMed’s secure smooth activation functions are tailored from two precise and cryptography-friendly approximations: the polynomial piecewise approximations (PLAs) and the SQNL activation function family. We provide a high-level overview of our insights below.
Piecewise linear approximations. Our first insight is to make use of the PLAs from the field of digital circuit design [46,73,74,86]. They are non-linear and low-degree polynomials with quantitative performance. With these approximations, we propose the secure function and secure function. They are efficient and accurate realizations of the tanh and the sigmoid functions in the secret sharing domain. We note that prior work [57,58,61] also makes use of the PLA approximation. However, during our experiments, we observed that their proposed approximation could induce the severe ‘vanishing gradient problem’, which makes the NNs quite imprecise. More details are given later.
Replacements with SQNL-family. Our second insight is to leverage the SQNL-family [79,80] from deep learning literature. Instead of a vanilla approximation, the SQNL-family is a suite of new activation functions demonstrating promising accuracy over comprehensive empirical evaluations. It utilizes the universal approximation theorem [30,31] and introduces a quadratic non-linearity. With such an observation, CryptMed devises the secure function, secure function, and the secure function for the tanh, sigmoid, and ELU activation functions. In this subsequent sections, we provide the details of their constructions.
Secure tanh function
As defined in Table 3, the tanh function involves exponentiation and division operations that are prohibitively expensive to be evaluated in secure domain. In CryptMed, we propose the secure function and the secure function that demonstrate promising precision as plotted in Fig. 4.
Securefunction. The function is a piecewise polynomial approximation resorting to the non-linear and low-degree approximation [46,86]. Such a PLA has quantitative performance, where the average error and the maximum error are and , respectively. Given the feature on each neuron x, the cleartext functionality can be reformulated to an MSB extraction based construction via
Such a reformulated construction can be viewed as five polynomials, and each of them is triggered by a control bit. Each control bit indicates the certain threshold x located in and is derived from the MSBs of the comparison results. For example, the first polynomial in Eq. (1) is defined as and is triggered by the control bit . That is to say, when , then , and the other control bits are ‘0’s, and thus the tanh activation of x is produced based on above polynomial. In addition, the control bit stems from the MSBs of two comparison results: the MSB indicates and the MSB indicates . Note that the control bit of the last polynomial in Eq. (4) is the MSB .
The tanh function and its approximation.
To this end, we present the secure realization of the secure function based on above reformulation. Given the additive shares of each neuron’s feature , the hospital and the medical service jointly compute the secure function as follows:
and run to get the MSB , , , , .
and compute the control bits , , , and .
and run to get the additively shared bits , , , , , and .
Securefunction. The function uses the SQNL-family [79] demonstrating promising accuracy over comprehensive empirical evaluations. Given the feature on each neuron x, the cleartext functionality can be reformulated to an MSB extraction based construction via
Such a construction consists of three polynomials that are triggered by the control bits , , and the indicating the thresholds , , and , respectively. Each control bit is derived from the MSBs of comparing x with the threshold boundaries.
With the above reformulation and the additive shares of each neuron’s feature , the hospital and the medical service jointly compute the secure function as follows:
and run to get the MSB , , and .
and compute the control bits and .
and run to get the additively shared bits , , , and .
and compute the sign bit .
and produce the activation .
Secure sigmoid function
The sigmoid function, as defined in Table 3, comprises the exponentiation and division operations which are knowingly computational extensive by a secure two-party computation protocol [22,47,63].
Vanishing gradient problem of the approximation
sigmoid
Thyroid
81.47%
5.16 ∼ 9.88%
86.55%
85.85%
MNIST
96.44%
11.35%
91.41%
96.75%
CIFAR-10
47.87%
10.03%
47.87%
59.08%
The sigmoid function and its approximations.
Prior work [57,58,61] leverages the first-order PLA to approximate the sigmoid function denoted as
Such an approximation seems plausible. However, in our experiments, we observed that the approximation could introduce severe ‘vanishing gradient problem’ making the NNs hardly converge during training. As a consequence, the prediction accuracy of the trained NNs is quit undesired, particularly when dealing with complex datasets. Table 4 showcases the vanishing gradient problem of the approximation in our experiments.
Instead, CryptMed takes advantage of the insights from the field of digital circuit design [73,74] and the machine learning literature [79,80] to propose more precise approximations, i.e., the second-order PLA () and the SQNL-family replacement (). Figure 5 illustrates the approximations of sigmoid used in CryptMed and the used in prior art. Note that, all these approximations can be converted to the MSB extraction based functions and can be efficiently evaluated in the secret sharing domain. In this subsequent section, we expatiate on the secure function and secure function.
Securefunction. The function is a non-linear and second-order approximation [73,74] of the sigmoid function, where the average error and the maximum error are quantified as and , respectively. Given the feature on each neuron x, the cleartext functionality can be converted to an MSB extraction based construction via
It comprises two polynomials that are triggered by the control bits and indicating the thresholds , and , respectively. Note that here we prune off the threshold , since the polynomial keeps zero () in such a threshold.
Given above reformulation and the additive shares of each neuron’s feature , the hospital and the medical service jointly compute the secure function as follows:
and run to get the MSB , , and .
and compute the control bits and .
and run to get , , and .
and compute the sign bit .
and produce the activation .
Securefunction. The function resorts to a new activation function LogSQNL from the SQNL-family [79,80], which is a precise replacement of the sigmoid function. Given the feature on each neuron x, the cleartext functionality and corresponding MSB extraction based construction are defined as follows
The MSB extraction based construction consists of two polynomials. They are activated by the control bits and when x in the thresholds , and , respectively. Similar to the function, the polynomial in threshold is trimmed due to its zero value. Given the additive shares of each neuron’s feature , the hospital and the medical service jointly compute the secure function as follows:
and run to get the MSB , , and .
and compute the control bits and .
and run to get , , and .
and compute the sign bit .
and produce the activation .
Secure exponential linear units function
As Fig. 6 illustrates, CryptMed proposes the function taking advantages of the SQLU function from the SQNL-family [79,80]. It is defined as follows
Given the additive shares of each neuron’s feature , the hospital and the medical service jointly compute the secure activation function as follows:
and run to get the MSB , .
and compute the control bits and .
and run to get the additively shared bits , , and .
and produce the activation .
The ELU activation function and its approximation.
Secure pooling layers
The MaxPool layer (or MinPool ) can be views as the pairwise maximum (or minimum) over features within the n-width pooling window. They can be realized based on the secure extraction as follows:
Based on above equations, given the secret shares of features , the hospital and the medical service perform the secure MaxPool (or MinPool) function as follows:
For :
and run to get .
and run to get additively shared MSB .
and compute the maximum value ; or the minimum value . sets .
Finally, outputs the shares as MaxPool (or MinPool) result.
The average pooling layer can be directly computed over additive secret shares via secure addition, where n is a cleartext hyper-parameter.
Security analysis
In this section, we present a comprehensive security analysis of CryptMed’s secure inference protocol in the semi-honest adversary model. CryptMed’s core secure inference protocol is devised based on standard additive secret sharing techniques [5,26], where all the input data (i.e., medical data, neural network model) are perfectly protected as additive secret shares that are uniformly distributed in ring . Besides, during CryptMed’s protocol execution, any message transcriptions are supported by standard Beaver’s multiplication/AND procedure [8] as uniformly distributed secret shares. CryptMed provides stringent cryptographic guarantees throughout the service procedure, where only the prescribed inference result can be learned by the hospital. Nothing else about each party’s private input can be deduced from the counterparty beyond what is revealed from the inference result. In what follows, we formally define the ideal functionality , security definition, and the security proof of CryptMed’s secure NN inference protocol under the ideal/real world paradigm.
The ideal functionality of CryptMed’s secure deep neural network inference consists of the following parts:
Input. The medical service submits the DNN model and the hospital submits the medical record X to .
Computation. Upon receiving and X, conducts DNN inference and produces the inference result .
Output. outputs only to the hospital, and returns no information to the medical service.
Given the ideal functionality, we formally define the security definition.
A protocol Π securely realizes the if it provides the following guarantees in the presence of a probabilistic polynomial time (PPT) semi-honest adversary with static corruption:
Corrupted hospital. A corrupted semi-honest hospital should learn no information about the medical service’s DNN model weights except the hyper-parameters of model architecture. Formally, there should exit a PPT simulator that can simulate the view of the corrupted hospital in real-world protocol execution: .
Corrupted medical service. A corrupted semi-honest medical service should learn no information about the medical record X submitted by the hospital. Formally, there should exist a PPT simulator that can simulate the view of the corrupted medical service in real-world protocol execution: .
CryptMed’s secure deep neural network inference protocol securely emulates the ideal functionalityunder the Security Definition
2
.
We present a simulator for the corrupted medical service or the corrupted hospital, in a way that the distribution of real-world protocol execution is computationally indistinguishable to the simulated distribution under our security definition.
Simulator for the corrupted hospital: Define the simulator of Beaver’s multiplication procedure is . The emulated view is indistinguishable from the real-world view of the corrupted hospital in the multiplication procedure.
In the preprocessing phase of Π, the simulator of the corrupted hospital generates the randomness to emulate the message in real-world protocol execution. Both messages are uniformly distributed in ring . Given the security of additive secret sharing, cannot distinguish the simulated message with the message received from real-world protocol execution. computes and .
In the online phase of Π, inputs the secret shares of medical record or the activation for secure linear layers and receives no messages for the linear layers. works in a dummy way by directly outputting inputs of . Thus, the output of is identically distributed to the view . For the non-linear layers, generates and runs to compute secure multiplication over and whenever interactions are required. outputs the simulated shares of activation returned from the secure non-linear layer. conducts the above computations for every layer. At the end, outputs the simulated shares of the last layer’s result , . The reconstructed value of these two shares is uniformly distributed in ring , same as the result from the real-world protocol execution. Therefore, the output of is computationally indistinguishable to the view of the corrupted hospital.
Simulator for the corrupted medical service:
In the preprocessing phase of Π, the corrupted medical service only inputs the secret shares of model and receives no message. works in a dummy way by directly outputting the inputs of . In this way, the output of is identically distributed to the view .
In the online phase of Π, generates and outputs the randomness to emulate the message (or ) in the real-world protocol execution. Given the security of additive secret sharing, cannot distinguish the emulated message with the one received from real protocol. For the non-linear layers, generates . Whenever interactions happen in the secure activation function, runs to perform the Beaver’s secure multiplication procedure over and received from the corrupted medical service . outputs the simulated shares of activation outputted from the secure activation function. conducts the above computations for every layer. Because all simulated intermediate messages are uniformly distributed in , and given the security of additive secret sharing and Beaver’s secure multiplication procedure, the output of is computationally indistinguishable to the view of the corrupted medical service.
□
Performance evaluation
We implement a proof-of-concept prototype of CryptMed in Java. We deploy our prototype to Australian MASSIVE M3 using m3i computation nodes. Each computation node has 2.7 GHz Intel Xeon Gold 6150 CPU and 384 GB RAM, running CentOS Linux 7 system. In our experiment, we choose to protect the data as additive secret shares in 32-bit ring . In line with prior secure inference systems [25,47,64], we deploy the computation nodes representing the medical service and the hospital in a dedicated network. For cleartext neural networks implementation and training, we use the Pytorch backend and train our models on a NVIDIA Tesla V100 GPU.
We evaluate CryptMed’s performance in terms of performance and accuracy. To evaluate the performance, we train 78 neural network models based on six architectures (see the Appendix) across 13 different activation functions. These activation functions include the secure comparison-based activations (ReLU, ReLU6, Leaky ReLU, Binary activation), the original smooth activations (tanh, sigmoid, ELU) and the secure smooth activations based on different approximation approaches (, , , , , and ). They are trained over MNIST, CIFAR-10, Breast Cancer, Diabetes, Liver Disease, and Thyroid. The goal of our evaluation is to answer two questions:
Is CryptMed practical and accurate for secure medical diagnostic service?
To answer above questions, we evaluate microbenchmarks of CryptMed’s secure layer functions and a series of end-to-end system-level inference performance.
Microbenchmarks
Secure layer functions. Table 5 summarizes the performance of commonly-used secure layer functions, including a series of secure linear layers (CONV, FC, BN, AvgPool) and the secure non-linear layers ReLU and MaxPool. For demonstration purpose, we set the parameters of the sliding windows as and for the CONV kernels, the pooling window as for the AvgPool and MaxPool, and the vectors for FC. As reported, all secure layers in CryptMed are lightweight and can be efficiently evaluated within 35 ms consuming at most 1 KB bandwidth.
Performance of secure layer functions
Secure layer
Conv.
FC
BN
ReLU
MaxPool
AvgPool
Time (ms)
1.25
2.16
8.44
1.74
22.7
31.2
0.05
Comm. (Bytes)
36
100
943
4
78
234
0
Computational overhead of the secure activation functions.
For secure non-linear activation functions, we first investigate their performance in terms of execution time and network consumption, including secure comparison-based activations , , , , and secure smooth activations , , , , . Figures 7 and 8 illustrate the computational and communication overheads of secure comparison-based activations (in respective left figures) and secure smooth activations (in respective right figures). As shown, all secure smooth activations require around more resources than secure comparison-based activations. Nevertheless, CryptMed’s secure non-linear activation functions can be accomplished within 10 ms for a single execution, as plotted in Fig. 9 by grabbing unit executions. In particular, for the secure smooth functions, the SQNL-family based approximations (, , ) are relatively lightweight than the PLA based approximations ( and ). In combination with their superior prediction accuracy, as shown later, the SQNL-family based secure smooth activations would be more desirable in practical medical diagnostic applications.
Communication overhead of the secure activation functions.
Secure non-linear layers comparison with GC. We compare CryptMed’s secure realizations of the secure ReLU and MaxPool with the common GC-based solutions, since prior works target only on these two non-linear layers. The GC baseline is implemented with FlexSC [77], a Java based two-party GC framework in the semi-honest setting. In our implementation, we adopt the free-XOR and half-AND optimizations for GC. All GC-based realizations are implemented with equivalent functionalities to ours. As Fig. 10 depicts, CryptMed’s realizations are , faster then the GC baseline for secure ReLU and MaxPool, respectively. For bandwidth consumption, CryptMed’s secure ReLU and MaxPool achieve respective , bandwidth savings compared with the GC baseline. Such improvements demonstrate that CryptMed’s secret sharing based design is indeed lightweight and much more efficient than the prior works relying on GC [47,56,64,65].
Unit time of the secure activation functions.
Performance comparison of the secure ReLU and MaxPool layers with GC baseline.
Prediction workload of NNs over different secure comparison-based activation functions
Dataset
Overhead
Binary
ReLU
ReLU6
LeakyReLU
Breast cancer
time (s)
0.28
0.22
0.35
0.28
comm. (KB)
8.14
8.64
10.56
8.64
Diabetes
time (s)
0.24
0.39
0.33
0.24
comm. (KB)
40.34
40.97
43.37
40.97
Liver disease
time (s)
0.19
0.19
0.34
0.19
comm. (KB)
8.22
9.22
13.07
9.22
Thyroid
time (s)
0.88
0.94
1.34
0.89
comm. (KB)
110.46
113.59
125.60
113.59
MNIST
time (s)
0.82
0.64
1.41
0.84
comm. (MB)
0.91
0.91
0.93
0.91
CIFAR-10
time (s)
423.58
146.9
820.07
436.05
comm. (MB)
496.18
498.83
508.98
498.83
Prediction workload of NNs over different secure smooth activation functions
Dataset
Overhead
Breast cancer
time (s)
0.45
0.45
0.63
0.38
0.45
comm. (KB)
13.48
13.48
19.83
12.06
13.48
Diabetes
time (s)
0.45
0.45
0.67
0.37
0.45
comm. (KB)
47.02
47.02
54.95
45.24
47.02
Liver disease
time (s)
0.53
0.53
0.89
0.40
0.53
comm. (KB)
18.91
18.91
31.60
16.07
18.91
Thyroid
time (s)
19.37
19.54
30.56
15.39
19.54
comm. (KB)
143.86
143.86
183.51
134.97
143.86
MNIST
time (s)
2.17
2.20
3.61
1.66
2.19
comm. (MB)
0.95
0.95
1.00
0.94
0.95
CIFAR-10
time (s)
1336.20
1350.77
2304.31
991.88
1350.39
comm. (MB)
524.41
524.41
557.91
516.90
524.41
Preprocessing overhead of secure NNs
Overhead
Breast cancer
Diabetes
Liver disease
Thyroid
MNIST
CIFAR-10
Time (s)
0.1
0.03
0.06
0.28
0.07
243
Comm. (MB)
0.003
0.0022
0.018
0.048
0.45
246.4
Prediction accuracy of different comparison-based activation functions
Dataset
Binary
ReLU
ReLU6
LeakyReLU
Breast cancer
88.81%
93.0%
93.7%
91.61%
Diabetes
68.75%
71.87%
73.43%
73.43%
Liver disease
71.72%
72.0%
66.89%
71.03%
Thyroid
48.77%
86.31%
86.20%
86.11%
MNIST
75.78%
97%
96.97%
97.57%
CIFAR-10
13.2%
81%
80%
82.68%
Prediction accuracy of different approximated smooth activation functions
Dataset
sigmoid family
tanh family
ELU family
sigmoid
tanh
ELU
Breast cancer
92.3%
86.01%
93.01%
95.8%
92.3%
72.02%
93.01%
95.1%
95.8%
Diabetes
68.75%
67.7%
74.47%
72.39%
70.31%
69.79%
70.31%
72.91%
71.87%
Liver disease
71.72%
65.51%
71.72%
71.03%
69.65%
71.03%
69.65%
70.34%
69.65%
Thyroid
81.47%
5.16 ∼ 9.88%
86.55%
85.85%
85.56%
64.2%
85.88%
85.5%
85.88%
MNIST
96.44%
11.35%
91.41%
96.75%
96.03%
38.62%
96.93%
97.58%
97.67%
CIFAR-10
47.87%
10.03%
47.87%
59.08%
76.0%
19.71%
74.19%
81.27%
83.87%
CryptMed’s protocol performance
Prediction workload. Tables 6 and 7 summarize the end-to-end prediction workload of CryptMed’s system with 9 different activation functions over real-world medical applications and the commonly-used machine learning benchmarks. In summary, CryptMed’s secure NN inference system is lightweight and low-latency over all medical datasets. By using the secure comparison-based activations, CryptMed can produce deep learning based medical conclusions within 1 s consuming less than 130 KB bandwidth for all evaluated medical datasets. For more complex secure smooth activations, CryptMed requires less than 31 s and 200 KB to produce a deep learning based medical conclusion. For the ML benchmarks, CryptMed evaluates MNIST within 4 s and 1 MB network resources. For the CIFAR-10 dataset on a complicated 10 layers NN, CryptMed produces an inference within 14 min, 509 MB using secure comparison-based activations, and 38 min, 558 MB using secure smooth activations. Furthermore, we report the performance of secure preprocessing overhead in Table 8. Note that, the costs of preprocecessing are only related to the NN sizes, i.e., the amount of weights in linear layers (non-linear layers do not produce any weights).
Prediction Accuracy. Tables 9 and 10 demonstrate the prediction accuracy of our system over different activation functions. We note that, for the original smooth activation functions sigmoid, tanh, ELU, and the used in prior work [58,61], we evaluate their accuracy in cleartext domain. For each family of the smooth functions, we highlight the most precise functions and underline the one facing with gradient vanishing problem. As shown, our proposed approximations do not introduce many losses to prediction accuracy. In particular, some of them from the SQNL-family can even enhance the accuracy. Consider both the evaluation costs and the accuracy, the secure smooth functions based on the SQNL-family would be better suitable for deep learning based medical diagnostic services.
Comparison with prior art. Table 11 compares CryptMed’s end-to-end inference performance with notable prior secure NN inference works. As shown, CryptMed requires the least network resources among all other prior works with up to bandwidth saving for MNIST and up to bandwidth saving for CIFAR-10. For the medical applications, CryptMed improves up to communication over XONN, the notable work investigating medical scenario with smaller quantized NNs and network trimming optimization.
Bandwidth (MB) comparison of CryptMed with prior art
MNIST
CIFAR-10
Breast cancer
Diabetes
Liver disease
MiniONN
15.8
MiniONN
9272
XONN
0.35
XONN
0.16
XONN
0.3
CryptoNets
372.2
FALCON
1278
XONN
4.29
XONN
2599
Chameleon
10.5
Chameleon
2650
Gazelle (ReLU)
∼5000
Delphi (ReLU)
∼5100
CryptMed
0.9
CryptMed
498
CryptMed
0.008
CryptMed
0.04
CryptMed
0.009
For the state-of-the-art work Delphi [56], their all ReLU version consumes total 5100 MB, whereas CryptMed only requires 498 MB, with a enhancement.3
Preprocessing: 243 MB in CryptMed and 4915 MB in Delphi.
Such significant improvement stems from the fact that CryptMed only involves lightweight secret sharing based secure computation through out the whole service procedure. In comparison, Delphi relies on the usages of heavy cryptography, i.e., homomorphic encryption for secure preprocessing and garbled circuits for secure non-linear layers. Regarding the overall runtime, we emphasize that it is not a fair comparison to directly compare the experiment results reported in [56]. The reason is that Delphi is implemented in a different programming language (Rust) with significant optimizations and acceleration from GPU computing. Our evaluation results are not based on such optimizations.
We note that secure evaluation of non-linear layers is the performance bottleneck in secure neural network inference [56]. To support the non-linearity introduced by the original ReLU, Delphi adopts a GC-based realization. For a fair comparison, we have reported in above Fig. 10 the performance of our design and the GC-based realization. The results have validated a significant performance boost of our design over the GC-based realization ( in runtime and in communication). Even with a direct (unfair) comparison, CryptMed’s overall inference time, with much simplified implementations, is still comparable to Delphi’s reported runtime ( in CryptMed against in Delphi) which is driven by aforementioned significantly optimized and sophisticated implementations.
Conclusion
In this paper, we present CryptMed, a new secure and lightweight NN inference system towards secure intelligent medical diagnostic services. Our protocol fully resorts to the lightweight additive secret sharing techniques, free of heavy cryptographic operations as seen in prior art. The commonly-used non-linear comparison-based and smooth activation functions are well supported in a secure, efficient, and accurate manner. With CryptMed, the privacy of the medical record of the hospital and the NN model of the medical service is provably ensured with practical performance.
Footnotes
Acknowledgments
This work was supported in part by the Australian Research Council (ARC) Discovery Projects (No. DP200103308, No. DP180103251, and No. DP190102835), by the ARC Linkage Project (No. LP160101766), by the HITSZ Start-up Research Grant (No. BA45001023), by the Guangdong Basic and Applied Basic Research Foundation under Grant No. 2021A1515110027, and by the Shenzhen Science and Technology Program under Grant No. RCBS20210609103056041.
More details of model architecture
In this section, we present the detailed model architectures used in our paper (see Tables 12–17).
References
1.
104th United States Congress, Health Insurance Portability and Accountability Act of 1996 (HIPPA), 1996.
2.
N.Agrawal, A.Shahin Shamsabadi, M.J.Kusner and A.Gascón, QUOTIENT: Two-party secure neural network training and prediction, in: Proc. of ACM CCS, 2019.
3.
J.Alvarez-Valle, P.Bhatu, N.Chandran, D.Gupta, A.Nori, A.Rastogi, M.Rathee, R.Sharma and S.Ugare, Secure medical image analysis with cryptflow, arXiv preprint, 2020. arXiv:2012.05064.
4.
A.Aly and N.P.Smart, Benchmarking privacy preserving scientific operations, in: International Conference on Applied Cryptography and Network Security, Springer, 2019, pp. 509–529. doi:10.1007/978-3-030-21568-2_25.
5.
M.Atallah, M.Bykova, J.Li, K.Frikken and M.Topkara, Private collaborative forecasting and benchmarking, in: Proc. of WPES, 2004.
6.
P.Baldi, R.Baronio, E.De Cristofaro, P.Gasti and G.Tsudik, Countering gattaca: Efficient and secure testing of fully-sequenced human genomes, in: Proc. of ACM CCS, 2011.
7.
M.Barni, P.Failla, R.Lazzeretti, A.-R.Sadeghi and T.Schneider, Privacy-preserving ECG classification with branching programs and neural networks, IEEE Trans. on Information Forensics and Security (2011).
8.
D.Beaver, Efficient multiparty protocols using circuit randomization, in: Proc. of Crypto, 1991.
9.
F.Boemer, R.Cammarota, D.Demmler, T.Schneider and H.Yalame, MP2ML: A mixed-protocol machine learning framework for private inference, in: Proceedings of the 15th International Conference on Availability, Reliability and Security, 2020, pp. 1–10.
10.
R.Bost, R.A.Popa, S.Tu and S.Goldwasser, Machine learning classification over encrypted data, Cryptology ePrint Archive (2014).
11.
F.Bourse, M.Minelli, M.Minihold and P.Paillier, Fast homomorphic evaluation of deep discretized neural networks, in: Annual International Cryptology Conference, Springer, 2018, pp. 483–512.
12.
A.Brutzkus, R.Gilad-Bachrach and O.Elisha, Low latency privacy preserving inference, in: International Conference on Machine Learning, PMLR, 2019, pp. 812–821.
13.
M.Byali, H.Chaudhari, A.Patra and A.Suresh, FLASH: Fast and robust framework for privacy-preserving machine learning, Proc. Priv. Enhancing Technol.2020(2) (2020), 459–480. doi:10.2478/popets-2020-0036.
14.
E.Check Hayden, Extreme cryptography paves way to personalized medicine, Nature News519(7544) (2015), 400. doi:10.1038/519400a.
15.
W.Chen, Y.Zhang, J.He, Y.Qiao, Y.Chen, H.Shi, E.X.Wu and X.Tang, Prostate segmentation using 2D bridged U-net, in: 2019 International Joint Conference on Neural Networks (IJCNN), IEEE, 2019, pp. 1–7.
16.
H.Cho, D.J.Wu and B.Berger, Secure genome-wide association analysis using multiparty computation, Nature Biotechnology36(6) (2018), 547–551. doi:10.1038/nbt.4108.
17.
D.Clevert, T.Unterthiner and S.Hochreiter, Fast and accurate deep network learning by exponential linear units (ELUs), in: 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2–4, 2016, Conference Track Proceedings, 2016.
18.
M.Dahl, J.Mancuso, Y.Dupis, B.Decoste, M.Giraud, I.Livingstone, J.Patriquin and G.Uhma, Private machine learning in tensorflow using secure computation, arXiv preprint, 2018. arXiv:1810.08130.
19.
A.P.Dalskov, D.Escudero and M.Keller, Secure evaluation of quantized neural networks, Proc. Priv. Enhancing Technol.2020(4) (2020), 355–375. doi:10.2478/popets-2020-0077.
20.
B.DasGupta and G.Schnitger, The power of approximation: A comparison of activation functions, in: NIPS, Denver, CO, 1992, pp. 615–622.
21.
M.De Cock, R.Dowsley, A.C.Nascimento, D.Railsback, J.Shen and A.Todoki, High performance logistic regression for privacy-preserving genome analysis, BMC Medical Genomics14(1) (2021), 1–18. doi:10.1186/1755-8794-3-1.
22.
D.Demmler, T.Schneider and M.Zohner, ABY – A framework for efficient mixed-protocol secure two-party computation, in: Proc. of NDSS, 2015.
23.
European Parliament and the Council, The General Data Protection Regulation (GDPR), 2016.
24.
M.Fredrikson, S.Jha and T.Ristenpart, Model inversion attacks that exploit confidence information and basic countermeasures, in: Proc. of ACM CCS, 2015.
25.
R.Gilad-Bachrach, N.Dowlin, K.Laine, K.Lauter, M.Naehrig and J.Wernsing, Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy, in: Proc. of ICML, 2016.
26.
O.Goldreich, S.Micali and A.Wigderson, How to play any mental game or a completeness theorem for protocols with honest majority, in: Proc. of STOC, 1987.
27.
Google DeepMind Health, 2020, https://deepmind.com/blog/announcements/deepmind-health-joins-google-health.
28.
D.Harris, A taxonomy of parallel prefix networks, in: The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, Vol. 2, IEEE, 2003, pp. 2213–2217.
29.
B.Hie, H.Cho and B.Berger, Realizing private and practical pharmacological collaboration, Science362(6412) (2018), 347–350. doi:10.1126/science.aat4807.
K.Hornik, M.Stinchcombe and H.White, Multilayer feedforward networks are universal approximators, Neural networks2(5) (1989), 359–366. doi:10.1016/0893-6080(89)90020-8.
32.
X.Hu, W.Zhang, K.Li, H.Hu and N.Yu, Secure nonlocal denoising in outsourced images, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)12(3) (2016), 1–23.
33.
Y.Huang, L.Malka, D.Evans and J.Katz, Efficient privacy-preserving biometric identification, in: Proc. of NDSS, 2011.
34.
A.Jacobi, M.Chung, A.Bernheim and C.Eber, Portable chest X-ray in coronavirus disease-19 (COVID-19): A pictorial review, Clinical Imaging (2020).
35.
K.A.Jagadeesh, D.J.Wu, J.A.Birgmeier, D.Boneh and G.Bejerano, Deriving genomic diagnoses without revealing patient genomes, Science357(6352) (2017), 692–695. doi:10.1126/science.aam9710.
36.
S.Jha, L.Kruger and V.Shmatikov, Towards practical privacy for genomic computation, in: 2008 IEEE Symposium on Security and Privacy (SP 2008), IEEE, 2008, pp. 216–230. doi:10.1109/SP.2008.34.
37.
C.Juvekar, V.Vaikuntanathan and A.Chandrakasan, GAZELLE: A low latency framework for secure neural network inference, in: Proc. of 27th USENIX Security, 2018.
38.
G.Kaissis, A.Ziller, J.Passerat-Palmbach, T.Ryffel, D.Usynin, A.Trask, I.Lima, J.Mancuso, F.Jungmann, M.-M.Steinbornet al., End-to-end privacy preserving deep learning on multi-institutional medical imaging, Nature Machine Intelligence3(6) (2021), 473–484. doi:10.1038/s42256-021-00337-8.
39.
G.A.Kaissis, M.R.Makowski, D.Rückert and R.F.Braren, Secure, privacy-preserving and federated machine learning in medical imaging, Nature Machine Intelligence2(6) (2020), 305–311. doi:10.1038/s42256-020-0186-1.
40.
M.Keller, MP-SPDZ: A versatile framework for multi-party computation, in: Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, 2020, pp. 1575–1590. doi:10.1145/3372297.3417872.
41.
N.Kumar, M.Rathee, N.Chandran, D.Gupta, A.Rastogi and R.Sharma, Cryptflow: Secure tensorflow inference, in: 2020 IEEE Symposium on Security and Privacy (SP), IEEE, 2020, pp. 336–353. doi:10.1109/SP40000.2020.00092.
42.
M.Leshno, V.Y.Lin, A.Pinkus and S.Schocken, Multilayer feedforward networks with a nonpolynomial activation function can approximate any function, Neural networks6(6) (1993), 861–867. doi:10.1016/S0893-6080(05)80131-5.
43.
S.Li, K.Xue, B.Zhu, C.Ding, X.Gao, D.Wei and T.Wan, FALCON: A Fourier transform based approach for fast and secure convolutional neural network predictions, in: Proc. of IEEE/CVF CVPR, 2020.
44.
W.Li, F.Milletarì, D.Xu, N.Rieke, J.Hancox, W.Zhu, M.Baust, Y.Cheng, S.Ourselin, M.J.Cardosoet al., Privacy-preserving federated brain tumour segmentation, in: International Workshop on Machine Learning in Medical Imaging, Springer, 2019, pp. 133–141. doi:10.1007/978-3-030-32692-0_16.
45.
X.Li, Y.Gu, N.Dvornek, L.H.Staib, P.Ventola and J.S.Duncan, Multi-site fMRI analysis using privacy-preserving federated learning and domain adaptation: ABIDE results, Medical Image Analysis65 (2020), 101765. doi:10.1016/j.media.2020.101765.
46.
C.-W.Lin and J.-S.Wang, A digital circuit design of hyperbolic tangent sigmoid function for neural networks, in: 2008 IEEE International Symposium on Circuits and Systems, IEEE, 2008, pp. 856–859.
47.
J.Liu, M.Juuti, Y.Lu and N.Asokan, Oblivious neural network predictions via minionn transformations, in: Proc. of ACM CCS, 2017.
48.
X.Liu, B.Wu, X.Yuan and X.Yi, Leia: A lightweight cryptographic neural network inference system at the edge, IEEE Transactions on Information Forensics and Security17 (2022), 237–252. doi:10.1109/TIFS.2021.3138611.
49.
X.Liu and X.Yi, Privacy-preserving collaborative medical time series analysis based on dynamic time warping, in: European Symposium on Research in Computer Security, Springer, 2019, pp. 439–460.
50.
X.Liu, Y.Zheng, X.Yi and S.Nepal, Privacy-preserving collaborative analytics on medical time series data, IEEE Transactions on Dependable and Secure Computing19(3) (2022), 1687–1702. doi:10.1109/TDSC.2020.3035592.
51.
X.Liu, Y.Zheng, X.Yuan and X.Yi, MediSC: Towards secure and lightweight deep learning as a medical diagnostic service, in: European Symposium on Research in Computer Security, Springer, 2021, pp. 519–541.
52.
Q.Lou and L.Jiang, SHE: A fast and accurate deep neural network for encrypted data, in: Proc. of NeurIPS, 2019, pp. 10035–10043.
53.
Q.Lou, W.-J.Lu, C.Hong and L.Jiang, Falcon: Fast spectral inference on encrypted data, Proc. of NeurIPS33 (2020).
54.
M.Y.Lu, R.J.Chen, D.Kong, J.Lipkova, R.Singh, D.F.Williamson, T.Y.Chen and F.Mahmood, Federated learning for computational pathology on gigapixel whole slide images, Medical image analysis76 (2022), 102298. doi:10.1016/j.media.2021.102298.
55.
Microsoft Project InnerEye, 2020, https://www.microsoft.com/en-us/research/project/medical-image-analysis/.
56.
P.Mishra, R.Lehmkuhl, A.Srinivasan, W.Zheng and R.A.Popa, DELPHI: A cryptographic inference service for neural networks, in: USENIX Security Symposium, 2020.
57.
P.Mohassel and P.Rindal, ABY3: A mixed protocol framework for machine learning, in: Proc. of ACM CCS, 2018.
58.
P.Mohassel and Y.Zhang, SecureML: A system for scalable privacy-preserving machine learning, in: Proc. of IEEE S&P, 2017.
59.
V.Nikolaenko, U.Weinsberg, S.Ioannidis, M.Joye, D.Boneh and N.Taft, Privacy-preserving ridge regression on hundreds of millions of records, in: Proc. of IEEE S&P, 2013.
R.Rachuri and A.Suresh, Trident: Efficient 4PC framework for privacy preserving machine learning, 2020.
63.
D.Rathee, M.Rathee, R.K.K.Goli, D.Gupta, R.Sharma, N.Chandran and A.Rastogi, SiRnn: A math library for secure RNN inference, in: IEEE Symposium on Security and Privacy, IEEE, 2021, pp. 1003–1020.
64.
M.S.Riazi, M.Samragh, H.Chen, K.Laine, K.Lauter and F.Koushanfar, XONN: XNOR-based oblivious deep neural network inference, in: Proc. of 28th USENIX Security, 2019.
65.
M.S.Riazi, C.Weinert, O.Tkachenko, E.M.Songhori, T.Schneider and F.Koushanfar, Chameleon: A hybrid secure computation framework for machine learning applications, in: Proc. of AsiaCCS, 2018.
66.
B.D.Rouhani, M.S.Riazi and F.Koushanfar, Deepsecure: Scalable provably-secure deep learning, in: Proceedings of the 55th Annual Design Automation Conference, 2018, pp. 1–6.
67.
T.Ryffel, A.Trask, M.Dahl, B.Wagner, J.Mancuso, D.Rueckert and J.Passerat-Palmbach, A generic framework for privacy preserving deep learning, arXiv preprint, 2018. arXiv:1811.04017.
68.
A.-R.Sadeghi and T.Schneider, Generalized universal circuits for secure evaluation of private functions with application to data classification, in: International Conference on Information Security and Cryptology, Springer, 2008, pp. 336–353.
69.
A.Salem, P.Berrang, M.Humbert and M.Backes, Privacy-preserving similar patient queries for combined biomedical data, Proc. of PETS (2019).
70.
A.Sanyal, M.Kusner, A.Gascon and V.Kanade, TAPAS: Tricks to accelerate (encrypted) prediction as a service, in: International Conference on Machine Learning, PMLR, 2018, pp. 4490–4499.
71.
M.J.Sheller, G.A.Reina, B.Edwards, J.Martin and S.Bakas, Multi-institutional deep learning modeling without sharing patient data: A feasibility study on brain tumor segmentation, in: International MICCAI Brainlesion Workshop, Springer, 2018, pp. 92–104.
72.
O.Tkachenko, C.Weinert, T.Schneider and K.Hamacher, Large-scale privacy-preserving statistical computations for distributed genome-wide association studies, in: Proc. of ACM AsiaCCS, 2018.
73.
M.Tommiska, Efficient digital implementation of the sigmoid function for reprogrammable logic, IEE Proceedings-Computers and Digital Techniques150(6) (2003), 403–411. doi:10.1049/ip-cdt:20030965.
74.
I.Tsmots, O.Skorokhoda and V.Rabyk, Hardware implementation of sigmoid activation functions using FPGA, in: 2019 IEEE 15th International Conference on the Experience of Designing and Application of CAD Systems (CADSM), IEEE, 2019, pp. 34–38. doi:10.1109/CADSM.2019.8779253.
75.
S.Wagh, D.Gupta and N.Chandran, Securenn: 3-party secure computation for neural network training, Proc. of PETS (2019).
76.
S.Wagh, S.Tople, F.Benhamouda, E.Kushilevitz, P.Mittal and T.Rabin, Falcon: Honest-majority maliciously secure framework for private deep learning, arXiv preprint, 2020. arXiv:2004.02229.
X.S.Wang, Y.Huang, Y.Zhao, H.Tang, X.Wang and D.Bu, Efficient genome-wide, privacy-preserving similar patient query based on private edit distance, in: Proc. of ACM CCS, 2015.
79.
A.Wuraola and N.Patel, SQNL: A new computationally efficient activation function, in: 2018 International Joint Conference on Neural Networks (IJCNN), IEEE, 2018, pp. 1–7.
80.
A.Wuraola, N.Patel and S.K.Nguang, Efficient activation functions for embedded inference engines, Neurocomputing442 (2021), 73–88. doi:10.1016/j.neucom.2021.02.030.
81.
P.Xie, B.Wu and G.Sun, BAYHENN: Combining Bayesian deep learning and homomorphic encryption for secure DNN inference, in: Proc. of IJCAI, 2019, pp. 4831–4837.
82.
L.Yu, L.Liu, C.Pu, M.E.Gursoy and S.Truex, Differentially private model publishing for deep learning, in: Proc. of S&P, IEEE, 2019.
83.
Q.Zhang, C.Wang, H.Wu, C.Xin and T.V.Phuong, GELU-net: A globally encrypted, locally unencrypted deep neural network for privacy-preserved learning, in: Proc. of IJCAI, 2018, pp. 3933–3939.
84.
W.Zheng, R.Popa, J.E.Gonzalez and I.Stoica, Helen: Maliciously secure coopetitive learning for linear models, in: Proc. of IEEE S&P, 2019.
85.
Y.Zheng, H.Cui, C.Wang and J.Zhou, Privacy-preserving image denoising from external cloud databases, IEEE Transactions on Information Forensics and Security12(6) (2017), 1285–1298. doi:10.1109/TIFS.2017.2656824.
86.
Y.Zheng, H.Duan, X.Tang, C.Wang and J.Zhou, Denoising in the dark: Privacy-preserving deep neural network-based image denoising, IEEE Transactions on Dependable and Secure Computing18(3) (2019), 1261–1275. doi:10.1109/TDSC.2019.2907081.
87.
Y.Zheng, H.Duan and C.Wang, Towards secure and efficient outsourcing of machine learning classification, in: Proc. of ESORICS, Springer, 2019.
88.
Y.Zheng, C.Wang and J.Zhou, Toward secure image denoising: A machine learning based realization, in: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2018, pp. 6936–6940. doi:10.1109/ICASSP.2018.8462073.
89.
A.Ziller, A.Trask, A.Lopardo, B.Szymkow, B.Wagner, E.Bluemke, J.-M.Nounahon, J.Passerat-Palmbach, K.Prakash, N.Roseet al., PySyft: A library for easy federated learning, in: Federated Learning Systems, Springer, 2021, pp. 111–139. doi:10.1007/978-3-030-70604-3_5.
