Sanitizing hidden activations for improving adversarial robustness of convolutional neural networks

Abstract

Deep learning is gaining significant traction in a wide range of areas. Whereas, recent studies have demonstrated that deep learning exhibits the fatal weakness on adversarial examples. Due to the black-box nature and un-transparency problem of deep learning, it is difficult to explain the reason for the existence of adversarial examples and also hard to defend against them. This study focuses on improving the adversarial robustness of convolutional neural networks. We first explore how adversarial examples behave inside the network through visualization. We find that adversarial examples produce perturbations in hidden activations, which forms an amplification effect to fool the network. Motivated by this observation, we propose an approach, termed as sanitizing hidden activations, to help the network correctly recognize adversarial examples by eliminating or reducing the perturbations in hidden activations. To demonstrate the effectiveness of our approach, we conduct experiments on three widely used datasets: MNIST, CIFAR-10 and ImageNet, and also compare with state-of-the-art defense techniques. The experimental results show that our sanitizing approach is more generalized to defend against different kinds of attacks and can effectively improve the adversarial robustness of convolutional neural networks.

Keywords

Adversarial examples sanitizing hidden activations adversarial robustness convolutional neural networks

1 Introduction

Machine learning is driving progress in many areas such as computer vision, natural language processing and self-driving cars. Like any technology, machine learning is not immune to attacks. There are different categories of adversarial threats on machine learning, such as poisoning, extraction, inference and evasion. As is well known, one fatal weakness of machine learning is its sensibility of adversarial examples, which are mainly exploited to launch evasion attacks. Adversarial examples are carefully crafted examples that have small difference from legitimate examples but can fool or mislead machine learning models. Adversarial examples pose a serious threat to the application of machine learning in safety- and security-critical scenarios such as face recognition, autonomous driving and medical diagnosis. Since first discovered by Szegedy et al. [1], adversarial examples remain a hot topic and have drawn widespread attention from researchers. A large number of attack approaches have been proposed for generating adversarial examples.

As we know, machine learning models usually lack interpretability and they do not explain their predictions. Especially for deep learning models, e.g., deep neural networks, they are faced with black-box nature and un-transparency problem. These issues have contributed to the raging of adversarial examples. As so far, it has not been able to explain the reason for the existence of adversarial examples, which hinders the research on defense techniques. Defenses against adversarial examples are in the extremely passive condition. More seriously, adversarial examples are not only difficult to defend but also difficult to detect. As Francois Chollet [2] argues, “There are no fixes for the fundamental brittleness of deep neural networks.”

In this paper, we focus on improving the adversarial robustness of convolutional neural networks so as to defend against adversarial examples. We first explore how adversarial examples behave inside the network through visualization. We find that adversarial examples produce perturbations that look like “noise” in hidden activations. These perturbations have an amplification effect through forwarding propagation and finally accumulate enough errors to fool the network. Motivated by this observation, we propose an approach, termed as sanitizing hidden activations, to improve the adversarial robustness of convolutional neural networks. The sanitizing approach is designed to eliminate or reduce the perturbations in hidden activations, which can help the network correctly recognize adversarial examples.

In summary, we make the following contributions:

(1) We ascertain the cause of adversarial examples fooling the network through visualization, which promotes the research for improving the adversarial robustness of neural networks.

(2) We propose a novel approach—sanitizing hidden activations—to improve the adversarial robustness of convolutional neural networks.

(3) We present three specific sanitizing methods and verify their effectiveness through comprehensive experiments.

The remainder of the paper is organized as follows. In Section 2, we introduce the related work, including adversarial attacks and defenses. In Section 3, we analyze perturbations in hidden activations through visualization of an MNIST network. In Section 4, we present the sanitizing approach for improving the adversarial robustness of convolutional neural networks. In Section 5, we present the experimental results. In Section 6, we conclude the work.

2 Related work

2.1 Adversarial attacks

Since Szegedy et al. [1] first demonstrated the existence of adversarial examples in neural networks, there are many types of adversarial attacks being proposed one after another. We just introduce three representative adversarial attacks, which are tested in our experiments in this article.

(1) Fast Gradient Sign Method (FGSM)

Goodfellow et al. [3] proposed a fast method named as fast gradient sign method (FGSM) to craft adversarial examples. FGSM is a one-step algorithm and can be formulated as: $\hat{x} = x + ɛ \cdot sign (\nabla_{x} J (θ, x, y))$ (1) where x denotes the original example, y is the true label of x, θ is the model parameters, J ( ) is the cost function of the model, ɛ is the magnitude of perturbations, and $\hat{x}$ is the adversarial example. FGSM updates the original example x by ɛ along the direction of the gradient of the cost function to obtain the adversarial example $\hat{x}$ .

FGSM is regarded as a benchmark method for crafting adversarial examples. There are many variants of FGSM, e.g., the basic iterative method (BIM) [4], the iterative least-likely class method (ILCM) [4], and the momentum iterative method (MIM) [5].

(2) Carlini and Wagner Attack (C&W)

Carlini and Wagner [6] introduced an optimization-based method, termed as C&W, to search for high-confidence adversarial examples with small magnitude of perturbations. C&W is formulated as: $\begin{matrix} min {∥ δ ∥}_{p} + c \cdot f (x + δ) \\ s . t . x + δ \in [0, 1]^{m} \end{matrix}$ (2) where f ( ) is an objective function such that C (x + δ) = t if and only if f (x + δ) ⩽0. The authors discussed three distance measurements: ℓ₀, ℓ₂, and ℓ_∞. Correspondingly, C&W has three forms: C&W₀, C&W₂, and C & W_∞. Since C&W can achieve high attack success rate with small perturbations, it is treated as one of the most effective adversarial attacks.

(3) Projected Gradient Descent (PGD)

Madry et al. [7] proposed an iterative attack called the projected gradient descent (PGD). The formulation of PGD is expressed as: $x^{t + 1} = \prod_{x + S} (x^{t} + α \cdot sign (\nabla_{x} L (θ, x, y)))$ (3)

In each iteration, PGD first updates the perturbed example by a gradient descent step α. Then it projects the perturbed example into the feasible solution space, i.e., a maximum perturbation ɛ of the original example.

2.2 Defenses against adversarial attacks

Currently, the defense techniques against adversarial attacks can be classified into five categories.

(1) Adversarial training

Adversarial training tries to train a robust model by injecting adversarial examples into training data. Kurakin et al. [8] proposed the naive adversarial training to scale adversarial training to large-scale datasets. Madry et al. [7] proposed the PGD-based adversarial training via training the model with adversarial examples generated by PGD attack. Tramer et al. [9] introduced the ensemble adversarial training that augments training data with adversarial examples transferred from other models.

(2) Gradient masking/regularization

Since numerous adversarial attacks utilize gradients of the model to calculate adversarial perturbations, some researchers propose reducing the effectiveness of adversarial attacks by hiding the gradients, which is termed as gradient masking or regularization. Lyu et al. [10] proposed a gradient regularization method by penalizing the gradient of the loss function to incorporate robustness into the network. Papernot et al. [11] introduced a defensive mechanism called defensive distillation to use the knowledge of the network to improve its own robustness. Ross and Doshi-Velez [12] studied input gradient regularization as a defense method for improving the robustness against adversarial attacks.

(3) Input transformation

Some research has attempted to remove the adversarial perturbations from inputs before feeding them into machine learning models, which is referred to as input transformation. Xie et al. [13] introduced a random transformation-based defense by making testing examples first go through two additional randomization layers. Song et al. [14] proposed PixelDefend to purify adversarial perturbations in inputs. Buckman et al. [15] proposed the thermometer encoding method to increase the robustness of the network to adversarial examples. Guo et al. [16] investigated strategies that defend against adversarial attacks with different image transformation techniques, such as bit-depth reduction, JPEG compression and total variance minimization. Bafna et al. [17] presented an sparse discrete Fourier transform to thwart L₀ attacks.

(4) Detection

Given the difficulty in recognizing adversarial examples correctly, some research aims to detect adversarial examples and then reject them. Meng and Chen [18] introduced a defense framework called MagNet, which includes one or more separate detector networks and a reformer network. Li and Li [19] used statistics of convolution filters in CNN-based neural networks to identify adversarial examples. Ma et al. [20] proposed a local intrinsic dimensionality-based detector to discriminate adversarial examples from legitimate ones. Xu et al. [21] proposed the feature squeezing method to detect adversarial examples in deep neural networks. Roth et al. [22] proposed a statistical test for detecting adversarial examples. Li et al. [23] proposed detection methods using the generative classifier’s logit values. Yin et al. [24] presented a detection method that provides performance guarantee to norm constrained adversarial examples.

(5) Miscellaneous

There are some miscellaneous defense methods. For example, Gao et al. [25] proposed to insert a masking layer immediately before the layer handling the classification. Cao and Gong [26] introduced the region-based classification defense method, which takes the majority prediction on examples that are uniformly sampled from a hypercube around the adversarial example. Pang et al. [27] presented a method that explores the interaction among individual networks to improve robustness for ensemble models. Verma and Swanu [28] proposed an approach which changes the way the output is represented and decoded to improve the adversarial robustness of neural network. Xiao et al. [29] advocated the use of k-Winners-Take-All activation for defending against gradient-based adversarial attacks.

Although many types of defense methods have been proposed, the progress of defense techniques is still not optimistic. As stated in [30], most of these defense methods can be easily bypassed by new carefully crafted adversarial examples.

3 Perturbations in hidden activations

Before introducing our approach for improving the robustness of convolutional neural networks, we first explain why adversarial examples can successfully fool the network.

We trained a lightweight network for the MNIST 1 dataset classification task, which is termed as MNIST-CNN. The architecture of MNIST-CNN is given in Table 1. This architecture is also used in [6] and [11]. Except for input and output layers, MNIST-CNN has eight hidden layers. For convenience, we term these hidden layers in turn as CONV1, CONV2, POOL3, CONV4, CONV5, POOL6, FC7 and FC8. Our training approaches and hyperparameters are identical to those presented in [6]. We achieve a test accuracy of 98.65% on MNIST-CNN.

Table 1
Model architecture of MNIST-CNN

Layer name Layer type Output shape

INPUT / 28×28×1

CONV1 Convolution+ReLU 26×26×32

CONV2 Convolution+ReLU 24×24×32

POOL3 Max Pooling 12×12×32

CONV4 Convolution+ReLU 10×10×64

CONV5 Convolution+ReLU 8×8×64

POOL6 Max Pooling 4×4×64

FC7 Fully Connected+ReLU 200

FC8 Fully Connected+ReLU 200

OUTPUT Softmax 10

Layer name	Layer type	Output shape
INPUT	/	28×28×1
CONV1	Convolution+ReLU	26×26×32
CONV2	Convolution+ReLU	24×24×32
POOL3	Max Pooling	12×12×32
CONV4	Convolution+ReLU	10×10×64
CONV5	Convolution+ReLU	8×8×64
POOL6	Max Pooling	4×4×64
FC7	Fully Connected+ReLU	200
FC8	Fully Connected+ReLU	200
OUTPUT	Softmax	10

We first feed the network with a legitimate example and an adversarial example respectively, and then observe hidden activation changes. The legitimate example is a clean digit “0” image randomly selected from the MNIST test data, and the adversarial example is the corresponding adversarial image generated by the untargeted PGD [7] attack under 40 iterations with a maximum perturbation ɛ = 0.3. For convolutional and max-pooling layers in MNIST-CNN, the hidden activation is a stack of feature maps. We just select one feature map for illustration. For fully connected layers in MNIST-CNN, the hidden activation is a vector. In order to see the changes more clearly, we reshape the vector to a two-dimensional matrix.

The hidden activations of different layers of MNIST-CNN associated with a legitimate “0” image and an adversarial “0” image are shown in Fig. 1. We can see that the activations of the adversarial “0” image are different from those of the legitimate “0” image. Obviously, the adversarial “0” image produces some perturbations to the activations of hidden layers. Intuitively, the perturbations in hidden activations increase gradually with network forward propagating. The perturbations have an amplification effect and finally accumulate enough errors to fool the network. Therefore, we can conclude that the reason for an adversarial example fooling a network is that the adversarial example causes errors to hidden activations and the errors are amplified through forward propagation, which eventually mislead the network to the wrong prediction.

Fig. 1

Hidden activations of MNIST-CNN by feeding the model with a legitimate “0” image and an adversarial “0” image, respectively. The adversarial “0” image is generated by PGD attack and is incorrectly recognized as the digit “6”. For convolutional and max-pooling layers, the feature maps for each pair examples are from the same channels of hidden activations. For fully connected layers FC7 and FC8, the activation vector is reshaped to a matrix with size 20×10.

4 Sanitizing hidden activations

According to the analysis in Section 3, adversarial examples fool the network by producing perturbations to hidden activations. A natural idea is to improve the robustness of convolutional neural networks by removing the perturbations in hidden activations. We propose an approach, termed as sanitizing hidden activations, to eliminate or reduce the perturbations in hidden activations.

For convolutional neural networks, only the convolutional layers and the fully connected layers compute activations, so our sanitizing approach is designed to implement on hidden activations (i.e., the outputs) of the convolutional layers or the fully connected layers. As Fig. 2 shows, if the hidden activation is a stack of feature maps, e.g., the activation of a convolutional layer, the sanitizing operation is conducted on each feature map independently. If the hidden activation is a vector, e.g., the activation of a fully connected layer, the sanitizing operation is directly conducted on the activation vector. For ease of exposition, the object to be sanitized is directly called the hidden activation in the following text.

Fig. 2

Sanitizing for the convolutional layer and the fully connected layer.

4.1 Non-local-means sanitizer

We first present the non-local-means sanitizer, which is referred from a common approach for image denoising–non-local means algorithm [31]. Given the hidden activation a = {a (i) ; i ∈ N}, the sanitized value $\hat{a} (i)$ for the element a (i) is computed as: $\hat{a} (i) = \sum_{j \in N} w (i, j) \cdot a (j)$ (4) where w (i, j) denotes the weight between the elements a (i) and a (j). w (i, j) is measured as the similarity between $a (N_{i})$ and $a (N_{j})$ , where $a (N_{k})$ denotes a square neighborhood of fixed size and centered at the element a (k). w (i, j) is defined as: $w (i, j) = \frac{1}{Z (i)} e^{- \frac{{∥ a (N_{i}) - a (N_{j}) ∥}_{2, σ}^{2}}{h^{2}}}$ (5) where $| | a (N_{i}) - a (N_{j} {) | |}_{2, σ}^{2}$ denotes the Gaussian weighted Euclidean distance between $a (N_{i})$ and $a (N_{j})$ , and σ > 0 is the standard deviation of the Gaussian kernel. Z (i) is the normalizing constant and is defined as: $Z (i) = \sum_{j} e^{- \frac{{∥ a (N_{i}) - a (N_{j}) ∥}_{2, σ}^{2}}{h^{2}}}$ (6) where h is a decay parameter that acts as a degree of filtering. A higher h results in a smoother image, at the expense of blurring features.

4.2 Median-filtering sanitizer

Another approach we introduced for sanitizing hidden activations is the median-filtering sanitizer, which uses a common non-linear technique for image denoising: median filtering. Median filtering is known to be excelled at removing the “salt-and-pepper” noise. We analytically investigate that some adversarial attacks perturbed hidden activations in the “salt-and-pepper” way. So we think this type of sanitizing approach will be effective in some cases.

The median-filtering sanitizer is defined as: $\hat{a} (i) = median {a_{j} | \forall j \in L_{i}}$ (7) where the element a (i) is replaced by the median value over a local region $L_{i}$ of the element a (i).

4.3 Mask sanitizer

Besides the above two approaches, we also propose another novel approach for sanitizing hidden activations. Its formulation is as follows: $\hat{a} (i) = {\begin{matrix} 0, \\ a (i), \end{matrix} \begin{matrix} if a (i) ⩽ η \cdot mean {a (j) | \forall j \in N} \\ otherwise \end{matrix}$ (8) where mean {a (j) | ∀ j ∈ N} denotes the mean value of the hidden activation and η (η ⩾ 0, default value is 1.0) is a scale parameter that controls the offset on the mean value. This sanitizer means that if the element a (i) is less than or equal to a specified value, its sanitized form takes the value of 0; otherwise, do nothing.

Because this sanitizer works like a “mask” that directly covers the elements with small values, it is called the mask sanitizer. The intuition for proposing this sanitizer is that we find only the elements with large values in hidden activations are crucial to the final prediction of the network, and adversarial attacks usually only have the ability to perturb the elements with small values. Experiments show that the mask sanitizer has amazing effectiveness in improving the robustness of convolutional neural networks.

5 Experiments

In this section, we conduct experiments to demonstrate the effectiveness of our sanitizing approach. In Section 5.1, we present the experimental setup. In Section 5.2, we evaluate the effectiveness of our sanitizing approach. In Section 5.3, we present the comparisons with state-of-the-art defenses. In Section 5.4, we present a case study on MNIST-CNN to explain why our sanitizing approach can improve the adversarial robustness of convolutional neural networks.

5.1 Experimental setup

Datasets. We evaluate the effectiveness of our sanitizing approach on three widely used image classification datasets: MNIST, CIFAR-10 2 and ImageNet 3 . The MNIST dataset consists of 70,000 28×28 gray handwritten digits in 10 classes, with a training set of 60,000 examples and a test set of 10,000 examples. The CIFAR-10 dataset consists of 60,000 32×32 color images in 10 classes, with 50,000 training images and 10,000 test images. ImageNet is the most famous image dataset containing tens of millions of natural images. We sub-sample from ImageNet and generate a tiny dataset with 100,000 training images and 10,000 test images across 200 classes, and all the images are resized to 256×256 pixels.

Networks. For MNIST, we use the MNIST-CNN model introduced in Section 3. Its accuracy is 98.65%. For CIFAR-10, we use the ALL-CNN model introduced in [32]. Its accuracy is 90.67%. For the tiny ImageNet dataset, we use the ResNet-152 model introduced in [33]. We adopt the pre-trained weights on the ImageNet dataset and fine-tune them to obtain a top-1 accuracy of 83.29%.

Sanitizing operations. The sanitizing approach is designed to implement on hidden activations (i.e., the outputs) of the convolutional layers or the fully connected layers. Therefore, for MNIST-CNN, the sanitizing operations are implemented on CONV1, CONV2, CONV4, CONV5, FC7 and FC8. Because ALL-CNN consists solely of convolutional layers, the sanitizing operations are implemented on all the convolutional layers. For ResNet-152, there are 152 convolutional and fully connected layers. To reduce the overhead of sanitizing operations, the sanitizing operations are implemented on the outputs of each building block and the last fully connected layer.

Adversarial attacks. We test our sanitizing approach against three typical adversarial attacks: FGSM, C&W and PGD. Please note that pixel values of all the images are scaled to be in the range of [0, 1] during the experiment. Therefore, for FGSM, the maximum perturbation ɛ = 0.3. For C&W, the use the version of C & W_∞, and all the hyperparameters are identical to those in [6]. For PGD, the maximum perturbation ɛ = 0.3, the attack step size α = 0.01 and the number of attack iterations n = 40. All the attacks are untargeted versions.

5.2 Effectiveness of sanitizing approach

On MNIST, CIFAR-10 and ImageNet, we evaluate the effectiveness of our sanitizing approach against three typical adversarial attacks: FGSM, C&W and PGD. The evaluation results are shown in Table 2.

Table 2
Evaluation results of sanitizing approach on MNIST, CIFAR-10 and ImageNet

Dataset Method Legitimate Adversarial attacks

FGSM C&W PGD

MNIST Baseline 98.65% 70.53% 2.34% 53.47%

Non-local-means sanitizer 97.25% 75.84% 80.27% 76.28%

Median-filtering sanitizer 97.39% 75.27% 82.94% 77.32%

Mask sanitizer 96.24% 81.95% 87.28% 83.57%

CIFAR-10 Baseline 90.67% 48.82% 1.03% 37.87%

Non-local-means sanitizer 88.24% 62.93% 72.37% 58.28%

Median-filtering sanitizer 89.18% 65.78% 76.86% 60.03%

Mask sanitizer 89.26% 76.18% 80.21% 78.34%

ImageNet Baseline 83.29% 10.23% 0.21% 9.89%

Non-local-means sanitizer 82.56% 57.27% 69.04% 59.83%

Median-filtering sanitizer 83.02% 55.49% 67.35% 62.45%

Mask sanitizer 82.20% 67.37% 73.84% 69.25%

Dataset	Method	Legitimate	Adversarial attacks
MNIST	Baseline	98.65%	70.53%	2.34%	53.47%
	Non-local-means sanitizer	97.25%	75.84%	80.27%	76.28%
	Median-filtering sanitizer	97.39%	75.27%	82.94%	77.32%
	Mask sanitizer	96.24%	81.95%	87.28%	83.57%
CIFAR-10	Baseline	90.67%	48.82%	1.03%	37.87%
	Non-local-means sanitizer	88.24%	62.93%	72.37%	58.28%
	Median-filtering sanitizer	89.18%	65.78%	76.86%	60.03%
	Mask sanitizer	89.26%	76.18%	80.21%	78.34%
ImageNet	Baseline	83.29%	10.23%	0.21%	9.89%
	Non-local-means sanitizer	82.56%	57.27%	69.04%	59.83%
	Median-filtering sanitizer	83.02%	55.49%	67.35%	62.45%
	Mask sanitizer	82.20%	67.37%	73.84%	69.25%

On the MNIST dataset, MNIST-CNN has a baseline accuracy of 98.65% on legitimate examples. On adversarial examples generated by FGSM, C&W and PGD, respectively, the accuracy of MNIST-CNN is 70.53%, 2.34% and 53.47%, respectively. The non-local-means sanitizer improves the accuracy of MNIST-CNN to 75.84%, 80.27% and 76.28% on adversarial examples generated by FGSM, C&W and PGD, respectively. The median-filtering sanitizer improves the accuracy of MNIST-CNN to 75.27%, 82.94% and 77.32% on adversarial examples generated by FGSM, C&W and PGD, respectively. The mask sanitizer improves the accuracy of MNIST-CNN to 81.95%, 87.28% and 83.57% on adversarial examples generated by FGSM, C&W and PGD, respectively.

We can see that our sanitizing approach has the best performance on defending against C&W attack. For example, the mask sanitizer improves the accuracy of MNIST-CNN by 84.94% from 2.34% to 87.28% on C&W attack. All the three sanitizers can improve the adversarial robustness of MNIST-CNN. The mask sanitizer performs the best among the three ones. Note that the sanitizing approach only has a minor impact on the legitimate accuracy. For example, the mask sanitizer just reduces the accuracy of MNIST-CNN from 98.65% to 96.24% on legitimate examples, which is completely acceptable.

The evaluation results on CIFAR-10 and ImageNet show concordance with those on MNIST. All of the three sanitizers can improve the adversarial robustness of networks, of which the mask sanitizer performs the best. On the CIFAR-10 dataset, the mask sanitizer achieves an accuracy of 76.18%, 80.21% and 78.34% on adversarial examples generated by FGSM, C&W and PGD, respectively. It reduces the legitimate accuracy from 90.67% to 89.26%. On the tiny ImageNet dataset, the mask sanitizer achieves an accuracy of 67.37%, 73.84% and 69.25% on adversarial examples generated by FGSM, C&W and PGD, respectively. It reduces the legitimate accuracy from 83.29% to 82.20%.

As we know, defending measures usually bring extra cost to the normal computation of neural networks. In experimenting we find that the non-local-means sanitizer brings about 5% extra overhead. The computational complexity of the median-filtering sanitizer and the mask sanitizer is very low. They bring about 0.2% extra overhead.

5.3 Comparisons with state-of-the-art defenses

We compare our sanitizing approach with three state-of-the-art defense techniques: PGD-based adversarial training [7], PixelDefend [14] and feature squeezing [21]. The PGD-based adversarial training adopts the PGD attack to generate adversarial examples and constitutes the current state-of-the-art in adversarial training. The PixelDefend purifies the perturbed adversarial examples. The feature squeezing reduces the color bit depth of each pixel to help the network recognize adversarial examples.

The comparison results on the MNIST dataset are shown in Fig. 3. We can see that the PGD-based adversarial training shows the best performance on defending against FGSM attack (with an accuracy of 87.62%) and PGD attack (with an accuracy of 89.60%). However, it is the least effective in defending against C&W attack (with an accuracy of 3.86%). Such results are highly consistent with the essence of the adversarial training. We know the adversarial training can only defend against attacks that are of the same type as the attack it uses to generate adversarial examples. The featuring squeezing performs worst overall, with an accuracy of 28.42%, 75.25% and 25.39% on FGSM, C&W and PGD, respectively; followed by the PixelDefend, with an accuracy of 73.40%, 85.49% and 75.74% on FGSM, C&W and PGD, respectively. Our mask sanitizer performs best overall. It achieves an accuracy of 81.95%, 87.28% and 83.57% on adversarial examples generated by FGSM, C&W and PGD, respectively.

Fig. 3

Comparison results on MNIST.

The comparison results on CIFAR-10 and ImageNet are shown in Figs. 4 and 5. We can see that whether on the CIFAR-10 dataset or on the ImageNet dataset, our mask sanitizer generally outperforms PGD-based adversarial training, PixelDefend and feature squeezing. Our sanitizing approach is a more generalized approach that can defend against different kinds of attacks.

Fig. 4

Comparison results on CIFAR-10.

Fig. 5

Comparison results on ImageNet.

5.4 Subtle changes to hidden activations

To explain why the sanitizing approach can help the network correctly recognize adversarial examples, we carry out a case study on MNIST-CNN. Similar to Section 3, we feed the network with a legitimate “0” image and an adversarial “0” image, respectively, and then observe the changes to hidden activations. The difference is that this time the model is a sanitized MNIST-CNN, which is sanitized using the mask sanitizer. The hidden activations are shown in Fig. 6. We can see that on the sanitized MNIST-CNN, distinctions that are sharp in hidden activations on un-sanitized MNIST-CNN have now become unobvious. Especially for the last two layers FC7 and FC8, the hidden activations of the adversarial “0” image tend to be consistent with those of the legitimate “0” image.

Fig. 6

Hidden activations of the sanitized MNIST-CNN by feeding the model with a legitimate “0” image and an adversarial “0” image, respectively. Both the legitimate “0” image (generated by PGD attack) and the adversarial “0” image can be correctly recognized by the network. The sanitizing operations are implemented on CONV1, CONV2, CONV4, CONV5, FC7 and FC8. For convolutional and max-pooling layers, the feature maps for each pair examples are from the same channels of hidden activations. For fully connected layers FC7 and FC8, the activation vector is reshaped to a matrix with size 20×10.

Our sanitizing approach can eliminate the adversarial perturbations in hidden activations, which helps the network correctly recognize the adversarial examples. Please note that the hidden activation images in Fig. 6 are not obtained by sanitizing each hidden activation image in Fig. 1. That is because the network is a “dynamic” model. The hidden activation of the former layer is first sanitized, and then input to the next layer for recalculation. So the next sanitizing operation is implemented on a new hidden activation.

For quantitative analysis, we define a layer-wise distance between hidden activations of adversarial and legitimate examples. The distance is a normalized Euclidean distance and is computed as: $ℓ_{2, N or} (a^{adv}, a^{leg}) = \sqrt{\sum {[Nor (a^{adv}) - Nor (a^{leg})]}^{2}}$ (9) where Nor (a) = (a - a_min)/ - (a_max - a_min), a^adv denotes the hidden activation of adversarial examples and a^leg denotes the hidden activation of legitimate examples.

We randomly select 100 MNIST legitimate examples and the corresponding adversarial examples (generated by PGD attack). Then we compute the average distances on different layers of MNIST-CNN before and after applying sanitizing operations. The results are shown in Fig. 7. We can see that the distances between adversarial examples and legitimate examples increase layer by layer before sanitizing. However, after sanitizing, the distances are effectively reduced. This further demonstrates that our sanitizing approach has the effect of making adversarial examples behave consistent with legitimate examples.

Fig. 7

Normalized Euclidean distances between hidden activations of adversarial and legitimate examples on different layers of MNIST-CNN.

6 Conclusion

We proposed an approach, termed as sanitizing hidden activations, to improve the adversarial robustness of convolutional neural networks. The sanitizing approach is designed to eliminate or reduce the perturbations in hidden activations to help the network correctly recognize the adversarial examples. Specifically, we designed three types of sanitizers: the non-local-means sanitizer, the median-filtering sanitizer and the mask sanitizer. We systematically evaluated the effectiveness of these sanitizers on MNIST, CIFAR-10 and ImageNet datasets. On MNIST dataset, our best sanitizer achieves accuracies of 81.95%, 87.28% and 83.57% on adversarial examples generated by FGSM, C&W and PGD, respectively. On CIFAR-10 dataset, our best sanitizer achieves accuracies of 76.18%, 80.21% and 78.34% on adversarial examples generated by FGSM, C&W and PGD, respectively. On ImageNet dataset, our best sanitizer achieves accuracies of 67.37%, 73.84% and 69.25% on adversarial examples generated by FGSM, C&W and PGD, respectively. The experimental results show that our proposed sanitizing approach can effectively improve the adversarial robustness of convolutional neural networks.

The limitations of our work mainly lie in that the sanitizing approach we proposed is designed to be implemented on convolutional layers or fully connected layers in convolutional neural networks. However, it is a hyperparameter optimization problem to decide which layers to be sanitized to achieve the best effect. In practice, it is time-consuming to find the optimal solution, but it is also an inevitable task in machine learning.

In future work, we will concentrate on designing more effective sanitizing approaches, researching new techniques for defending against adversarial examples and explaining the reason for the existence of adversarial examples.

Footnotes

References

Szegedy

, Zaremba

, Sutskever

, Bruna

, Erhan

, Goodfellow

, Fergus

, Intriguing properties of neural networks, in: Proceedings of the 2nd International Conference on Learning Representations, 2014.

Heaven

, DEEP TROUBLE FOR DEEP LEARNING, Nature 574 (2019), 163–166.

Goodfellow

I.J.

, Shlens

, Szegedy

, Explaining and Harnessing Adversarial Examples, in: Proceedings of the 3rd International Conference on Learning Representations, 2015.

Kurakin

, Goodfellow

, Bengio

, Adversarial examples in the physical world, in: Workshop Track Proceedings of the 5th International Conference on Learning Representations, 2017.

Dong

, Liao

, Pang

, Su

, Zhu

, Hu

, Li

, Boosting Adversarial Attacks with Momentum, in: Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 9185–9193.

Carlini

, Wagner

, Towards Evaluating the Robustness of Neural Networks, Proceedings of 2017 IEEE Symposium on Security and Privacy, 2017, pp. 39–57.

Madry

, Makelov

, Schmidt

, Tsipras

, Vladu

, Towards Deep Learning Models Resistant to Adversarial Attacks, in: Proceedings of the 6th International Conference on Learning Representations, 2018.

Kurakin

, Goodfellow

, Bengio

, Adversarial Machine Learning at Scale, in: Proceedings of the 5th International Conference on Learning Representations, 2017.

Tramer

, Kurakin

, Papernot

, Goodfellow

, Boneh

, McDaniel

, Ensemble Adversarial Training: Attacks and Defenses, in: Proceedings of the 6th International Conference on Learning Representations, 2018.

10.

Lyu

, Huang

, Liang

H.-N.

, A Unified Gradient Regularization Family for Adversarial Examples, in: Proceedings of 2015 IEEE International Conference on Data Mining, 2015, pp. 301–309.

11.

Papernot

, McDaniel

, Wu

, Jha

, Swami

, Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks, in: Proceedings of 2016 IEEE Symposium on Security and Privacy, 2015, pp. 582–597.

12.

Ros

A.S.

, Doshi-Velez

, Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing Their Input Gradients, in: Proceedings of the 32nd AAAI Conference on Artificial Intelligence, 2018, pp. 1660–1669.

13.

Xie

, Wang

, Zhang

, Ren

, Yuille

, Mitigating Adversarial Effects Through Randomization, in: Proceedings of the 6th International Conference on Learning Representations, 2018.

14.

Song

, Kim

, Nowozin

, Ermon

, Kushman

, PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples, in: Proceedings of the 6th International Conference on Learning Representations, 2018.

15.

Buckman

, Roy

, Raffel

, Goodfellow

, Thermometer Encoding: One Hot Way to Resist Adversarial Examples, in: Proceedings of the 6th International Conference on Learning Representations, 2018.

16.

Guo

, Rana

, Cisse

, van der Maaten

, Countering Adversarial Images using Input Transformations, in: Proceedings of the 6th International Conference on Learning Representations, 2018.

17.

Bafna

, Murtagh

, Vyas

, Thwarting Adversarial Examples: An L_0-Robust Sparse Fourier Transform, in: Proceedings of Annual Conference on Neural Information Processing Systems 2018, 2018, pp. 10096–10106.

18.

Meng

, Chen

, MagNet: a Two-Pronged Defense against Adversarial Examples, in: Proceedings of the 2017 ACM Conference on Computer and Communications Security, 2017, pp. 135–147.

19.

, Li

, Adversarial Examples Detection in Deep Networks with Convolutional Filter Statistics, in: IEEE International Conference on Computer Vision, 2017, pp. 5775–5783.

20.

, Li

, Wang

, Erfani

S.M.

, Wijewickrema

S.N.R.

, Schoenebeck

, Song

, Houle

M.E.

, Bailey

, Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality, in: Proceedings of the 6th International Conference on Learning Representations, 2018.

21.

, Evans

, Qi

, Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks, in: Proceedings of 2018 Network and Distributed System Security Symposium, 2018.

22.

Roth

, Kilcher

, Hofmann

, The Odds are Odd: A Statistical Test for Detecting Adversarial Examples, in: Proceedings of the 36th International Conference on Machine Learning, 2019, pp. 5498–5507.

23.

, Bradshaw

, Sharma

, Are Generative Classifiers More Robust to Adversarial Attacks? in: Proceedings of the 36th International Conference on Machine Learning, 2019, pp. 3804–3814.

24.

Yin

, Kolouri

, Rohd

G.K.

, Adversarial example detection and classification with asymmetrical adversarial training, in: Proceedings of the 37th International Conference on Machine Learning, 2020.

25.

Gao

, Wang

, Lin

, Xu

, Qi

, DeepCloak: Masking Deep Neural Network Models for Robustness Against Adversarial Samples, in: Workshop Track Proceedings of the 5th International Conference on Learning Representations, 2017.

26.

Cao

, Gong

N.Z.

, Mitigating Evasion Attacks to Deep Neural Networks via Region-based Classification, in: Proceedings of the 33rd Annual Computer Security Applications Conference, 2017, pp. 278–287.

27.

Pang

, Xu

, Du

, Chen

, Zhu

, Improving Adversarial Robustness via Promoting Ensemble Diversity, in: Proceedings of the 36th International Conference on Machine Learning, 2019, pp. 4970–4979.

28.

Verma

, Swami

, Error Correcting Output Codes Improve Probability Estimation and Adversarial Robustness of Deep Neural Networks, in: Proceedings of Annual Conference on Neural Information Processing Systems 2019, 2019, pp. 8643–8653.

29.

Xiao

, Zhong

, Zheng

, Enhancing Adversarial Defense by k-Winners-Take-All, in: Proceedings of the 37th International Conference on Machine Learning, 2020.

30.

Carlini

, Wagner

, Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods, in: Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, 2017, pp. 3–14.

31.

Buades

, Coll

, Morel

J.-M.

, A Non-Local Algorithm for Image Denoising, in: Proceedings of 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005, pp. 60–65.

32.

Springenberg

J.T.

, Dosovitskiy

, Brox

, Riedmiller

M.A.

, Striving for Simplicity: The All Convolutional Net, in: Proceedings of the 3rd International Conference on Learning Representations, 2015.

33.

, Zhang

, Ren

, Sun

, Deep Residual Learning for Image Recognition, in: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.