CAMGAN: Combining attention mechanism generative adversarial networks for cartoon face style transfer

Abstract

In this paper, we presents an apporch for real-world human face close-up images cartoonization. We use generative adversarial network combined with an attention mechanism to convert real-world face pictures and cartoon-style images as unpaired data sets. At present, the image-to-image translation model has been able to successfully transfer style and content. However, some problems still exist in the task of cartoonizing human faces:Hunman face has many details, and the content of the image is easy to lose details after the image is translated. the quality of the image generated by the model is defective. The model in this paper uses the generative adversarial network combined with the attention mechanism, and proposes a new generative adversarial network combined with the attention mechanism to deal with these problems. The channel attention mechanism is embedded between the upper and lower sampling layers of the generator network, to avoid increasing the complexity of the model while conveying the complete details of the underlying information. After comparing the experimental results of FID, PSNR, MSE three indicators and the size of the model parameters, the new model network proposed in this paper avoids the complexity of the model while achieving a good balance in the conversion task of style and content.

Keywords

Generative adversarial networks attention mechanism style transfer image cartoonization

1 Introduction

Image style transfer refers to the transfer of the style and contour features of one image to another image while ensuring that the basic structure of the original picture does not change. The ultimate goal is to achieve a perfect combination of generated image style content and source image content.

Image style transfer can be used in the fields of artistic creation, film and television special effects, etc. Besides, it is a research hotspot and difficulty in the current image translation field. With the rapid development of the mobile internet, unique style pictures are created through style transfer and shared on social networking sites, which are very popular among people. As one of the more popular art forms in present society, cartoon animation has appeared in every aspect of our lives. The production process of modern cartoon animation is usually that artists use various sources from real world scenes to create content while adding their own artistic characteristics, this cartoon drawing method is usually more common in Japanese cartoon works, the artist artistically processed real-world photos into usable cartoon scene materials, and some famous animation scenes were created. In the real world, the transition from a real scene to a cartoon scene requires a cartoonist to carefully sculpt one by one, and usually requires a lot of energy and time to complete a scene. In deep learning, we can use the model to cartoonize images of real scenes by learning the style of specific cartoon style images. This process is called image cartoonization, the cartoonization of images can bring a variety of artistic materials to practitioners, and assist humans in their work to a certain extent, reducing people’s workload and resource consumption.

The pioneering work of style transfer between images began with Gatys et al. [1]. Traditional non-parametric image style transfer methods are mainly based on physical model rendering and texture synthesis. The non-parametric image style transfer method can only extract the low-level features of the image, not the high-level abstract features. When dealing with images with more complex colors and textures, the final image synthesis effect is rough and difficult to meet actual needs [1 –4].In response to this problem, relevant personnel have conducted a lot of research and proposed methods based on convolutional neural networks [5 –8] or generative adversarial networks [9–11 , 37], and achieved good results in certain specific scenarios. However, due to the complexity and diversity of human faces, when the above-mentioned work performs cartoon style conversion on human face images, there will be insufficient extraction of abstract details. As a result, the converted pictures appear to have mismatches in style and content that is too roughand noisy.

The goal of this paper is to propose a generative adversarial network model that combines an attention mechanism, by giving the style of the cartoon character avatar picture, the cartoon style conversion of the unmatched real scene character avatar is performed, after the conversion, the generated image should maintain a certain balance between content and style, and achieve the task goal of consistent style, clear content and complete details.

In order to achieve the goal of this paper, which is to maintain a balance between style and content. while considering the completeness of face detail extraction, two changes have been made to the current model:First of all, this article uses smaller convolution kernel in the generator to process the image to ensure the image quality after the style conversion. Secondly, in order to ensure that the model can learn more style details and abstract details of style content, this paper adds a channel attention mechanism between the up-sampling stage of the generator, so that the model can learn more complete style and content details, and maintain the consistency of the generated graphs in style and content.

The paper has the following contributions:

First of all, this paper proposes a new generative adversarial network that combines the attention mechanism, which achieves a good balance between style and content in the cartoon face style transfer work. Secondly, the model in this paper does not increase the complexity of the model while ensuring that the quality of the generated cartoon images is better than the former model.

2 Related work

2.1 Attention mechanism

Attention Model(AM), first introduced for Machine Translation [13] has now become a predominant concept in neural network literature [14], attention has become enormously popular within the Artificial Intelligence(AI) community as an essential component of neural architectures for a remarkably large number of applications in Natural Language Processing [15] and Computer Vision [16]. With the deepening of research, the attention mechanism is divided into hard attention mechanism, soft attention mechanism, global attention mechanism and local attention mechanism, etc [17 –19]. The proposal of visual attention has made it popular in many computer vision tasks [20], aiming at the generative adver network in the field of computer vision. Zhang et al. proposed a self-attention mechanism generative adversarial network. Applying the self-attention mechanism to the generative adversarial network helps to capture remote dependencies [21]. Aiming at the complexity of the attention mechanism model, the channel attention mechanism proposed by Wang et al. is more effective while reducing the complexity of the model [22].

2.2 Generative adversarial network

Generative adversarial network [23] was first proposed by Goodfellow, the network model is composed of a generator and a discriminator, the game theory is adopted to train the entire model through the mutual game between the generator and the discriminator. The task of the generator is to generate a distribution that is as highly similar to the real distribution as possible, and the task of the discriminator is to determine whether the distribution generated by the generator is the real distribution or the generator generated. In the ideal state, the discriminator cannot distinguish between the generated picture and the real picture, and it reaches the optimal state of the network model, which is the Nash equilibrium point, but this point is difficult to achieve so far. As the most popular generative model network at present, it was originally proposed to solve the problem of sample acquisition difficulty in deep learning mainly for sample generation. After previous research on GAN network, its main function is to focus on image segmentation and data Enhancement, semantic segmentation, etc. The CycleGAN [24] and pix2pix [25] proposed by Jun-Yan Zhu et al. introduced the image style transfer problem on the basis of the generative adversarial network, using two pairs of GAN networks, as well as the adversarial loss function and the cycle consistency loss function to ensure the stability of training and image arrival. Style conversion between images, and CycleGAN does not require paired data sets, but the generated images have the disadvantage of being unclear. Pix2pixHD proposes a high-resolution solution on the basis of pix2pix, but it comes with resource consumption. On the basis of CycleGAN, GANILLA [26] proposed its own improvement to the generator network to complete the transfer of cartoon illustration style, but the effect on the task of cartoonization of portraits is not very good, and the balance of the converted image content and style is poor. The proposal of CGAN [27] allows GAN to use images and corresponding labels for training, and use the given labels to generate specific real-detail change images in the test phase, but it is also different from the research content of thisarticle.

Stargan and stylegan [28, 29] solves the problem of multi-domain image content conversion. The two models are mainly aimed at the detailed multi-domain style conversion of real-life images, and the image can be changed in terms of detailed features through a given label. CartoonGAN [30] can use real pictures to generate any cartoon-style pictures based on the input specific style pictures, but there will be artifacts when generating cartoon pictures, which is more obvious in the case of more green content. Secondly, it differs from the task of cartoonizing faces in this article in style and content. Compared with the former, AnimeGAN [31] proposes a new loss function, which improves the cartoon effect of the generated image. However, some of the generated pictures still have some phenomena such as loss of details of distant portraits and unobvious cartoon styles of close-range characters. White-boxGAN [32] will have spots when the cartoon style is changed. In this paper, a generative adversarial network structure is used to synthesize images with the same distribution as the target domain.

2.3 Image-to-image translation

The image-to-image translation is similar to the machine translation task in the natural language field. The task of machine translation is the conversion from text to text, such as the conversion from Chinese to English. However, the task of image-to-image translation is to convert a picture from one form of expression to another. The fundamental relationship in the field of image-to-image translation is actually the pixel-to-pixel relationship in the image.

The task of image-to-image translation [33, 34] solves the problem of image conversion from a source domain to another target domain. Image enhancement [35], rain and fog removal [36], image style transfer, season conversion, and coloring of hand-drawn manuscripts are all applied in the field of image-to-image translation tasks.

This paper uses an unpaired image-to-image conversion framework to complete the task of cartoon style transfer of real-life images, combined with the attention mechanism, so that the converted images have a good balance of content and style.

3 CAMGAN

In the study of the problem of face cartoon style conversion task, it was found that the former’s work had some problems[gan, Cyclegan, pix2pix, ganilla]. In previous studies, when dealing with face images with more detailed and more complex data, it is difficult for the style transfer results to achieve satisfactory results in the balance between content and style. The background content color changes, the content details are not learned enough, the content structure changes, and the problems of poor quality of the generated images will more or less exist. In response to the above problems, a new generator network is designed, which combines the channel attention mechanism on the basis of the predecessors, which can deliver styles while retaining the content. First of all, we will introduce the model in detail. Secondly, we will compare the model with previous work on cartoon face style transfer. Finally, we will conduct related parameter experiments on the model to verify the effectiveness of our model.

3.1 Model details

The CAMGAN model is a GAN-based model that proposes a new generator network combined with channel attention mechanism, as shown in Fig. 2. In order to ensure the completeness of the details of the image content, CAMGAN uses low-level features to save the content details when transferring image styles. CAMGAN’s generator model includes two stages: down-sampling stage and up-sampling stage. In the down-sampling stage, we use a modified ResNet-18 [37] network. Since the bottom layer contains information such as the morphological characteristics, edges and shapes of the image, this is very important information in the image cartoonization task. Therefore, in the down-sampling stage, we connect each down-sampled layer with the feature information of the previous layer to ensure the integrity and consistency of the content information of the image during transmission.

Fig. 1

Use CAMGAN model for cartoon style transfer.

Fig. 2

CAMGAN generator structure combined with channel attention mechanism.

First, here we combine a 7x7 kernel convolutional layer with instance norm [38] and max pooling layer. The reason for choosing to combine with in here is that when using the GAN model for style transfer, the effect of instance norm is better than that of batch norm. Then we use the two remaining blocks as a layer, a total of four layers and eight small layers to transfer and process image information. Each remaining block is similar to the above, starting with a convolutional layer, and then connecting in and ReLU to maintain the integrity of the underlying information and the stability of the network. Then pass the information to the next convolutional layer and instance norm, using a concat layer and convolutional layer as the end, and the output of the previous remaining block as the input of the next block. Taking into account the completeness of the extraction of the content details, we will use the convolution with a stride of 2 for each layer of feature maps after the first layer to halve, and the convolution kernel of all remaining blocks is 3x3.

Fig. 3

Combined with global average pooling (GAP) to obtain the channel attention mechanism of aggregated features, H, W, C represent the spatial size of the image.

Before the operation of the up-sampling stage, this article adds a channel attention mechanism with an adaptive core (ECA-NET) between the up-sampling and the up-sampling (show in Fig. 3). The size of the adaptive core can be adaptively changed according to the size of the channel dimension C. According to the experimental results, the model combined with the channel attention mechanism is helpful for effective channel attention learning while avoiding the reduction of dimensionality [39]. In the channel attention mechanism, a band matrix with KxC parameters that is not completely isolated between groups is used to learn channel attention as shown below Eq.(1). ${[\begin{matrix} ω^{1, 1} & \dots & ω^{1, k} & 0 & 0 & \dots & \dots & 0 \\ 0 & ω^{2, 2} & \dots & ω^{2, k + 1} & 0 & \dots & \dots & 0 \\ ⋮ & ⋮ & ⋮ & ⋮ & ⋱ & ⋮ & ⋮ & ⋮ \\ 0 & \dots & 0 & 0 & \dots & ω^{C, C - K + 1} & \dots & ω^{C, C} \end{matrix}]}_{,}$ (1) For the calculation of the weight, use Eq.(2): $ω_{i} = σ (\sum_{n = 1}^{k} ω_{i}^{j} Y_{i}^{j}), y_{i}^{j} ε Ω_{i}^{k},$ (2) Where $Ω_{i}^{k}$ represents the set of channels adjacent to $Y_{i}$ . We can also use a more effective way to let all channels share the same learning parameter Eq.(3). $ω_{i} = σ (\sum_{n = 1}^{k} ω^{j} Y_{i}^{j}), y_{i}^{j} ε Ω_{i}^{k},$ (3) For this method, we can use 1D convolution with a kernel size of k to achieve (C1D stands for 1D convolution), Eq.(4): $ω = σ (C 1 D_{K} (Y)),$ (4) The channel attention mechanism captures local cross-channel interactions to process and transfer information, ensuring the complete transfer of model details.

In the down-sampling stage, we use skip connections and longer connections to pass lower-level features to the output layer, which helps to preserve low-level feature information. This is a very important step to improve the content quality of the generated image. In the upsampling stage, the main structure is convolution combined with the upsampling layer and the summation layer, and the output processed by the downsampling layer and the channel attention mechanism is used as the input. At the same time, the size of the feature map is increased in the upsampling layer. To match the size of the feature map of the previous layer. After successive up-sampling and summation layers of convolutional layers, CAMGAN finally outputs a 3x3 channel style transfer image through a 7x7 convolution.

Compared with the traditional GAN network, in order to ensure the quality of the generated pictures, we use PatchGAN [25] in the discriminator part to identify the generated images. The traditional GAN model discriminator maps the input to a real number to represent the probability that the generated sample is a real sample. Compared with the traditional GAN model discriminator, patchgan is completely composed of convolutional layers, and then probabilistic judgment is made for each NxN color block in the image, and the final output is an NxN matrix. Each element X_ij of this NxN matrix represents the probability that each patch in the original image is a true sample. Then each element of the output matrix is averaged as the output of the discriminator to determine whether the result is true or false. Compared with the traditional GAN discriminator model, patchgan considers the influence of different parts of the image in the output result and makes a decision based on many factors. In most studies in recent years, it can be found that the resolution and detail requirements of images in computer vision research tasks are getting higher and higher. Therefore, the birth of patchgan helps the model to pay more attention to details in training. In CAMGAN,we use a 70x70 patchgan, and the filter sizes are 64, 128, and 256 respectively.

The CAMGAN model still uses CycleGAN’s training methods and ideas. CAMGAN and CycleGAN have two generators and two discriminators, one generator and one discriminator are a set, the model goal is to learn the mapping function between domain X and domain Y, as shown in Fig. 4(a). The mapping function learned by CAMGAN should be cyclically consistent: for each image in the domain X after the cyclic conversion, the original image that is the same as X should be obtained as much as possible. For example, X → G → F (G (X)) ≈ X, as shown in Fig. 4(b). For domain Y, the rule of Y → F → G (F (Y)) ≈ Y should also be followed, as shown in Fig. 4(c).

Fig. 4

CAMGAN training method.

In terms of the loss function, in addition to the maximum and minimum loss of each generator and discriminator as shown in Eq.(5). $\begin{matrix} min_{G} max_{D} V (D, G) & = E_{X} \sim P_{data (X)} [log D (X)] \\ + E_{X \sim P_{Z} (Z)} [log (1 - D (G (Z)))], \end{matrix}$ (5)

In addition, in order to ensure the quality of the image generation, the consistency of the image structure, and the stability of training, we also use a cyclic consistency loss function and an adversarial loss function such as Eq.(6) and formula Eq.(7). $\begin{matrix} L_{GAN} (G, & D_{Y}, X, Y) = E_{y} \sim {Pdata}_{(y)} [log D_{Y} (y)] \\ + E_{x} \sim {Pdata}_{(x)} [log (1 - D_{Y} (G (x)))], \end{matrix}$ (6) $\begin{matrix} L_{cyc} (G, F) & = E_{x} \sim {Pdata}_{(x)} [‖ F (G (x)) - x ‖_{1}] \\ + E_{y} \sim {Pdata}_{(y)} [‖ G (F (y)) - y ‖_{1}], \end{matrix}$ (7)

Regarding the cyclic consistency loss, we use the L1 distance, which is the Manhattan distance. The idea of cyclic consistency comes from dual learning. The function of cyclic consistency loss function is to control image generation at the pixel level, reduce the mapping space, and preserve the contour of the input image better. Then through the introduction of L1 distance, the optimization process takes into account the changes of the image pixel level. The goal of the model is to optimize all losses, as shown in Eq.(8).It is especially important that the value of the parameter λ in the cyclic consistency loss function plays a decisive role in the entire loss function. $\begin{matrix} L (G, F, D_{X}, D_{Y}) & = L_{GAN} (G, D_{Y}, X, Y) \\ + L_{GAN} (F, D_{X}, Y, X) \\ + λ L_{cyc} (G, F) . \end{matrix}$ (8)

Considering the complexity of the model implementation, this article uses PyTorch to implement the model. In deep learning tasks, especially in the field of image translation, the situation of having paired data sets does not necessarily exist, and having paired data sets often requires a certain degree of complexity. When CAMGAN performs image style conversion, it does not need paired images, only two datasets with different styles, one for the source domain and the other for the target domain. According to the research content and goals of this article, the character image of the real scene is used as the source domain, and the cartoon style image is used as the target domain to obtain the cartoon style. The images used in all models are adjusted to 256x256 pixels. The model is trained using the Adam optimizer with a learning rate of 2x10-4, and 200 epochs are one model training. After discussion, when the number of data sets is determined to be sufficient, the pre-training method is not used to pre-train the model weights. All experiments start from scratch. For the experimental equipment, we use RTX3070 for modeltraining.

4 Experiment

4.1 Dataset

This paper uses unpaired data sets for model training, which can better reflect the performance of the model in the task of image cartoon style transfer. The real scene character data set comes from the FFHQ data set [40], from which 10,000 portrait images of various ages and genders with diverse real scenes are collected as the source domain. For the acquisition of target domain images, 10,000 cartoon portrait images from the white-boxgan work of Wang et al. were used. During the training process, the resolution of all images is 256x256.

4.2 Evaluation metrics

This paper uses the above data set to perform cartoon image style conversion and compares with the results of CycleGAN and ganilla. Four indicators of FID [41], average psnr, average ssim, and average mse are used to evaluate the performance of the model. The evaluation is carried out to prove that the performance of CAMGAN in the task of portrait cartoon style transfer is better than the previous two as shown in Table 1.

Table 1
Comparison of fid, average psnr, ssim, and mse indicators

Methods CycleGAN GANILLA CAMGAN

FID 183.15 145.95 137.00

PSNR 18.39 18.38 18.40

SSIM 0.72 0.71 0.72

MSE 0.018 0.018 0.017

Methods	CycleGAN	GANILLA	CAMGAN
FID	183.15	145.95	137.00
PSNR	18.39	18.38	18.40
SSIM	0.72	0.71	0.72
MSE	0.018	0.018	0.017

4.3 Experimental result

CycleGAN and GANILLA both use two pairs of generator discriminator networks. The structure of the two generators is similar. The latter improves the generator network on the basis of it, so that the network model has a better balance in the conversion of cartoon illustrations. The model in this paper combines the channel attention mechanism on the basis of the first two. From the comparison of the model parameters, it can be seen from Table 2. The channel attention mechanism does not increase the complexity of the model while delivering more complete context information and achieves better results. After experimental verification, the performance of the model in this paper is better than the previous two in the task of cartoonizing portraits, as shown in Fig. 5.

Table 2
Model parameter size comparison

Network G_A G_B D_A D_B Total

CAMGAN 7.228M 7.228M 2.765M 2.765M 19.986M

CycleGAN 11.379M 11.379M 2.765M 2.765M 28.288M

GANILLA 7.228M 7.228M 2.765M 2.765M 19.986M

Network	G_A	G_B	D_A	D_B	Total
CAMGAN	7.228M	7.228M	2.765M	2.765M	19.986M
CycleGAN	11.379M	11.379M	2.765M	2.765M	28.288M
GANILLA	7.228M	7.228M	2.765M	2.765M	19.986M

Fig. 5

Comparison of camgan, cyclegan and ganilla generation effects.

5 Conclusion

In this paper, we propose a new generative adversarial network model CAMGAN that combines the channel attention mechanism. The goal of the model task is to convert real-world character images into cartoon-style character images. The main contributions of this paper are as follows: A new combination of channel attention mechanism is proposed to generate a confrontation network, while retaining the content of the source domain image, the animation style conversion is carried out to achieve a certain balance between the two. Experiments have shown that CAMGAN can convert portrait photos of real people into animation-style images. Compared with the previous model, combined with the channel attention mechanism CAMGAN, the model complexity has not increased, and the image after CAMGAN conversion is in Image content, background color and image details are better than the former. In future work, in order to obtain better quality generated images, we plan to conduct research to generate better image quality generative confrontation networks, and also consider applying animation style conversion to video frame-by-frame conversion.

Footnotes

Acknowledgments

This work is supported by Key Program of National Natural Science Foundation of China, Grant/Award Number: U2003208; National Natural Science Foundation of China, Grant/Award Number: 61962057; Major science and technology projects of autonomous region, Grant/Award Number: 2020A03004-4.

References

Gatys

L.A.

, Ecker

A.S.

and Bethge

, A neural algorithm of artistic style [J], arXiv preprint arXiv: 1508. 06576, 2015.

Gatys

L.A.

, Bethge

, Hertzmann

, et al., Preserving color in neuralartistic style transfer [J], arXiv preprint arXiv: 1606. 05897, 2016.

Novak

and Nikulin

, Improving the neural algorithm of artistic style [J], arXiv preprint arXiv: 1605. 04603, 2016.

Rujie

, Content aware neural style transfer [J], arXiv preprint arXiv:1601. 04568, 2016.

Ulyanov

, Lebedev

, Vedaldi

, Lempitsky

V.S.

, Texture networks: Feed-forward synthesis of textures and stylized images, in: International Conference on Machine Learning, 2016, pp. 1349–1357.

Johnson

, Alahi

, Fei-Fei

, Perceptual losses for real-time style transfer and super-resolution, in: IEEE European Conference on Computer Vision, Springer, 2016, pp. 694–711.

, Wand

, Precomputed real-time texture synthesis with markovian generative adversarial networks, in: IEEE European Conference on Computer Vision, Springer, 2016, pp. 702–716.

, Fang

, Yang

, Wang

, Lu

and Yang

M.-H.

, Diversified texture synthesis with feed-forward networks, in: IEEE Conference on Computer Vision and Pattern Recognition, 2017.

Choi

, Choi

, Kim

, Ha

J.-W.

, Kim

, Choo

, Stargan: Unified genera- tive adversarial networks for multi-domain image-to-image translation, in: IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 8789–8797 .

10.

Karacan

, Akata

, Erdem

and Erdem

, Learning to generate images of outdoor scenes fromat tributes and semantic layouts, (2016). arXiv:1612.00215.

11.

Isola

, Zhu

J.-Y.

, Zhou

and Efros

A.A.

, Image-to-image translation with conditional adversarial networks, IEEE Conference on Computer Vision and Pattern Recognition, 2017.

12.

, Zhang

, Ren

, Sun

, Deep residual learning for image recognition, in: IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.

13.

Cho

, Courville

and Bengio

, Describing multimedia content using attention-based encoder-decoder networks, IEEE Transactions on Multimedia 17(11) (2015), 1875–1886.

14.

Chaudhari

, Polatkan

, Ramanath

, et al., An Attentive Survey of Attention Models[J], 2019.

15.

Galassi

, Lippi

and Torroni

, Attention in Natural Language Processing, IEEE Transactions on Neural Networks and Learning Systems (2020), 1–18.

16.

Wang

and David

M.J.

, Tax, Survey on the attention based RNN model and its applications in computer vision, arXiv preprint arXiv:1601.06823 (2016).

17.

Bahdanau

, Cho

and Bengio

, Neural Machine Translation by Jointly Learning to Align and Translate, In 3rd International Conference on Learning Representations, 2015.

18.

, Ba

, Kiros

, Cho

, Courville

, Salakhudinov

, Zemel

and Bengio

, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (Proceedings of Machine Learning Research, Vol. 37) (2015), 2048–2057.

19.

Luong

, Pham

and Christopher

, Manning, Effective Approaches to Attention-based Neural Machine Translation, In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2015, 1412–1421.

20.

Mnih

, Heess

, Graves

and Kavukcuoglu

, Recurrent Models of Visual Attention, In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, 2014, MIT Press, 2204–2212.

21.

Zhang

, Goodfellow

, Metaxas

and Odena

, Self-attention generative adversarial networks, In International Conference on Machine Learning, PMLR, 2019, 7354–7363.

22.

Wang

, Wu

, Zhu

, et al., ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks[C], 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2020.

23.

Goodfellow

I.J.

, Pouget-Abadie

, Mirza

, et al., Generative Adversarial Networks[J], Advances in Neural Information Processing Systems 3 (2014), 2672–2680.

24.

Zhu

J.Y.

, Park

, Isola

, et al., Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks[J], 2017.

25.

Isola

, Zhu

J.Y.

, Zhou

, et al., Image-to-Image Translation with Conditional Adversarial Networks[C], IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2016.

26.

Hicsonmez

, Samet

, Akbas

, et al., GANILLA: Generative adversarial networks for image to illustration translation[J], Image & Vision Computing 95 (2020), 103886.

27.

Hensman

and Aizawa

, cGAN-based Manga Colorization Using a Single Training Image[J], 2017.

28.

Choi

, Choi

, Kim

, et al., StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation[C], 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2018.

29.

Karras

, Laine

, Aittala

, et al., Analyzing and Improving the Image Quality of StyleGAN[J], 2019.

30.

Chen

, Lai

K.Y.

and Liu

Y.J.

, CartoonGAN: Generative Adversarial Networks for Photo Cartoonization[C], IEEE/CVF Conference on Computer Vision & Pattern Recognition, IEEE, 2018.

31.

Chen

, Liu

and Chen

, AnimeGAN: A Novel Lightweight GAN for Photo Animation[M], Artificial Intelligence Algorithms and Applications, 2020.

32.

Wang

and Yu

, Learning to Cartoonize Using White-Box Cartoon Representations[C], 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2020.

33.

Huang

, Liu

M.-Y.

, Belongie

and Kautz

, Multimodal unsupervised image-to-image translation, In Proceedings of the European Conference on Computer Vision (ECCV), pages 172–189, 2018.

34.

Isola

, Zhu

J.-Y.

, Zhou

and Aefros

, Image-to-image translation with conditional adversarial networks, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1125–1134, 2017.

35.

Ignatov

, Kobyshev

, Timofte

, Vanhoey

and Gool

L.V.

, Wespe: weakly supervised photo enhancer for digital cameras, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 691–700, 2018.

36.

Dong

, Pan

, Xiang

, et al., Multi-Scale Boosted Dehazing Network with Dense Feature Fusion[J], arXiv, 2020.

37.

, Zhang

, Ren

, Sun

, Deep residual learning for image recognition, in: IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.

38.

Ulyanov

, Vedaldi

and Lempitsky

, Instance Normalization: The Missing Ingredient for Fast Stylization, 2016.

39.

Wang

, Wu

, Zhu

, et al., ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks[C], 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2020.

40.

Karras

, Laine

and Aila

, A style-based generator architecture for generative adversarial networks, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4401–4410, 2019.

41.

Heusel

, Ramsauer

, Unterthiner

, Nessler

and Hochreiter

, Gans trained by a two time-scale update rule converge to a local nash equilibrium, In Advances in Neural Information Processing Systems, pages 6626–6637, 2017.

CAMGAN: Combining attention mechanism generative adversarial networks for cartoon face style transfer

Abstract

Keywords

1 Introduction

2 Related work

2.1 Attention mechanism

2.2 Generative adversarial network

2.3 Image-to-image translation

3 CAMGAN

3.1 Model details

4.1 Dataset

4.2 Evaluation metrics

Table 1 Comparison of fid, average psnr, ssim, and mse indicators Methods CycleGAN GANILLA CAMGAN FID 183.15 145.95 137.00 PSNR 18.39 18.38 18.40 SSIM 0.72 0.71 0.72 MSE 0.018 0.018 0.017

Table 2 Model parameter size comparison Network G_A G_B D_A D_B Total CAMGAN 7.228M 7.228M 2.765M 2.765M 19.986M CycleGAN 11.379M 11.379M 2.765M 2.765M 28.288M GANILLA 7.228M 7.228M 2.765M 2.765M 19.986M

Footnotes

Acknowledgments

References

Table 1
Comparison of fid, average psnr, ssim, and mse indicators

Methods CycleGAN GANILLA CAMGAN

FID 183.15 145.95 137.00

PSNR 18.39 18.38 18.40

SSIM 0.72 0.71 0.72

MSE 0.018 0.018 0.017

Table 2
Model parameter size comparison

Network G_A G_B D_A D_B Total

CAMGAN 7.228M 7.228M 2.765M 2.765M 19.986M

CycleGAN 11.379M 11.379M 2.765M 2.765M 28.288M

GANILLA 7.228M 7.228M 2.765M 2.765M 19.986M