Dual-task attention-guided character image generation method

Abstract

Human body pose transfer is to transform the character image from the source image pose to the target pose. In recent years, the research has achieved great success in transforming the human body pose from the source image to the target image, but it is still insufficient in the detailed texture of the generated image. To solve the above problems, a new two-stage TPIT network model is proposed to process the detailed texture of the pose-generated image. The first stage is the source image self-learning module, which extracts the source image features by learning the source image itself and further improves the appearance details of pose-generated image. The other stage is to change the pose of the figure gradually from the source image pose to the target pose. Then, by learning the feature correlation between source and target images through cross-modal attention, texture transmission between images is promoted to generate finer-grained details of the generated image. A large number of experiments show that the model has superior performance on the Market-1501 and DeepFashion datasets, especially in the quantitative and qualitative evaluation of Market-1501, which is superior to other advanced methods.

Keywords

Posture transfer self-attention mechanism dual-tasking mechanism character image generation generating adversarial networks

1 Introduction

Pose-guided human image generation model is to replace a person in a particular pose in an image with a desired pose, and then to recreate the image of the person by the desired pose, i.e. to convert the pose of a person in the image to another pose. Human Pose Transfer (HPT) is used in many applications, such as automated fashion design, film effects, creative media production, and virtual reality, among others. There is much to investigate in pose transfer, such as video generation with a series of poses [1], data expansion for person re-identification [2], and person re-identification [3].

However, factors such as rearrangement of pose appearance features, inference of occlusion regions, and picture fidelity requirements make pose transfer still very challenging. Early work using Generative Adversarial Networks (GANs) [4] and Variational Autoencoders (VAEs) [5], pose transfer has made great progress, however, the early work lacks local deformation modeling and does not handle the detailed parts of the composite image of a person well. For example, the color of the clothes is not correct or the head of the character is deformed, etc. To solve this problem and to able to preserve the appearance features and texture features of conditional character images. an improved feature fusion mechanism [6], a local distortion method, was proposed by Siarohin et al. Attention mechanisms [7,8] and optical flow [9,10] were applied to improve spatial transformation capabilities for local feature transfer, in addition to which some methods [11,12] incorporate additional labels in the model to make it more robust in handling complex poses and to efficiently synthesize realistic results, such as human pose maps, to provide semantic guidance for pose changes.

As can be seen, with the rapid development of deep learning, the pose transformation task for processing source-to-target images has evolved greatly, but the appearance of the generated images still has not made a breakthrough, and the existing methods do not achieve texture mapping between source and target images well, resulting in the blurred distorted and unrealistic appearance of the generated images.

The motion field may contain noise information the cluttered background, which cannot effectively participate in the generation of motion details in synthetic images. Therefore, depth-aware attention (learn depth-aware attention) is used to capture texture details related to the appearance of the source image, and pixel-level geometric constraints are applied to the motion field as a way to generate finer facial and appearance details.

Inspired by the ViTDet [13] method proposed by Meta, which directly uses the features of the last layer of ViT for processing to get the same performance as building FPNs, the cross-modal attention module is added at the end of the framework in this paper, and the source image learns itself to capture the image-related micro-motions and textures to improve the reasonable texture mapping between the source image to the target image by establishing texture transformation between source and target images, enhancing the details of the generated images and preserving the appearance details of the source images.

Based on the above idea, this paper proposes a dual-task attention-guided figure image generation (TPIT) model to support pose transfer image generation. It performs the source image learning itself through an SLS module, where the DCABlock module selectively performs the information collection of the source image. With the help of the cross-modal attention module, it enables the generated images to retain a more detailed appearance and texture features of the source images. In this paper, the proposed TPIT model is evaluated qualitatively and quantitatively by conducting extensive experiments on two different datasets DeepFashion [14] and Market-1501 [15]. The experiments show that the proposed TPIT model can generate generated images with higher quality, finer granularity, and better preservation of facial details of the person compared to other models.

In summary, the contributions of this paper are as follows:

A novel model (TPIT) is proposed for the pose transfer image generation task, which effectively incorporates the source image information into the generation network through two mechanisms, eliminates the image blur distortion problem through a cross-modal (i.e., key point and source image) attention mechanism, improves the texture transfer from the source image to the target image, and improves the quality of the generated images.

Experimental results show that the TPIT model also achieves good generative results compared with more advanced techniques, especially on the Market-1501 dataset, where the TPIT model shows excellent performance on SSIM and FID.

In the remaining sections, the literature and theory related to the relevant papers are reviewed in Section 2, the model approach proposed in this paper is described in Section 3, comprehensive experiments including ablation experiments are performed and results are derived in Section 4, and conclusions are given in Section 5.

2 Related work

2.1 Pose guided person image generation

In recent years, several methods have been proposed to solve the pose transfer problem. Lassner et al. [16] combined VAE [5] and GAN to generate images of different clothes under the given conditions of a 3D model of a person. Ma et al. [17] proposed a two-stage model that directly concatenates the target image pose and the source image to generate the target image in a coarse-to-fine manner. Pumarola et al. [18] proposed an unsupervised way to reconstruct the source image by using the generated result and the stitching of the source image pose as input. Essner et al. [19] combined VAE [5] and U-Net [20] to decompose the appearance and pose of a person’s image. These methods are CNN-based [41] and cannot handle the appearance misalignment problem. To deal with the appearance misalignment problem, Siarohin et al. [6] proposed a deformable GAN capable of decomposing the overall deformation by a set of local affine transforms as a way to handle the misalignment due to different poses. Zhu et al. [8] proposed to progressively transform the source image by a sequence of columnar pose attention transfer blocks (PATBs) ground to transform the source image, however, this method does not explicitly learn the spatial transformation between different poses and may lose useful information during multiple transfers, making the synthetic image under-detailed. To enhance the detail generation and texture transformation of the generated images, Li et al. [21], Ren et al. [9], and Tabejamaat et al. [10] proposed a warping operation to warp the source image features by estimating the dense optical flow and subsequently generating the images. Besides, Zhang et al. [12] and Lv et al. [11] improve the granularity of the generated images by adding semantic parsing tags, and they output improve the granularity of the generated images by adding semantic parsing tags, and they output the generated images based on the predicted target semantic parsing. However, these target semantic representations are not easily collected and may be unreliable, which can lead to the inconsistent texture of the generated images. Therefore, the TPIT model drives the generation of finer-grained structures and details through source image learning and attention mechanisms.

2.2 Generative adversarial networks

Generative adversarial networks (GANs), proposed by Goodfellow et al. [22] and others, consist of generators and discriminators and are trained adversarially to generate realistic images, and GANs have been used in a large number of studies, such as text image translation [23], image synthesis [24], and so on. In this paper, the TPIT model uses GANs for pose change.

2.3 Attention mechanism

The attention mechanism uses an efficient way of modeling context, and many researchers have used it with good results on computer vision tasks, such as target classification [25], target detection [26], image generation [27], and so on. During these years, attention learning has evolved into different types of attention, such as multi-headed attention [28], and self-attention [29]. Among them, self-attention is dealing with global information, and Brock, [30] et al. then added the self-attention layer to the model (BIGGAN) to enable the encoder and decoder to efficiently model the image regions, capture the long-range relationships, and improve the quality of the image generation. The TPIT model proposes to add the self-attention mechanism to learn the relationship between one of the pixel points and other pixel points and to generate more detailed and realistic images using the location feature drive.

3 Method

Figure 1 shows the overall framework of the method in this paper (TPIT), which uses a TPIT model consisting of 18 channel heat maps that encode the locations of 18 joints in the human body. Each modular part of the TPIT model approach is described in detail in the following sections.

Fig. 1

An overview of our methods. It mainly includes two modules, one is the conversion module from the source image to the target image, and the other is the source image self-learning module. Then, a more detailed facial structure and more accurate detailed pictures are generated through the cross-modal (i.e. key points and source images) attention module.

3.1 Source task learning module

The source image I_s will be input into the encoder En_s, and the features of the input information will be extracted with the encoder En_s, and then the pose transfer will be performed by applying a series of ResBlocks modules, and the output result after a series of ResBlocks will be the feature $F_{0}^{s}$ and the feature $F_{0}^{s}$ will be aligned with the source image pose P_s.

$F_{0}^{s}$ will go through the DCABlock module for source image feature extraction, and each DCABlock module is the same structure to input $F_{t - 1}^{s}$ into the DCABlock module, and the DCABlock first uses an MHSA module to obtain the correlation between the inputs. In addition, this DCABlock module also updates $F_{t - 1}^{s}$ to $F_{t}^{s}$ using an instance normalization module, a Multi-Layer Perceptron (MLP) module with multiple fully connected layers, respectively to increase the capacity of DCABlock: $F_{s 1} = IN [MHSA (F_{t - 1}^{s}) + F_{t - 1}^{s}]$ (1) $F_{t}^{s} = IN [F_{s 1} + MLP (F_{s 1})]$ (2)

After N DCABlock modules finally get F_d.

3.2 Posture attention transfer module

The pose attention transfer module is consisted of a series of pose transfer blocks, where the source image pose P_s and the target image pose P_t are input into the encoder together, and the encoder extracts the features of both classes as $F_{0}^{p}$ , inputs the source image into the encoder to obtain the result $F_{0}^{I}$ , and inputs $F_{0}^{p}$ and $F_{0}^{I}$ together into the pose transfer block DPATBlock module for pose conversion. The DPATBlock module has two branching structures, the $F_{0}^{p}$ and $F_{0}^{I}$ input into the DPATBlock module gradually pass through the convolution layer (Conv), normalization layer (BN), Relu [31], convolution layer (Conv), and normalization layer (BN) modules to obtain $D_{t - 1}^{p}$ and $D_{t - 1}^{I}$ , respectively, and then $D_{t - 1}^{p}$ and $D_{t - 1}^{I}$ are gradually updated to $F_{t - 1}^{p}$ and $F_{t - 1}^{I}$ , where the equations are respectively: $F_{t}^{I} = [σ ({conV}_{p} (F_{t - 1}^{p})) * {conV}_{I} (F_{t - 1}^{I})] + F_{t - 1}^{I}$ (3) $F_{t}^{p} = {conV}_{p} (F_{t - 1}^{p}) | | F_{t}^{I}$ (4) where σ is sigmoid, which maps $D_{t - 1}^{p}$ into the (0,1) range by the sigmoid function, and * denotes Element-wise multiply, which multiplies $D_{t - 1}^{I}$ and the output after going through the function.+indicates that the output after the element-wise product and $F_{t - 1}^{I}$ are added, and || indicates that the $D_{t - 1}^{p}$ and output of the element-wise sum are spliced along the depth axis.

The final $F_{t}^{p}$ and $F_{t}^{I}$ are obtained after N DPATBlock modules are updated.

3.3 Cross-modal attention module

To be able to learn the texture structure and details of the source image effectively, the TPIT model uses a cross-modal attention mechanism, i.e., keypoints and images, to generate an attention graph to guide the feature $F_{t}^{p}$ to generate the image. As shown in Fig. 1, a source image is input into the encoder and F_d is obtained by the source task learning module, and three potential feature maps F_q, F_k, and F_v are obtained by linear projection of F_d and distorted source image features F_w through three different 1×1 convolutional layers with kernels W_q, W_k, and W_v, respectively. F_q, F_k, and F_v can represent self-attentive queries, keys, and values in the mechanism. In this way, the geometry-related query features F_q generated from the source image can be fused with the appearance-related key features F_k to generate a key point bootstrap for generating feature F_g. Finally, feature F_g is input to the decoder to generate the final synthetic image I_g.

3.4 Training

The overall loss function is: $L_{s \to t} = λ_{adv} L_{adv} + λ_{L 1} L_{L 1} + λ_{per} L_{per}$ (5) where the adversarial loss of the discriminator D is used to penalize the difference in distribution between the generated image and the target image. In the work of this paper, it is also used to constrain the generated image results to be more realistic. The loss of L_adv is formulated as:

$L_{adv} = E [log (D_{A} \cdot D_{S})] + E [log (1 - D_{A} \cdot D_{S})]$ (6)

L1 denotes the calculated pixel-level L1 loss between the generated image and the target image. The sum of the L1 losses can be written as: $L_{L 1} = | | I_{g} - I_{t} | | 1$ (7)

To reduce image pose distortion and make the generated images look more natural and smooth, the TPIT model integrates the perceptual loss L_per. Perceptual lossL_per is defined on the feature space and is used to improve the visual quality of the generated images, and it has a great role in style transfer [32], super-resolution [33], and pose transfer generated image tasks [19, 6]. In addition to this, a pre-trained VGG19 model [26] was used to extract features in multi-scale space.

Where the L_per equation is: $L_{per} = \sum_{i} | | \emptyset_{i} (I_{g}) - \emptyset_{i} (I_{t}) | | 1$ (8) where ∅_i denotes the i-th feature of the VGG network [34].

4 Experiment

In this section, a large number of experiments are conducted to evaluate the proposed TPIT model and compare it with existing models in terms of both quantitative scores and subjective visual realism, in addition to which ablation experiments are conducted to explore the proposed TPIT model.

4.1 Dataset and metric

4.1.1 Dataset

In this paper, experiments are conducted to evaluate the proposed TPIT model on two datasets: (i.e., DeepFashion [14] dataset and Market-1501 [15] dataset.) The DeepFashion dataset contains 52,712 images of people with clear and clean backgrounds and 256×256 resolution.

The Market-1501 dataset contains 32,668 low-resolution images with a resolution of 128×64. The Market-1501 dataset is more challenging due to its relatively low resolution, complex backgrounds, and diverse lighting conditions. In the experiments of this paper, the DeepFashion dataset and the Market-1501 dataset were partitioned according to Zhedong [2], and 101,966 training and 8,570 testing pairs were collected by this method for the DeepFashion dataset, and 263,632 training and 12,000 testing pairs were collected for the Market-1501 dataset. tests. In addition, 18 human posture keypoints were extracted from the Human Posture Estimator (HPE) [35].

4.1.2 Metrics

In this work, several metrics are used to evaluate the quality of the generated images. The structural similarity metric SSIM [36] and the peak signal-to-noise ratio metric (PSNR) are used in the evaluation to assess the low-level similarity between the generated and target images.The SSIM metric evaluates the quality of the generated images by calculating the global variance and the mean value of the generated images to assess the structural similarity of the generated images. In this paper, Fr $\overset{´}{e}$ chet Inception Distance (FID) [37] and Learned Perceptual Image Patch Similarity (LPIPS) [38] are also used as evaluation metrics. The FID metric is used to calculate the Wasserstein-2 distance between the distribution of the generated results and their corresponding ground truth to measure the realism of the generated images, and the LPIPS metric, which is more consistent with human perception, is used to assess the visual quality through the LPIPS metrics to assess the visual quality. In addition to these evaluation metrics, visualization results are reported in this paper for comparison with existing methods.

4.2 Implementation details

The experiment is done based on the Pytorch framework, and the model is optimized using the Adam optimizer with the parameters β₁ set to 0.5 and β₂ set to 0.999. The initial learning rate l_r is set to 2×10-4, and λ_L1 = 1, λ_per = 1, and λ_adv = 2 for the loss functions in Equations (5). Leaky ReLU [39] is applied after each convolution or normalization layer of the discriminator, and its negative slope coefficient is set to 0.2. The batch sizes are set to 7 and 32 on the DeepFashion dataset and the Market-1501 dataset, respectively. The whole experiment is conducted on a 3060 PC server.

4.3 Comparison with previous methods

4.3.1 Quantitative comparison

In this paper, the proposed TPIT model is compared with several new models available nowadays, including PATN [8], XingGAN [7], BiGraphGAN [40], PISE [12], and SPIG [11]. The results of the evaluation of the quality of the generated images is given on the left side of Tables 1 and 2. From the results, it can be seen that the TPIT model obtained good evaluation results on Martket-1501, obtaining two best results and two suboptimal results. The TPIT model ranked highest in the structural similarity SSIM and FID metric measures, where the structural similarity SSIM reached 0.4801, improving the performance over the SSIM in the SPIG work by 34.62%, 70.4% higher than that in PATN. The FID metric reached 12.5304. It is 45.66% lower than the FID in the SPIG work, and 44.75% lower than PATN. On the Market-1501 dataset, the TPIT model generates higher quality, finer, and more vividly textured generated images than other methods. In addition, the number of parameters of our TPIT ranked the second, which was reduced by 63.85% compared with SPIG, which clearly indicated that the calculation cost of the TPIT model was reduced and the model efficiency was improved compared with other models.

Table 1
Qualitative comparison between the DeepFashion test set and advanced methods, ↑ (↓) means the higher (lower) the better

Model DeepFashion

SSIM↑ PSNR↑ FID↓ LPIPS↓

PATN [8] 0.7717 18.2543 20.7500 0.2536

XingGAN [7] 0.7706 17.9226 39.3194 0.2928

BiGraphGAN [40] 0.773 18.5208 20.951 0.2444

PISE* [12] 0.7682 18.5208 11.5144 0.2080

SPIG* [11] 0.7758 18.5867 12.7027 0.2102

Ours 0.7743 18.5501 19.0806 0.2470

Model	DeepFashion
PATN [8]	0.7717	18.2543	20.7500	0.2536
XingGAN [7]	0.7706	17.9226	39.3194	0.2928
BiGraphGAN [40]	0.773	18.5208	20.951	0.2444
PISE* [12]	0.7682	18.5208	11.5144	0.2080
SPIG* [11]	0.7758	18.5867	12.7027	0.2102
Ours	0.7743	18.5501	19.0806	0.2470

Table 2

Qualitative comparison between the DeepFashion test set and advanced methods, ↑ (↓) means the higher (lower) the better

Model	Market1501				Number of parameters↓	User study
	SSIM↑	PSNR↑	FID↓	LPIPS↓		R2G↑(M)	G2R↑(M)	R2G↑(DF)	G2R↑(DF)
PATN [12]	0.2818	14.2622	22.6814	0.3194	41.36 M	22.67	50.24	12.42	24.61
XingGAN [30]	0.3044	14.4458	22.5198	0.3058	42.77 M	35.28	65.16	21.61	33.75
BiGraphGAN [40]	0.325	14.448	28.915	0.3048	42.48 M	35.76	65.91	22.39	34.16
SPIG* [21]	0.3139	14.4894	23.0573	0.2777	117.13 M	37.93	66.53	19.47	36.80
Ours	0.4801	14.4526	12.5304	0.2911	42.34 M	38.01	66.76	18.50	32.07

4.3.2 Qualitative comparison

Figures 2 and 3 show a qualitative comparisn on of the TPIT model for the DeepFashion and Market-1501 datasets. It can be observed from the figures that in complex textures, the TPIT model-generated images are more detailed and vivid than those generated by other competing methods in complex backgrounds, especially on the Market-1501 dataset, for example, the hair shorts in row 4 and the arms of the characters in row 6 of Fig. 3. In complex backgrounds, only the generated images of the TPIT model retain the clothing of the source image details. This is mainly because the source image learning module and the cross-modal attention module of the TPIT model greatly optimize the generation task. When the source image is in a large pose change, the TPIT model still completes the complex change, generating a clear and realistic image that is consistent with the appearance of the source image, such as the figure in the fifth row of Fig. 2.

Fig. 2

Qualitative comparison with several advanced methods on the DeepFashion dataset.

Fig. 3

Qualitative comparison with several advanced methods on the Market-1501 dataset.

4.4 User study

In addition to using the evaluation metrics above, this paper also uses user studies to test the quality of the generated images. Two metrics, R2 G and G2 R, are used to show the assessment of image quality by real human perception, where R2 G is the percentage of real images considered as generated images and G2 R is the percentage of generated images considered as real images, with ↑ (↓) indicating higher (lower) the better. Fifty-five real images and 55 generated images were randomly selected from the test sets in the datasets DeepFashion and Market-1501, and these images were disrupted. Thirty volunteers were recruited for the user experiment, and the volunteers first practiced the first 10 of 110 images, and in the last 100 images, the volunteers had to choose whether the image was a generated image or a real image within 1 second. The test results of the TPIT model and other methods are shown on the right side of Table 2, and it can be seen from the table that TPIT you have a good improvement in the evaluation of the first generated image, which reflects TPIT generates a realistic image, the superiority of the quality of the generated image.

4.5 Ablation study

To explore the role and contribution of each partial component of the model in this paper in the character generation image task, in this section, ablation experiments are conducted on the dataset as DeepFashion dataset. Model without source task learning module (w/STL): this model focuses only on source-to-target image learning, removes the source task learning module, replaces the RESBlock module and DCABlock module in the model with the encoding module, and directly inputs the source image into the encoding module. Replace the branch input information module (w/I_s): change the input information I_s of the source task learning module into source image I_s and source image pose P_s and target image pose P_t to be spliced and input into the encoding module. Model without cross-modal attention module (w/CAM): based on w/I_s, remove the cross-modal attention module, add the multi-head attention module, and input and into the multi-head attention module.

Figure 4 and Table 3 show the qualitative and quantitative results of the model ablation experiments in this paper, which prove the validity of the complete TPIT model. The model without the source task learning module is unstable compared to the complete model as seen in Fig. 4, the blurring of clothing edges in the generated images as seen in Fig. 4, and the importance of the source image learning module being able to integrate source image information well as seen in Table 3. The model without the cross-modal attention module does not learn the pose transfer from the source image to the target image and the image texture of the source image well, which leads to blurred details and loss of many appearance details in the images generated by the model. The model with the cross-modal attention module can generate a more vivid detail representation, which reflects the importance of the cross-modal attention module in the method. The images generated by the model with the replacement of the branching input information module show that the generated images are partially defective and blurred, such as the white spots on the shorts in Fig. 4. Compared with the other models, it can be seen that the TPIT complete model can generate a nice global appearance and generate finer-grained, realistic local detail textures, which further demonstrates the advantage of the TPIT model in handling this task.

Table 3
Qualitative results of ablation experiment on DeepFashion dataset

Methods SSIM↑ PSNR↑ FID↓ LPIPS↓

w/CAM 0.7199 16.2148 42.8965 0.3250

w/STL 0.7681 18.2530 17.9587 0.2476

w/I_s 0.7676 18.2462 19.2231 0.2517

Full 0.7743 18.5501 19.08062 0.2470

Methods	SSIM↑	PSNR↑	FID↓	LPIPS↓
w/CAM	0.7199	16.2148	42.8965	0.3250
w/STL	0.7681	18.2530	17.9587	0.2476
w/I_s	0.7676	18.2462	19.2231	0.2517
Full	0.7743	18.5501	19.08062	0.2470

Fig. 4

Qualitative comparison of ablation studies on the DeepFashion dataset.

5 Conclusion

In this paper, a novel (TPIT) model network is proposed to handle challenging pose transfers. The TPIT model introduces a cross-modal (i.e., keypoint and source image) attention mechanism and a source image self-learning module to capture image-correlated micro-motions and textures, and this correlation can be used as a strong guide for transferring source textures to the target generated image, resulting in finer-grained generated image details. Experimental results demonstrate that the TPIT network exhibits superior performance in both subjective visual realism and objective quantitative scores while improving computational efficiency and significantly reducing the complexity of the model compared to existing previous models. In future work, we will consider optimizing the parameter settings in the model, or consider replacing the attention mechanism module in the model to improve the appearance details of the person generation images.

Footnotes

Acknowledgment

This research was funded by the Tai ‘an science and technology innovation development project (Grant No. 2022GX075).

References

Walker

Jacob

, Marino

Kenneth

, Gupta

Abhinav

and Hebert

Martial

, The pose knows: Video forecasting by generating pose futures, In Proc. ICCV (2017), pages 3352–3361.

Zheng

Zhedong

, Zheng

Liang

and Yang

, Unlabeled samples generated by GAN improve the person re-identification baseline in vitro, In Proc. ICCV, pages 3774–3782, 2017.

Zhang

Quan

, Lai

Jianhuang

and Xie

Xiaohua

, Learning modal-invariant angular metric by cyclic projection network for vis-nir person re-identification, IEEE Transactions on Image Processing 30 (2021), 8019–8033.

Goodfellow

Ian

, Pouget-Abadie

Jean

, Mirza

Mehdi

, Xu

Bing

, Warde-Farley

David

, Ozair

Sherjil

, Courville

Aaron

and Bengio

Yoshua

, Generative adversarial nets, In Advances in Neural Information Processing Systems (NeurIPS), pages 2672–2680, 2014.

Kingma

Diederik P.

and Welling

Max

, Auto-encoding variational bayes, arXiv preprint arXiv:1312.6114, 2013.

Siarohin

Aliaksandr

, Sangineto

Enver

, Lathuiliere

Stéphane

and Sebe

Nicu

, Deformable gans for pose-based human image generation, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3408–3416, 2018.

Tang

Hao

, Bai

Song

, Zhang

, Torr

Philip H.S.

and Sebe

Nicu

, Xinggan for person image generation, In European Conference on Computer Vision (ECCV), pages 717–734, 2020.

Zhu

Zhen

, Huang

Tengteng

, Shi

Baoguang

, Yu

Miao

, Wang

Bofei

and Bai

Xiang

, Progressive pose attention trans-fer for person image generation, In Proceedings of the IEEE Conference on computer Vision and Pattern Recogni-tion (CVPR), pages 2342–2351, 2019.

Ren

Yurui

, Yu

Xiaoming

, Chen

Junming

, Li

Thomas H.

and Li

, Deep image spatial transformation for person image generation, In Proceedings of the IEEE Conference on com-puter Vision and Pattern Recognition (CVPR), pages 7687–7696, 2020.

10.

Tabejamaat

Mohsen

, Negin

Farhood

and Bre-mond

Francois

, Guided flow field estimation by generating indepen-dent patches, British Machine Vision Conference (BMVC), 2021.

11.

Zhengyao

, Li

Xiaoming

, Li

Xin

, Li

, Lin

Tianwei

, He

Dongliang

and Zuo

Wangmeng

, Learning semantic person image generation by region-adaptive normalization, In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition (CVPR), pages 10806–10815, June 2021.

12.

Zhang

Jinsong

, Li

Kun

, Lai

Yu-Kun

and Yang

Jingyu

, Pise:Person image synthesis and editing with decoupled gan, In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition (CVPR), pages 7982–7990 June 2021.

13.

Yanghao

, Mao

Hanzi

, Girshick

Ross

and He

Kaiming

, Exploring Plain Vision Transformer Backbones for Object Detection, arXiv preprint arXiv:2203.16527v2, 2022.

14.

Liu

Ziwei

, Luo

Ping

, Qiu

Shi

, Wang

Xiaogang

and Tang

Xiaoou

, Deepfashion: Powering robust clothes recognition and retrieval with rich annotations, In Proceedings of the IEEE Conference on computer Vision and Pattern Recogni-tion (CVPR), pages 1096–1104, June 2016.

15.

Zheng

Liang

, Shen

Liyue

, Tian

, Wang

Shengjin

, Wang

Jing-dong

and Tian

, Scalable person re-identification:A benchmark, In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1116–1124, 2015.

16.

Lassner

Christoph

, Pons-Moll

Gerard

and Gehler

Peter V.

, A generative model of people in clothing, In Proc. ICCV, pages 853–862, 2017.

17.

Liqian

, Jia

, Sun

Qianru

, Schiele

Bernt

, Tuyte-laars

Tinne

and Van Gool

Luc

, Pose guided person image genera-tion, In Advances in Neural Information Processing Systems, pages 406–416, 2017.

18.

Pumarola

Albert

, Agudo

Antonio

, Sanfeliu

Alberto

and Moreno-Noguer

Francesc

, Unsupervised person image syn-thesis in arbitrary poses, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 2018.

19.

Esser

Patrick

, Sutter

Ekaterina

and Ommer

Björn

, A variational u-net for conditional appearance and shape generation, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8857–8866, 2018.

20.

Ronneberger

Olaf

, Fischer

Philipp

and Brox

Thomas

, U-net:Convolutional networks for biomedical image segmentation, In Medical Image Computing and Computer-Assisted Intervention (MICCAI), pages 234–241, 2015.

21.

Yining

, Huang

Chen

and Loy

Chen Change

, Dense intrinsic appearance flow for human pose transfer, In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition (CVPR), pages 3688–3697, June 2019.

22.

Goodfellow

Ian

, Pouget-Abadie

Jean

, Mirza

Mehdi

, Xu

Bing

, Warde-Farley

David

, Ozair

Sherjil

, Courville

Aaron

and Bengio

Yoshua

, Generative adversarial nets, In NIPS, 2014.

23.

Reed

Scott

, Akata

Zeynep

, Yan

Xinchen

, Logeswaran

Lajanugen

, Schiele

Bernt

and Lee

Honglak

, Generative adversarial text to image synthesis, In ICML, 2016.

24.

Goodfellow

Ian

, Pouget-Abadie

Jean

, Mirza

Mehdi

, Xu

Bing

, Warde-Farley

David

, Ozair

Sherjil

, Courville

Aaron

and Bengio

Yoshua

, Generative adversarial nets, In NIPS, 2014.

25.

Cao

Junxu

, Chen

, Guo

Jun

and Shi

Ruichao

, Attention-guided Context Feature Pyramid Network for Object Detection, arXiv preprint arXiv:2005.11475v1, 2020.

26.

Cao

Yuhang

, Chen

Kai

, Loy

Chen Change

and Lin

Dahua

, Prime Sample Attention in Object Detection.[J], CoRR, 2019, abs/1904.04821.

27.

Huan

and Jinglei

Tang

, Dairy Goat Image Generation Based on Improved-Self-Attention Generative Adversarial Networks[J], IEEE Access, 2020.

28.

Wang

Hui

, Xu

Jiawen

, Yan

Ruqiang

, Sun

Chuang

and Chen

Xuefeng

, Intelligent Bearing Fault Diagnosis Using Multi-Head Attention-Based CNN[J], Procedia Manufacturing 49 (2020).

29.

Nazi Zabir

, Rabbi

Mashrur Fazla

, Amirul

Islam Md

and Shumit

Saha

, Fibro-CoSANet: pulmonary fibrosis prognosis prediction using a convolutional self attention network.[J], Physics in Medicine and Biology 66(22) (2021).

30.

Brock

Andrew

, Donahue

Jeff

and Simonyan

Karen

, Large scale GAN training for high fidelity natural image synthesis, arXiv preprint arXiv:1809.11096, 2018.

31.

Nair

Vinod

and Hinton

Geoffrey E.

, Rectified linear units improve restricted boltzmann machines, In Proc. ICML, pages 807–814, 2010.

32.

Johnson

Justin

, Alahi

Alexandre

and Fei-Fei

, Perceptual losses for real-time style transfer and super-resolution, In Proc. ECCV, pages 694–711, 2016.

33.

Ledig

Christian

, Theis

Lucas

, Huszar

Ferenc

, Caballero

Jose

, Cunningham

Andrew

, Acosta

Alejandro

, Aitken

Andrew P.

, Tejani

Alykhan

, Totz

Johannes

, Wang

Zehan

and Shi

Wenzhe

, Photo-realistic single image super-resolution using a generative adversarial network, In Proc. CVPR, pages 105–114, 2017.

34.

Simonyan

Karen

and Zisserman

Andrew

, Very deep convolutional networks for large-scale image recognition, In International Conference on Learning Representations (ICLR), 2015.

35.

Cao

Zhe

, Simon

Tomas

, Wei

Shih-En

and Sheikh

Yaser

, Realtime multi-person 2d pose estimation using part affinity fields, In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition (CVPR), pages 1302–1310, July 2017.

36.

Wang

Zhou

, Bovik

A.C.

, Sheikh

H.R.

and Simoncelli

E.P.

, Image quality assessment: from error visibility to structural similarity, IEEE Transactions on Image Processing 13(4) (2004), 600–612.

37.

Heusel

Martin

, Ramsauer

Hubert

, Unterthiner

Thomas

, Nessler

Bernhard

and Hochreiter

Sepp

, Gans trained by a two time-scale update rule converge to a local nash equilibrium, In Advances in Neural Information Processing Systems(NeurIPS), pages 6629–6640, 2017.

38.

Zhang

Richard

, Isola

Phillip

, Efros

Alexei A.

, Shechtman

Eli

and Wang

Oliver

, The unreasonable effectiveness of deep features as a perceptual metric, In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition (CVPR), pages 586–595, June 2018.

39.

Maas

Andrew L.

, Hannun

Awni Y.

and Ng

Andrew Y.

, Rectifier nonlinearities improve neural network acoustic models, In Proc. ICML 30 (2013), 3.

40.

Tang

Hao

, Bai

Song

, Torr

Philip H.S.

and Sebe

Nicu

, Bipartitegraph reasoning gans for person image generation, In BMVC, 2020.

41.

Tian, Chunpeng, et al. Arc fault detection using artificial intelligence: Challenges and benefits. Mathematical Biosciences and Engineering 20(7) (2023), 12404–12432.

Dual-task attention-guided character image generation method

Abstract

Keywords

1 Introduction

2 Related work

2.1 Pose guided person image generation

2.2 Generative adversarial networks

2.3 Attention mechanism

3 Method

3.4 Training

4.1 Dataset and metric

4.1.1 Dataset

4.1.2 Metrics

4.2 Implementation details

4.3 Comparison with previous methods

4.3.1 Quantitative comparison

4.5 Ablation study

Table 3 Qualitative results of ablation experiment on DeepFashion dataset Methods SSIM↑ PSNR↑ FID↓ LPIPS↓ w/CAM 0.7199 16.2148 42.8965 0.3250 w/STL 0.7681 18.2530 17.9587 0.2476 w/I s 0.7676 18.2462 19.2231 0.2517 Full 0.7743 18.5501 19.08062 0.2470

Footnotes

Acknowledgment

References

Table 3
Qualitative results of ablation experiment on DeepFashion dataset

Methods SSIM↑ PSNR↑ FID↓ LPIPS↓

w/CAM 0.7199 16.2148 42.8965 0.3250

w/STL 0.7681 18.2530 17.9587 0.2476

w/I_s 0.7676 18.2462 19.2231 0.2517

Full 0.7743 18.5501 19.08062 0.2470