Res-LGAN for human motion style transfer with more features preserved

2.1 Motion style transfer

Motion style transfer is a widely studied problem in computer graphics, which aims to extract a style from a motion clip and apply it to another neutral motion clip. This is useful in various fields such as computer animation and video games, and researchers can also use stylized motions to train networks or conduct further research.

The task of motion style transfer is challenging because of the complexity of the motion style representation. Current methods for motion style transfer can be broadly categorized into two groups: traditional methods and deep learning methods.

Traditional methods have made significant progress in the field of motion style transfer [26]. These methods typically focus on completing the transfer task in the frequency domain by comparing the differences in the Fourier coefficients of different motions[30]. Another approach involves the use of dynamic models for motion style transfer. Despite some success, traditional methods still have limitations such as difficulty in handling high-dimensional motion data and lack of flexibility in handling different motion styles.

The use of neural network technology for style transfer was first introduced by [5], where it was used to generate art with a similar style to famous paintings. Recently, with the advancement of deep learning, many works have applied deep learning networks to the task of motion style transfer [20, 31]. However, these approaches often neglect the artifacts and inconsistencies of the generated motions, such as foot-sliding or trajectory distortion. Holden and his colleagues [12] proposed a model that uses neural networks and additional constraints to address this issue.

As previously mentioned, when transferring motion styles, preserving the original motion features is also crucial for the quality of the generated motions. Our proposed method Res-LGAN not only addresses the problem of artifacts and unnatural actions but also considers basic motion properties such as velocities and trace, without significantly increasing computation cost.

2.2 Generative adversarial networks

Generative adversarial networks (GANs) is a novel approach which based the idea of game theory as the generative model proposed by [6]. Generative adversarial networks have currently been developed greatly and are widely applied in multiple research fields [24, 25]. [28] show that combines RNNs and adversarial training for motion synthesis. Although GANs have powerful generated abilities, they might meet some troubles while training, such as the difficulty of convergence, gradient vanishing. WGAN, introduced by [2, 7], avoid the problems like mode collapse, and provide meaningful learning curves useful for debugging and hyper-parameter searches. Besides, various types of GANs have appeared for different tasks. [3] propose Info-GAN as an information-theoretic extension to the generative adversarial nets in order to learn disentangled representations. Jun-Yan et al. define a new model for the unpaired image to image translation named Cycle-Consistent Adversarial Networks (Cycle-GANs)[33]. They introduce a cycle consistency loss to enhance ability of the model by regularizing structured data with a long history.

[18] introduce an alternative type of GANs called Conditional generative adversarial nets (Conditional-GANs). Conditional-GANs are constructed from the conception of the combination of additional information and input data, and this property can be used for motion style transfer. Our network is different from theirs because our target is transferring style to other neutral motions rather than synthesis a completely new motion. Energy-based GANs introduce an energy view to explain the adversarial architecture in [32]. Our models have the capacity to save the representation and original information of input data inspired by EBGAN and Conditional-GAN.

2.3 Residual LSTM

Our decision to use a neural network built with recurrent neural networks (RNNs) and Long Short-Term Memory (LSTM) cells was influenced by previous research [8]. [8] demonstrated the powerful capabilities of RNNs for motion synthesis. While RNNs are effective for sequence processing and have been used in various applications, such as speech recognition and music generation, they do have limitations. RNNs can struggle with long-term memory and may experience issues with gradient vanishing or exploding. Long Short-Term Memory, as proposed by [11], is a novel architecture that can address these limitations and has shown strong performance in tasks such as information extraction, speech processing, and natural language processing (NLP) applications.

It is commonly thought that deeper networks offer better performance, as they can handle more complex representations of data. However, deeper architectures can be difficult to train. Recently, [21] introduced an alternative type of LSTM network, which aims to overcome these difficulties. The Residual LSTM network is similar to convolutional residual networks as proposed by [10], and shows that adding residual connections between LSTM layers can lead to improved capabilities. With this in mind, the generator of our model is constructed with residual LSTM architecture.

3.1 Network sructure

We adopted Residual LSTM [21] and Conditional-GAN [18] as our baseline models.

The architecture of our transfer module is based on the idea of Residual LSTM [21], which is an improved LSTM module as mentioned above. The generator G is adapted from Residual LSTM and contains a down-sampling encoder, a transfer module, and an up-sampling decoder. The encoder module is used to obtain the representations of the original motions. As previously mentioned, LSTM can handle the spatial-temporal representations of the input data. Therefore, the encoder is constructed with two linear layers and two LSTM layers. The decoder module is used to reconstruct the motion data, similar to the decoder of the refine network. However, unlike the refine network, the decoder of the generator is constructed with linear layers and LSTM layers. The core of our model is the transfer module of the generator, which consists of a stacked residual containing two LSTM layers. The transfer module has the capacity to stylize motions in the hidden space. The residual connection does not add any additional parameters. Therefore, compared to bi-directional LSTM layers, residual LSTM does not increase the complexity of our model. The structure of the generator G is shown in Table 1. The D consists of a feature module and a difference module. In order to precisely handle differences between real motion data and generated motion data, we use a trick by comparing differences of motions. We denote dif as the differences between adjacent frames, which is used to compare the difference between adjacent poses. By applying this trick, the capacity of the discriminator is obviously improved. The structure of the discriminator D is shown in Table 2

R is only a simple convolutional auto-encoder since convolutional neural networks (CNNs) have achieved particular success in learning motion manifolds. In the process of generating desired motions, only using adversarial loss does not guarantee that the generated motions are natural and smooth, such as wrong bone length and discrete locomotion. Inspired by [14] which is widely used in modifying artificial motions, we pre-trained the refine network R, which consists of a few convolution and pooling layers to handle motion features in the latent space. The structure of the refine net R is shown in Table 3.

The primary objective of Res-LGAN is to generate stylized motion based on the original, neutral motion while preserving its basic features. In order to achieve natural and smooth motion, we employ three loss functions to optimize our models. In addition to the standard adversarial loss used in GANs, we utilize a refinement loss with L2 norm to modify the generated motions, and a reconstruction loss to preserve the characteristics of the original motion. In GANs, the generator and discriminator engage in a minimax game, where the generator (G) attempts to generate natural motion that is indistinguishable from real motion for the discriminator (D).

G-D Adversarial Loss. The stylized motion M generated by G and the real style motion S are fed to the discriminator D. By adjusting the hyper-parameters, the generator can produce realistic stylized motions. The function is: L adv = E I [ log D ( I ) ] + E S [ log D ( S ) ](1) + E I [ log ( 1 - D ( G ( I ) ) ) ](2)

where generator G tries to minimize the loss function Eq. 2 while discriminator D tries to maximize it.

Reconstruction Loss. Unlike in [12], we propose a new loss function to reduce computation costs. Building on the principle of energy-based generative adversarial networks, we incorporate a reconstruction loss to preserve the essential features of the input motion. When stylizing neutral motions, it is important to maintain information such as trace, bone angles, and velocities. To accomplish this, we use the L2 norm for G to retain these features, defined as: L rec = E(3)

where M is generated from G (I) and generator G tries to minimize Eq 3.

Refine Loss. As previously mentioned, our Refine Network consists of an encoder, which is constructed using two 1D convolutional layers and two pooling layers. Due to the similarity of this architecture to an autoencoder, the loss function is: L ref = E [ ∥ C - C ˆ ∥ 2 ](4)

To pre-train the refine net, we use ground truth motions C as input and encode them to the hidden space. We then decode them back to reconstruct C ˆ , and optimize the network by minimizing Eq 4.

Full Generator Loss. Finally, the full objective loss function is: L D = L adv(5)

while the full loss function for G is: L G = L adv + L rec(6)

In summary, Our model has the following features:

LSTM can handle the spatial-temporal representations of the input data. The use of a residual structure makes the model easier to optimize and increases the accuracy of the model by increasing the depth, while avoiding the gradient disappearance problem caused by increasing the depth of the model. Due to the characteristics of the residual structure, the input data can be directly passed to the output data, and the model only needs to learn the change between the output data and the input data, which not only ensures the integrity of the data but also simplifies the learning goal and difficulty of the model.

5.1 Dataset

We chose the CMU Mocap dataset [4] on which to perform motion style transfer using Res-LGAN.

CMU Mocap The Carnegie Mellon University Motion Capture Database is a collection of motions with various expressions, including neutral, depressed, and zombie-like. As our model is designed to transfer styles to neutral motions, we selected 1835 neutral motions and 122 style motions, after cropping and standardizing. These motions have a size of 66 × 240. We use 1500 of the selected motions for training and the remaining motions for testing.

5.2 Experiment results

To compare Res-LGAN with state-of-the-art methods, we reproduce and train the model of Fast Style Transfer [12] and Real-Time Motion Style Transfer [23]. Both models also consider preserving motion features, but unlike our Res-LGAN model, they adopt additional constraints. The results are evaluated qualitatively on the CMU Mocap dataset. Additionally, we conduct an ablation study on the proposed reconstruction loss function (and the refine net) to evaluate their effects.

OPEN IN VIEWER

Fig. 3

The result of experiment to evaluate the capacity of preserving original features of Res-LGAN. This figure shows walking and running motions in our experiments.

OPEN IN VIEWER

Fig. 4

The result of experiment to evaluate the capacity of transferring styles of Res-LGAN. This figure shows the results of Fast Neural Transfer[12] and Res-LGAN for transferring tired style to the same neutral motion.

Qualitative Evaluation In this section, we evaluate the quality of the generated motions and the feature preservation ability of our model. To do this, we divide the evaluation into two sections: feature preservation and style transfer, and perform different experiments for each.

First, we select neutral motions to evaluate the motion quality and the original content preservation ability of our Res-LGAN. We use two different types of motions (running and walking) with a neutral style from the CMU Mocap dataset as input. The results are shown in Figure 3. As Res-LGAN is trained using neutral motions, it is easy to observe whether the features of the original motions are preserved. From Figure 3, it can be seen intuitively that Res-LGAN can synthesize motions that are close to the content of the ground truth, although some joints in the running actions have changed bone angles. Our Res-LGAN model behaves like an auto-encoder when it is trained using neutral motions. This model can confidently handle the human motion manifold and has the capacity to preserve motion features.

Next, we evaluate both the content preservation ability and the style transfer ability of different models. In this section, we focus on the style transfer ability assessment, so we choose the same type of motion with varying styles, including tired and zombies-like styles. The experimental results are shown in Figure 4. As shown in Figure 4, the model of [12] achieves motion style transfer while changing the pace of the original motion, while the generated motions of Res-LGAN have a pace that is closer to the original. In short, Res-LGAN, which we proposed, can transfer style to the neutral motion and preserves more original features, resulting in more precise stylized motions which animators and researchers desire.

Furthermore, we test the ability to transfer different styles to different neutral motions as is shown in Figure 1. It can be seen that both tired and zombies-like styles are transferred successfully.

In conclusion, our Res-LGAN model can transfer different styles to different neutral motions and preserves more original features, resulting in more precise stylized motions which animators and researchers desired.

Quantitative Evaluation Because the movement of joints in the human body during exercise can be approximately represented as periodic motion, for example, arm swing during walking, and the hand joint is a more complex, periodic helical motion, it can be expanded using the discrete Fourier transform into the form of a Fourier series. The same joint coordinates are different in human movement data with different styles. For example, in a zombie style walk, the amplitude of the hand joint is small, while in a normal walk the amplitude of the hand joint sway is larger. Therefore, different movements have a correlation between style and frequency domain amplitude components, while basic movement characteristics have a correlation with time domain signal components.

According to [30], difference of style is highly correlated in the frequency domain magnitude component regardless of what kinds of motions. Therefore, we build on this idea to evaluate the capacities of Res-LGAN model quantitatively by comparing frequency domain magnitude component and time domain signals.

OPEN IN VIEWER

Fig. 5

Time domain signals of results. Top: Tired motions; bottom: Zombie-like motions.

OPEN IN VIEWER

Fig. 6

Frequency domain magnitude component of results. Top: Tired motions; bottom: Zombie-like motions.

First, we compare the differences of the time domain signals of each body joints (hips, spine, neck and head) which belong to the generated motions and original neutral motions to evaluate the feature-preserved ability of our model.

The Figure 5 illustrates a data sequence of the hip joint, with the position information of the joint node on the ordinate axis and the slope of the data curve indicating the motion rate of the joint. As depicted, compared to [12] and [23], the Res-LGAN model curve exhibits a high level of consistency with the original motion curve, with the overall rate of change remaining consistent with that of the original motion. In stylized movements, each joint typically requires corresponding changes. For instance, a "tired" style of movement would entail greater fluctuations in the hip joint, as opposed to the more regular movement seen in a normal walk. As a result, there may be slight variations in the joint position at certain points in time. However, it can be inferred from the figure that the Res-LGAN generated stylized motion preserves the time-domain information of the hip joint to a significant extent, thus effectively maintaining the essential characteristics of the underlying motion.

OPEN IN VIEWER

Fig. 7

The ablation study to evaluate the reconstruction loss function.

Then we compute frequency domain magnitude component to evaluate style transferring ability of Res-LGAN. Because the key joints of different style are different, we choose different joints group for them. For example, we select shoulder and hands for tired style while forearm and hands for zombie style. Figure 6 shows the frequency domain magnitude component of these two style motions.

As can be seen from the Figure 6, the Res-LGAN model is most similar to the frequency domain amplitude component curve of the input style movement. Due to differences in original movement speed, direction, and other information, the model makes some exaggerations or convergences when generating stylized movement, resulting in amplitude max values not being entirely consistent with the input style movement. However, overall, the model still conforms to the desired movement style.

According to the quantitative evaluation above, our Res-LGAN model achieves motion style transfer task in preserving the features of original motions.

Ablation Study To demonstrate the significant role of the reconstruction loss function in optimizing the generator G to learn the ability to preserve the original motion features (velocities and trace), we conduct an ablation study on the reconstruction loss function by removing it and comparing the Res-LGAN model without reconstruction loss with the full Res-LGAN. The experiment is also performed on the CMU Mocap dataset.

We choose two different styles and two different actions (tired and zombies; walking and running) to evaluate the capacity of the proposed reconstruction loss. As is shown in Figure 7, the motions generated by Res-LGAN without reconstruction loss do not contain enough motion features.

In summary, the reconstruction loss function effectively helps Res-LGAN generate motions that are more consistent with the features of the original input motions compared to Res-LGAN without reconstruction loss.