UViT: Efficient and lightweight U-shaped hybrid vision transformer for human pose estimation

Abstract

Pose estimation plays a crucial role in human-centered vision applications and has advanced significantly in recent years. However, prevailing approaches use extremely complex structural designs for obtaining high scores on the benchmark dataset, hampering edge device applications. In this study, an efficient and lightweight human pose estimation problem is investigated. Enhancements are made to the context enhancement module of the U-shaped structure to improve the multi-scale local modeling capability. With a transformer structure, a lightweight transformer block was designed to enhance the local feature extraction and global modeling ability. Finally, a lightweight pose estimation network— U-shaped Hybrid Vision Transformer, UViT— was developed. The minimal network UViT-T achieved a 3.9% improvement in AP scores on the COCO validation set with fewer model parameters and computational complexity compared with the best-performing V2 version of the MobileNet series. Specifically, with an input size of 384×288, UViT-T achieves an impressive AP score of 70.2 on the COCO test-dev set, with only 1.52 M parameters and 2.32 GFLOPs. The inference speed is approximately twice that of general-purpose networks. This study provides an efficient and lightweight design idea and method for the human pose estimation task and provides theoretical support for its deployment on edge devices.

Keywords

Pose estimation multi-branch structure lightweight network context enhancement attention mechanism

1 Introduction

Human pose estimation is a fundamental task in computer vision that involves localizing and identifying anatomical keypoints on the human body, such as wrists, elbows, and knees. It has wide-ranging applications in areas such as behavior recognition [1, 2], human-computer interaction [3, 4], and tracking [5, 6]. In recent years, significant advancements have been made in pose estimation, particularly with the development of Convolutional Neural Networks (CNNs) [7 –11] and the introduction of the transformer architecture [12 –16].

To improve performance, existing human pose estimation models often employ wider and deeper network architectures, resulting in a significant increase in parameters and Floating-point Operations (FLOPs). This increased computational complexity leads to slower inference speeds, making it challenging to deploy on mobile devices. In the meantime, there is a growing demand for efficient and lightweight human pose estimation networks. Although various techniques have been proposed to improve efficiency and lightweight design, they often suffer a significant drop in performance. Therefore, further research is still needed in the field of efficient and lightweight human pose estimation.

The hybrid architecture [17 –21] design of CNN and Transformer motivates the construction of a lightweight and low-latency network for mobile visual tasks. This design combines spatial induction bias, insensitivity to data augmentation of CNN, and the adaptive weighting and global processing advantages of ViTs. Inspired by this, this work focuses on efficient and lightweight human pose estimation. To that end, the U2-Net [22] architecture is first redesigned to achieve lightweightness and verify its efficient usage of contextual information in human pose estimation tasks. Thereafter, a mixed pooling attention mechanism is proposed to enhance the feature extraction capability of the contextual feature extraction module. Furthermore, based on this contextual feature extraction module, the UViT-Residual Block, a hybrid architecture feature extraction module, is introduced to enhance local and global modeling capability. To further demonstrate the effectiveness of the lightweight network, a lightweight pose estimation network— U-shaped Hybrid Vision Transformer (UViT)— is developed, as shown in Fig. 1.

Fig. 1

UViT network architecture. (a) U_Context Block. (b) UViT Stem Block. (c) UViT-Residual Block. (d) The proposed UViT network, is described in Sec. 3. More details and other variants are shown in Table 3.

To validate the proposed UViT network, this work first defined general-purpose Networks and lighweight Networks for performance comparison, as shown in Table 1. Then, extensive experiments are conducted on the widely-used COCO [23] and MPII [24] datasets. The proposed UViT model has a smaller size and FLOPs compared with the general-purpose network while offering a higher accuracy compared with the lightweight network. For example, compared with MobileNetV2 (achieving 66.8 in AP score on COCO test-dev set with 9.6M parameters and 3.33 GFOPs), the smallest network UViT-T can achieve 70.2 in AP with only 1.52 M parameters and 2.32 GFLOPs. Compared to the general-purpose network, although there is a performance gap, UViT networks have significantly lower GFLOPs and parameters, yet achieve nearly twice the inference speed. The fusion of contextual information and modeling of global information plays a potentially significant role in lightweight networks.

An efficient pose-estimation network called the UViT is proposed. The key to this algorithm is the fusion strategy of contextual information and the ability of local and global feature modeling.

Extensive experiments on two benchmark datasets, COCO and MPII, demonstrate the effectiveness of the proposed approach: on the COCO dataset, UViT and GFLOPs reduce the number of parameters by factors of 6.6 and 4.5, respectively, compared to the state-of-the-art HRNet; in addition, compared to the best network in the MobileNet series, the accuracy is improved by more than 5% via half the parameters.

Validation experiments on the information fusion strategy was designed to demonstrate the effectiveness of the fusion of multi-scale contextual information in local modeling, thereby exhibiting the advantages of the combination of CNN and lightweight transformers for feature extraction.

Table 1

Definition of General-Purpose Networks and Lightweight Networks for performance comparison, including parameter count ranges, model names, and corresponding references

Network scale	#Params (M)	Model
General-purpose networks	12–207.9	Hourglass [25], SwinT [12], SimpleBaseline [26], TokenPose [16], RIFormer [27], TransPose [15], HRNet-W32 [11], DARK [28], IntegralRegression [7]
Lightweight networks	1.1–9.6	MobileNetV2 [29], ShuffleNetV2 [30], Small HRNet [31], LPN [32], LiteHRNet-30 [33]

The remainder of this paper is organized as follows. Section 2 provides an overview of the work related to human pose estimation, attention mechanisms, and hybrid architecture designs. Section 3 details the mixed pooling-enhanced attention mechanism, context-enhanced module, UViT-Residual block, and proposed lightweight pose estimation network. Section 4 provides extensive results and a comprehensive analysis of the challenging COCO and MPII datasets; a comparison with other state-of-the-art and real-time methods; and a qualitative assessment and comparison of the inference speed of the UViT network. Finally, Section 5 provides the concluding remarks.

2 Related work

2.1 Lightweight pose estimation

With the introduction of the DeePose [34], methods based on deep convolutional neural networks have made significant progress in the field of pose estimation, novel network structures and methods have achieved better performance on MPII [24] and COCO [23] benchmarks, especially the high-resolution representation network, HRNet [11]. However, these studies have a more complex architecture and high computational complexity to improve the accuracy of pose estimation, which limits their effective deployment on edge devices. In this work, the main focus is on the design of efficient and lightweight networks for human pose estimation.

To make pose estimation networks more lightweight, Zhang et al. [35] use a model-training method for fast pose distillation using the hourglass network [25] as a teacher model to train lightweight student networks. Bulat et al. [36] binarized the model to speed up the inference time while compressing the model parameters; however, the performance was degraded. Zhang et al. [32] use lightweight convolutional modules to build the network and propose an iterative training strategy with a reduced number of model parameters; however, its computational complexity remains high. Yu et al. [33] use a conditional channel-weighting operation instead of a 1×1 convolution in a shuffle block, which improves the computational efficiency of the network. Wang et al. [37] propose a novel human pose estimation method that utilizes an unbiased data processing technique, introduces lightweight bottleneck blocks with re-parameterized structures, and utilizes multi-branch and single-branch structures to decouple training and deployment phases; this results in improved accuracy and faster inference speed. Zhang et al. [38] propose an ultra-lightweight end-to-end pose distillation network that can perform real-time pose estimation in resource-constrained environments and demonstrates good generalization capabilities. Although lightweight models for pose estimation have been proposed, they often face the issue of sacrificing accuracy and performance in pursuit of lightweight design. Therefore, developing a lightweight pose estimation network that is both efficient and accurate remains an unresolved problem.

The proposed U-Net [39] architecture employs an encoder-decoder architecture and effectively integrates multi-scale contextual information, resulting in improved feature extraction capability and excellent performance in pixel-level visual tasks. First, U-Net achieves parameter sharing between the encoder and decoder, significantly reducing the number of trainable parameters— this reduces computational and storage overheads and mitigates the risk of overfitting. Second, U-Net adopts the design of local connections and small receptive fields, thus reducing the computational load at each layer and the range of contextual information to be processed. The utilization of small receptive fields restricts the focus of each convolutional kernel to local regions, thereby enhancing the ability for local perception of features. Finally, U-Net employs downsampling and upsampling operations to adjust image size and extract contextual information at different scales. U-Net successfully achieves the objective of maintaining high feature extraction capability while reducing the computational complexity.

Following its design principles, a U-shaped Hybrid Vision Transformer network (UViT) is presented, which offers advantages in terms of inference speed, computational complexity, and model size. In contrast to U-Net, considering the lightweight design of the network, a combination of 3×3 group convolution and Inverted Residual block [29] is adopted to downsample by a factor of 4 in the feature extraction stem phase. Additionally, in the En_1 stage of the encoder, contextual features are effectively extracted using a context enhancement module. Furthermore, in the other encoding stages, the UViT-Residual module is employed to perform both local and global feature modeling. Finally, in the decoding stage, the contextual features are integrated with the corresponding outputs from each encoding stage, achieving comprehensive context feature fusion. Its efficiency and effectiveness are validated on the COCO [23] and MPII [24] benchmarks.

2.2 Attention mechanism

The attention mechanism is a type of human cognitive system-based cognitive awareness of things-specific information, and has been proven to be an effective method for improving the performance of deep learning systems, especially for tasks such as target detection [14, 40], semantic segmentation [41, 42], production remanufacturing [43, 44], and pose estimation [45, 46].

Through careful empirical investigation, Hu et al. [47] proposed a Squeeze-and-Excitation Network (SENet), which is a representative work in the computer vision field, to apply the attention mechanism to the channel dimension, and numerous subsequent works based on the channel domain are modified based on this. Considering that SENet, which uses Global Average Pooling (GAP) to model the global context and leads to the loss of spatial information, SPA-Net [48] takes an alternative approach using a spatial pyramid structure composed of multiple adaptive average pooling (AAP) to model the local and global contextual semantic information such that the spatial semantic information is more fully utilized. Aiming at the considerably limited receptive field obtained by the convolution operation at each layer, Wang et al. [46] proposed a nonlocal block, which can directly obtain the dependency between two positions without considering the distance; however, the amount of calculation is overly large, and GCnet [49] and CCNet [50] have effectively improved the calculation of the nonlocal block. For deep neural networks, attention mechanisms contribute to performance improvement, however, when the model is small, the computational cost associated with attention mechanisms cannot be ignored, which often leads to significantly lower performance compared to larger networks. To enhance the accuracy and performance of pose estimation while maintaining a lightweight design, there is a need to develop new methods to improve existing attention mechanisms.

Through comparative analysis, CoordinateAttention [51] decomposes channel attention into two one-dimensional feature-coding processes to capture remote dependencies and retain accurate location information. The obtained feature map is then separately coded into direction-aware and position-sensitive attention maps to enhance the representation of the objects of interest. Consequently, it can be applied to the proposed lightweight network, which can effectively improve the network performance without increasing excessive computation. However, CoordinateAttention falls short of simultaneously capturing both local detailed information and global semantic information in the image. To address this limitation and obtain a balance between lightweight design and improved performance, a Mixed Pooling Coordinate Attention mechanism is proposed. This mechanism significantly enhances the accuracy of pose estimation tasks while requiring minimal additional computational cost.

2.3 Hybrid architecture of CNN and transformer

Since the emergence of the ViT [52] model in 2020, researchers have discovered the great potential of transformer architectures in the visual field. However, the transformer model has numerous parameters and requires high computational power, which makes it difficult to deploy the model on a mobile terminal. Although there are many excellent works on transformer architecture in the visual field, such as the Swin-Transformer [12] in 2021, which is better and lighter than ViT, a large gap remains in model parameters and inference speed compared with CNN-based lightweight models (such as MobileNet series [29 , 54]). Vision Transformer (ViT) is limited because of the lack of spatial inductive biases and the requirement for large-scale datasets along with appropriate data augmentation, regularization, and distillation techniques. To address these limitations and achieve performances comparable to those of CNNs, recent research has introduced hybrid architectures combining CNNs and Transformers. This approach aims to efficiently capture both local and global information while balancing network performance and efficiency. By incorporating CNNs, these hybrid architectures alleviate the issues of massive parameters and computational complexity associated with Transformers.

Early research on ViT introduced convolutions [52, 55], replaced the patchy stem with convolutional operations, or required window-based [12, 56] attention to implicitly integrate CNN and Transformer; these effectively reduced the computational complexity and parameters of ViTs but maintained certain spatial inductive biases. Recent research has focused on hybrid architecture designs that enhance feature representation by better information exchange between tokens [57, 58]. Furthermore, based on the latency analysis of different components in ViT-based models presented in the RIFormer [27], the token mixer occupies a significant portion of the total latency in the backbone network, accounting for 46.3%. Therefore, the latency introduced by the token mixer cannot be ignored. By replacing [59, 60] or even removing inefficient token mixers [27, 61], better performance can be achieved in lightweight network designs [62, 63].

In conclusion, although various lightweight techniques and methods based on deep learning have been proposed, their performance in specific downstream tasks may vary depending on the specific task and dataset characteristics. The pure transformer model is large in size and requires a large amount of data for training to outperform the CNN-based model, and the CNN is the mainstream approach on downstream tasks but lacks a global view, as shown in Table 2. To address these limitations, a hybrid approach is proposed that combines the strengths of both CNN and Transformer models. In this study, the primary motivation is to develop an efficient and lightweight network for human pose estimation. By employing a U-shaped structure with a context feature extraction module for local modeling and incorporating lightweight transformers for global modeling, UViT achieves an efficient and compact model with improved feature extraction capacity.

Table 2
Comparison of different neural network architectures

Architecture Type Advantages Disadvantages

CNN Architecture Strong capturing ability for local features; Efficient in processing image data, suitable for large-scale image processing tasks; Simple network hierarchy structure, easy to train and understand Limited modeling capability for long-range dependencies; Large number of parameters, high computational resource requirements; Poor performance in handling sequence data

Transformer Architecture Effective in modeling long-range dependencies; Explicit modeling of global information, suitable for sequence data processing tasks; Better interpretability and visualization capabilities Large number of parameters, high computational resource requirements; Potential excessive self-attention computations; Limited modeling capability for local features

CNN-Transformer Hybrid Architecture Combination of CNN and Transformer advantages, comprehensive modeling of local features and global information; Fewer parameters, and computational resource requirements Requires careful design and adjustment to balance the roles of both architectures; Performance limitations may exist for specific tasks and datasets

Architecture Type	Advantages	Disadvantages
CNN Architecture	Strong capturing ability for local features; Efficient in processing image data, suitable for large-scale image processing tasks; Simple network hierarchy structure, easy to train and understand	Limited modeling capability for long-range dependencies; Large number of parameters, high computational resource requirements; Poor performance in handling sequence data
Transformer Architecture	Effective in modeling long-range dependencies; Explicit modeling of global information, suitable for sequence data processing tasks; Better interpretability and visualization capabilities	Large number of parameters, high computational resource requirements; Potential excessive self-attention computations; Limited modeling capability for local features
CNN-Transformer Hybrid Architecture	Combination of CNN and Transformer advantages, comprehensive modeling of local features and global information; Fewer parameters, and computational resource requirements	Requires careful design and adjustment to balance the roles of both architectures; Performance limitations may exist for specific tasks and datasets

3 Method

3.1 Mixed pooling coordinate attention mechanism

The attention mechanism of SE [47] only considers channel coding, neglecting the importance of location information. Woo et al. [64] introduce spatial information coding through a large-scale convolution kernel; however, it can only capture local relationships and cannot model remote dependencies. GCNet [49], A2-Nets [65], SCNet [66], and CCNet [50] use nonlocal mechanisms to capture spatial information, however, the computation is extremely large to be suitable for lightweight networks. Coordinate Attention [51] can effectively obtain the global receptive field and encode precise location data. Inspired by this, the coordinate attention mechanism was improved to suit pose estimation tasks.

Consider an input feature map that can be viewed as a two-dimensional space where each position represents a pixel in the feature map. This feature map can represent an image or any other two-dimensional structured data. The coordinate attention mechanism determines the weights for each position based on its spatial coordinates and importance in the computation. To ensure the weights add up to one, the weights are normalized and subsequently applied to the corresponding pixel values in the input feature map, resulting in a weighted representation of each feature at each position. This process transforms the spatial coordinates into weights, which are then used to weight the corresponding positions in the feature map according to their importance. The coordinate attention mechanism enables the network to better focus on regions of interest and extract spatially relevant features.

In the initial feature aggregation stage, the mixed pooling strategy is adopted to better retain the texture and background features of the image, effectively improve the performance of visual tasks, and concatenate with the results of feature aggregation in the X and Y directions to enhance the feature representation. Therefore, given the input X, mixed pooling is used to encode in the horizontal and vertical directions in the (H, 1) and (1, W) spatial ranges and the output of the channel in the height h and width w directions can be expressed as $\begin{matrix} Z_{c}^{h} (h) = \frac{1}{W} \sum_{0 \leq i < W} [x_{Avg} (h, i), x_{Max} (h, i)] \end{matrix}$ (1) $\begin{matrix} \begin{matrix} Z_{c}^{h} (h) = \frac{1}{W} \sum_{0 \leq i < W} [x_{Avg} (h, i), x_{Max} (h, i)] \end{matrix} \end{matrix}$ (2)

where represents the concatenation operation along the spatial dimension and and denote the adaptive average pooling and adaptive maximum pooling operations, respectively. A concatenation operation is then performed on $Z_{c}^{h} (h)$ and $Z_{c}^{w} (w)$ , and they are sent to a shared 1 × 1 convolution transformation function F₁, yielding $\begin{matrix} Z = δ (F_{1} ([Z_{c}^{h} (h), Z_{c}^{w} (w)])) \end{matrix}$ (3) where δ represents the h-Swish [51] nonlinear activation function. Subsequently, Z is split into two independent tensors, $Z_{p}^{h}$ and $Z_{p}^{w}$ , along the spatial dimension. F_h and F_w are two 1×1 convolutional transformation functions that ensure consistency in the number of channels with the input X. $\begin{matrix} Z^{h} = σ (F_{h} (Z_{p}^{h})) \end{matrix}$ (4) $\begin{matrix} Z^{w} = σ (F_{w} (Z_{p}^{w})) \end{matrix}$ (5) where σ denotes the sigmoid nonlinear activation function. Finally, the outputs Z^h and Z^w are expanded and used as attention weights and multiplied by the input. The final output can be written as $\begin{matrix} y (i, j) = x (i, j) \times Z^{h} (i) \times Z^{w} (j) \end{matrix}$ (6)

The average pooling method weakens the performance of most distinctive features because the maximum pool method ignores the performance of some effective features. Therefore, based on the coordinate attention mechanism, using the mixed pooling strategy along the horizontal and vertical directions can maximize the retention of image texture features and obtain global context information, which helps locate the region of interest more accurately.

3.2 Context enhancement module

First, for the U2-Net [22] network structure, the first two coding stages are downsampled to one-quarter resolution using convolutional and Inverted Residual block [29] to reduce the computational and output at decoding to one-quarter resolution with the number of intermediate channels set to 32. Keypoint detection is then performed on the COCO val. For input resolutions of 256×192 and 384×288, the AP scores were 64.8 and 67.5, respectively, which are superior to MobilenetV2 in terms of accuracy, number of parameters, and FLOPs, demonstrating the powerful contextual feature extraction capability of the U-shaped structure. Inspired by this, a U-shaped network structure was redesigned to further enhance local modeling capability.

For a network with coding and decoding structures, the size of downsampling is often 32 times, which leads to the loss of a large amount of detailed information, resulting in unsatisfactory performance for downstream tasks. Dilated convolution [67] can effectively enlarge the receptive field while maintaining the height and width of the original input feature map. However, a gridding effect exists. According to the design criterion of the Hybrid Dilated Convolution (HDC) [67] expansion factor, dilation rates of [1 , 5] were set to obtain a large receptive field and a more uniform distribution of pixel utilization compared to the settings in U2-Net [22], effectively avoiding the gridding effect [68].

Based on the application of a mixed pooling coordinate attention mechanism and dilated convolution, a U-shaped structure was constructed with a large receptive field and enhanced representation of objects of interest. The improved attention mechanism is placed before the summing operation with the shortcut branch. When the sampling rate was 32, the pool operation was removed, and the expansion factor was reasonably designed according to HDC standards to enlarge the receptive field.

3.3 UViT-Residual block

The U-Net [39] network, proposed in 2015, is salient in the field of image segmentation. Its U-shaped structure addresses the disadvantage that an FCN [69] cannot capture context and location information. It adopts a completely symmetrical U-shaped structure to strengthen feature fusion, which retains the spatial information of the bottom features and exploits the rich semantic information of the high level.

Standard convolutional operations can be seen as a stack of three sequential operations: (1) unfolding, where the input image is decomposed into a matrix form, (2) matrix multiplication, which learns local features, and (3) folding, which recombines the feature maps into tensor form for further processing. Inspired by MobileViT [17], the UViT-Residual block replaces matrix multiplication with a stack of transformer layers to enable global modeling, giving the network convolution-like properties such as spatial bias. This modeling approach provides insights for designing shallow and narrow models, resulting in lightweight structures. For example, in the UViT-T model, the transformer dimensions and feed-forward network (FFN) dimensions are set to [64, 80, 96] and [128, 160, 192], respectively, in the three stages of the encoder. This configuration results in a low FLOPs of only 0.257 G, accounting for only 25% of the total FLOPs of the entire model. This indicates that the UViT model achieves lightweight design by setting dimension parameters appropriately, while still maintaining sufficient expressiveness and accuracy.

Inspired by this, the Context enhancement module was combined with the transformer to build a UViT-Residual block for local and global modeling. Based on the MobileViT [17] network structure and ablation experiment, patch size was set to 2 to better capture details, set the number of heads of the multi-head attention mechanism to 4, divide the model sizes according to the input dimensions of the transformer, and conduct ablation experiments in feature fusion mode. The use of the UViT-Residual block enhanced the network feature extraction capability without increasing the computational burden. Experiments showed that the UViT-Residual block is considerably friendly to lightweight networks.

3.4 Lightweight pose network

Multibranch networks are widely used in human pose estimation, and they can better integrate multiresolution features to address the problem of scale variation. U-Net [39] is widely used in the field of image segmentation to effectively obtain contextual and location information. Based on U-Net [39], U2-Net [22] adopts a two-layer nested U-shaped structure, which better fuses features with different scales and receptive fields and captures global information from different scales. The transformer network architecture has significant potential in the visual field; however, owing to the lack of spatial induction bias and large model parameters, ... . To better meet the needs of mobile deployment, the combination of CNN and Transformer gives full play to their respective advantages and provides ideas for building a lightweight and low-latency network for mobile vision tasks.

Based on the U2-Net design principles, a lightweight pose-estimation network (UViT) was proposed. In contrast to U2-Net, the 2D convolution and Inverted Residual block [29] are used to downsample to one-quarter resolution in the Stem stage and output at one-quarter resolution in the decoding stage, significantly reducing the number of parameters and the size of the FLOPs. Considering the high resolution, increased computational complexity, and redundant information at the first stage of the encoder (En_1), the enhanced U-shaped structure is utilized for contextual feature extraction. Additionally, dilated convolution is utilized for the low-resolution processing of this module to effectively expand the receptive field. The UViT-Residual block is used in encoder stages En_2, En_3, and En_4 to extract effective feature information by a joint transformer structure for local and global modeling. The model size was divided according to the input dimension of the transformer block to improve the global modeling capability because the U-shaped structure has a powerful contextual feature extraction capability. More details and other variants are presented in Table 3. The main body adopts the Encoder-Decoder structure, and the hybrid structure of CNN and Transformer is used in the encoding stage. Following upsampling in the decoding stage, the context information is fused through the skip connection structure. C_mid represents the number of intermediate channels, and Concat represents the concatenation operation.

Table 3
Architectures for UViT. The stem contains a stride 2 3×3 convolution and an Inverted Residual block. The main body adopts an encoder-decoder architecture

Layer Output size Operator Output Repeat Modules

channels UViT-T UViT-S UViT-B

Image 256×256 3 – – –

Stem 64×64 Conv2d 32 1 1 1 1

Inverted Residual block 64 1

En_1 32×32 U_Context Block 64 1 C_mid:16 C_mid:16 C_mid:16

En_2 16×16 UViT-Residual Block 64 2 Transformer dimension

[64, 80, 96] [96, 120, 144] [144, 192, 240]

En_3 8×8 UViT-Residual Block 64 4 Head dimension

[16 , 24] [24 , 36] [36 , 60]

En_4 8×8 UViT-Residual Block 64 3 FFN dimension

[128, 160, 192] [192, 240, 288] [288, 384, 480]

Decoding Stage 16×16 U_Context Fusion Block 128 1 – – –

32×32

64×64

GFLOPs 1.03 1.20 1.58

#Pramas 1.52M 2.27M 4.31M

Layer	Output size	Operator	Output	Repeat	Modules
Image	256×256		3	–	–	–
Stem	64×64	Conv2d	32	1	1	1	1
		Inverted Residual block	64	1
En_1	32×32	U_Context Block	64	1	C_mid:16	C_mid:16	C_mid:16
En_2	16×16	UViT-Residual Block	64	2	Transformer dimension
					[64, 80, 96]	[96, 120, 144]	[144, 192, 240]
En_3	8×8	UViT-Residual Block	64	4	Head dimension
					[16 , 24]	[24 , 36]	[36 , 60]
En_4	8×8	UViT-Residual Block	64	3	FFN dimension
					[128, 160, 192]	[192, 240, 288]	[288, 384, 480]
Decoding Stage	16×16	U_Context Fusion Block	128	1	–	–	–
	32×32
	64×64
GFLOPs					1.03	1.20	1.58
#Pramas					1.52M	2.27M	4.31M

4 Experiments

The proposed model was evaluated on the COCO [23] and MPII [24] datasets, ablation experiments were performed on related techniques, and the results were compared with those of other state-of-the-art methods.

4.1 COCO and MPII keypoint detection

4.1.1 Dataset and evaluation metric

Data from more than 200,000 photos and 250,000 pedestrians with 17 human keypoint-labeled information, with an average of two and a maximum of 13 people per image, make up the COCO dataset, a fairly large general-purpose dataset for computer vision tasks. UViT network is trained on the train2017 dataset, which contains 57,000 image data and 150,000 pedestrian messages, and the performance is evaluated on the val2017 and test-dev2017 datasets containing 5,000 and 20,000 images, respectively. Based on the keypoint similarity, object keypoint similarity (OKS) [23] was used to analyze the metrics. $\begin{matrix} OKS = \frac{\sum_{i} exp (- d_{i}^{2} / 2 s^{2} k_{i}^{2}) δ (v_{i} > 0)}{\sum_{i} δ (v_{i} > 0)} \end{matrix}$ (7) where, d_i is the Euclidean distance between the true and predicted values of each keypoint, v_i is the visibility marker of the true value, s is the target scale, and k_i is the keypoint constant for controlling the decay. The average precision (AP) and average recall (AR) were calculated based on the OKS metric.

The MPII dataset contains approximately 25,000 images and more than 40,000 human targets annotated with 16 nodes of information and head direction annotation, 3D torso, and body part occlusion. To evaluate the performance of the MPII dataset, the standard metric known as PCKh@0.5 [24] (head-normalized probability of correct keypoint) is utilized. This metric calculates the percentage of correctly localized keypointsby comparing the normalized distance between the detected keypoints and their corresponding ground truth values with a predetermined threshold. More details are shown in Table 4.

Table 4

Human keypoint detection dataset

Dataset	COCO	MPII
Purpose	Object detection, instance segmentation, keypoints	Human pose estimation, multi-person activity
Size	∼200K images, ∼250K annotated people	∼25K images, ∼40K annotated people
Annotations	Keypoint annotations (17 keypoints)	2D joint annotations (16 keypoints), activity labels
Challenges	Large variation in poses, occlusions, crowded scenes, scale changes	Diverse viewpoints, occlusions, non-rigid poses
Evaluation	Object Keypoint Similarity (OKS), Average Precision (AP), Average Recall (AR)	Percentage of Correct Keypoints (PCK), Percentage of Detected Joints

4.1.2 Training

The network was trained on an NVIDIA 3080Ti GPU with a minibatch size of 32 to update the parameters, employing a supervised learning approach using annotated keypoint data. The Adam optimizer with aninitial learning rate of 3e^-4 was adopted. To address the problem of training instability caused by an initial learning rate setting that was overly large, the learning rate was set using the warmup strategy, the warmup learning rate was set to 0.001, and the warm-up type was linear. A fixed aspect ratio of 4 : 3 was applied to the human detection boxes, and the boxes were subsequently cropped from the photos. Regarding COCO, the image was downsized to 256×192 or 384×288, and regarding MPII, it was downsized to 256×256. The data enhancement strategy includes random rotation ([-60°, 60°]), random scaling ([0.75, 1.25]), and a random flip probability of 0.5. The COCO dataset additionally adds a half-body transformation data enhancement strategy; the transformation threshold was set to 8, and the probability of performing a flip was set to 0.3.

4.1.3 Testing

Regarding the COCO dataset, the top-down method was used for human keypoint detection in the COCO val2017 human detection results, and regarding the MPII dataset, manually labeled target-bounding boxes were used for pose estimation. Using a post-Gaussian filter, the heat maps were estimated and the heat maps for the original and flipped images were averaged. By applying a quarter offset in the direction of the response with the highest and the second-highest response, each keypoint position was identified.

4.2 Results

4.2.1 Results on the validation set

Depending on the size of the transformer block input dimensions, three different versions of the lightweight pose estimation network (UViT-T, UViT-S, and UViT-B) were designed, as presented in Table 3. Table 5 presents the experimental results, where the smallest network outperforms other lightweight methods with an AP score of 68.5 after being trained from scratch with a 256×192 input size, only with 1.52M parameters and 1.03 GFLOPs. (i) The overall performance of the UViT network is better than that of the lightweight pose estimation network, which is close to a general-purpose network, but the parameters and FLOPs of the network are smaller. (ii) Compared to SimpleBaseline, there is a little gap (0.6 points) between UViT-B and the corresponding one (ResNet-50); however, the parameters and FLOPs of UViT-B are only 12.7% and 17.7%, respectively, of SimpleBaseline. (ii) Although HRNet has achieved good results, owing to the extensive use of parallel convolution, the speed of the model reasoning is significantly lower than that of the UViT lightweight network, and it is difficult to deploy on mobile devices, as shown in Fig. 3.

Table 5
Comparisons on the COCO val set. Parameters and GFLOPs are calculated solely for the pose estimation network, excluding the calculations for human detection and keypoint grouping tasks

Model Backbone Input size #Params GFLOPs AP AP⁵⁰ AP⁷⁵ AP^M AP^L AR

General-purpose networks

Hourglass [25] Hourglass-8 256×192 25.10M 14.3 66.9 – – – – –

SwinT [12] SwinT-t 256×192 32.77M 6.13 72.4 90.1 80.6 69.0 79.1 78.2

SimpleBaseline [5] ResNet-50 256×192 34.00M 8.90 70.4 88.6 78.3 67.1 77.2 76.3

TokenPose [16] TokenPose-L/D24 256×192 27.5M 11.0 75.8 90.3 82.5 72.3 82.7 80.9

RIFormer [27] HRFormer-B 256×192 43.2M 12.2 75.6 90.8 82.8 71.7 82.6 80.8

TransPose [15] TransPose-H-A4 256×192 17.30M 17.5 75.3 – – – – 80.3

HRNet-W32 [11] HRNet-W32 256×192 28.50M 7.10 73.4 89.5 80.7 70.2 80.1 78.9

Lightweight networks

MobileNetV2 [29] MobileNetV2 256×192 9.60M 1.48 64.6 87.4 72.3 61.1 71.2 70.7

MobileNetV2 [29] MobileNetV2 384×288 9.60M 3.33 67.3 87.9 74.3 62.8 74.7 72.9

ShuffleNetV2 [30] ShuffleNetV2 256×192 7.60M 1.28 59.9 85.4 66.3 56.6 66.2 66.4

ShuffleNetV2 [30] ShuffleNetV2 384×288 7.60M 2.87 63.6 86.5 70.5 59.5 70.7 69.7

Small HRNet [31] HRNet-W16 256×192 1.30M 0.54 55.2 83.7 62.4 52.3 61.0 62.1

Small HRNet [31] HRNet-W16 384×288 1.30M 1.21 56.0 83.8 63.0 52.4 62.6 62.6

UViT networks

UViT-T UViT-T 256×192 1.52M 1.03 68.5 87.8 76.2 65.4 74.9 74.5

UViT-T UViT-T 384×288 1.52M 2.32 70.6 88.4 77.7 67.2 77.2 76.2

UViT-S UViT-S 256×192 2.27M 1.20 69.4 88.2 77.0 66.4 75.5 75.2

UViT-S UViT-S 384×288 2.27M 2.70 71.6 88.8 78.8 68.3 78.1 77.2

UViT-B UViT-B 256×192 4.31M 1.58 69.8 88.6 77.6 67.0 75.8 75.7

UViT-B UViT-B 384×288 4.31M 3.57 72.3 89.1 79.4 68.8 78.8 77.7

Model	Backbone	Input size	#Params	GFLOPs	AP	AP⁵⁰	AP⁷⁵	AP^M	AP^L	AR
General-purpose networks
Hourglass [25]	Hourglass-8	256×192	25.10M	14.3	66.9	–	–	–	–	–
SwinT [12]	SwinT-t	256×192	32.77M	6.13	72.4	90.1	80.6	69.0	79.1	78.2
SimpleBaseline [5]	ResNet-50	256×192	34.00M	8.90	70.4	88.6	78.3	67.1	77.2	76.3
TokenPose [16]	TokenPose-L/D24	256×192	27.5M	11.0	75.8	90.3	82.5	72.3	82.7	80.9
RIFormer [27]	HRFormer-B	256×192	43.2M	12.2	75.6	90.8	82.8	71.7	82.6	80.8
TransPose [15]	TransPose-H-A4	256×192	17.30M	17.5	75.3	–	–	–	–	80.3
HRNet-W32 [11]	HRNet-W32	256×192	28.50M	7.10	73.4	89.5	80.7	70.2	80.1	78.9
Lightweight networks
MobileNetV2 [29]	MobileNetV2	256×192	9.60M	1.48	64.6	87.4	72.3	61.1	71.2	70.7
MobileNetV2 [29]	MobileNetV2	384×288	9.60M	3.33	67.3	87.9	74.3	62.8	74.7	72.9
ShuffleNetV2 [30]	ShuffleNetV2	256×192	7.60M	1.28	59.9	85.4	66.3	56.6	66.2	66.4
ShuffleNetV2 [30]	ShuffleNetV2	384×288	7.60M	2.87	63.6	86.5	70.5	59.5	70.7	69.7
Small HRNet [31]	HRNet-W16	256×192	1.30M	0.54	55.2	83.7	62.4	52.3	61.0	62.1
Small HRNet [31]	HRNet-W16	384×288	1.30M	1.21	56.0	83.8	63.0	52.4	62.6	62.6
UViT networks
UViT-T	UViT-T	256×192	1.52M	1.03	68.5	87.8	76.2	65.4	74.9	74.5
UViT-T	UViT-T	384×288	1.52M	2.32	70.6	88.4	77.7	67.2	77.2	76.2
UViT-S	UViT-S	256×192	2.27M	1.20	69.4	88.2	77.0	66.4	75.5	75.2
UViT-S	UViT-S	384×288	2.27M	2.70	71.6	88.8	78.8	68.3	78.1	77.2
UViT-B	UViT-B	256×192	4.31M	1.58	69.8	88.6	77.6	67.0	75.8	75.7
UViT-B	UViT-B	384×288	4.31M	3.57	72.3	89.1	79.4	68.8	78.8	77.7

Owing to the combination of CNN and Transformer, and the effective ability of context feature extraction and global modeling, the UViT network achieved a good balance between accuracy and computational complexity. The AP scores of the UViT-S and UViT-B reached 71.6 and 72.3, respectively.

4.2.2 Results on COCO test-dev set

Table 6 presents the pose estimation performance of UViT networks and other state-of-the-art methods on the COCO test-dev set. With an input size of 384×288, the proposed network accuracy significantly exceeds that of small networks, and it is superior in terms of network parameters and computational complexity. Compared with the general-purpose network, although the performance in terms of accuracy is slightly inferior, it is superior in terms of model size and FLOPs.

Table 6
Comparisons on the COCO test-dev set. Parameters and GFLOPs are calculated solely for the pose estimation network, excluding the calculations for human detection and keypoint grouping tasks

Model Backbone Input size #Params GFLOPs AP AP50 AP75 AP^M AP^L AR

General-purpose networks

CPN [8] Res-Inception 384×288 – – 72.1 91.4 80.0 68.7 77.2 78.5

SimpleBaseline [5] ResNet-50 256×192 34.00M 8.90 70.0 90.0 77.9 66.8 75.8 75.6

HRNet-W32 [11] HRNet-W32 384×288 28.50M 16.00 74.9 92.5 82.8 71.3 80.9 80.1

TransPose [15] TransPose-H-A4 256×192 17.30M 17.50 74.7 91.9 82.2 71.4 80.7 –

DARK [28] HRNet-W48 384×288 63.60M 32.90 76.2 92.5 83.6 72.5 82.4 81.1

TokenPose [16] TokenPose-L/D24 384×288 29.8M 22.10 75.9 92.3 83.4 72.2 82.1 80.8

IntegralRegression [7] ResNet-101 256×256 45.00M 11.00 67.8 88.2 74.8 63.9 74.0 –

Lightweight networks

MobileNetV2 [29] MobileNetV2 384×288 9.60M 3.33 66.8 90.0 74.0 62.6 73.3 72.3

ShuffleNetV2 [30] ShuffleNetV2 384×288 7.60M 2.87 62.9 88.5 69.4 58.9 69.3 68.9

LPN [32] ResNet-50 256×192 2.9M 1.0 68.7 90.2 76.9 65.9 74.3 74.5

Small HRNet [31] HRNet-W16 384×288 1.30M 1.21 55.2 85.8 61.4 51.7 61.2 61.5

UViT networks

UViT-T UViT-T 384×288 1.52M 2.32 70.2 90.5 78.0 67.0 75.9 75.7

UViT-S UViT-S 384×288 2.27M 2.70 71.2 90.9 79.2 68.0 76.9 76.7

UViT-B UViT-B 384×288 4.31M 3.57 71.8 91.1 79.7 68.7 77.4 77.2

4.2.3 Results on MPII validation set

Table 7 shows the comparison results between the UViT network and the lightweight pose estimation network. Compared with MobileNetV2 and ShuffleNetV2, UVIT-T achieved higher accuracy and lower FLOPS. With the increase of the model scale, UVIT-B reached the PCKh of 88.3, which was 1.3, 2.9, 5.5, and 6.1 points higher than Small HRNet-W16, ShuffleNetV2, MobileNetV2, and LiteHRNet-30 respectively.

Table 7
Comparisons on the MPII val set. Parameters and GFLOPs are calculated with an input size of 256×256

Model #Params GFLOPs PCKh

General-purpose networks

SimpleBaseline [5] 68.6M 20.9 91.5

HRNet-W32 [11] 28.5M 9.5 92.3

TokenPose-L/D6 [16] 21.4M – 90.2

TransPose-H-A6 [15] 17.5M – 90.3

Lightweight networks

MobileNetV2 [29] 9.60M 1.97 85.4

ShuffleNetV2 [30] 7.60M 1.70 82.8

Small HRNet-W16 [31] 1.30M 0.72 80.2

LiteHRNet-30 [33] 1.80M 0.42 87.0

UViT networks

UViT-T 1.52M 1.38 87.9

UviT-S 2.27M 1.59 88.1

UViT-B 4.31M 2.10 88.3

Model	#Params	GFLOPs	PCKh
General-purpose networks
SimpleBaseline [5]	68.6M	20.9	91.5
HRNet-W32 [11]	28.5M	9.5	92.3
TokenPose-L/D6 [16]	21.4M	–	90.2
TransPose-H-A6 [15]	17.5M	–	90.3
Lightweight networks
MobileNetV2 [29]	9.60M	1.97	85.4
ShuffleNetV2 [30]	7.60M	1.70	82.8
Small HRNet-W16 [31]	1.30M	0.72	80.2
LiteHRNet-30 [33]	1.80M	0.42	87.0
UViT networks
UViT-T	1.52M	1.38	87.9
UviT-S	2.27M	1.59	88.1
UViT-B	4.31M	2.10	88.3

4.3 Ablation Study

Ablation experiments were performed using the UViT-T network on the MPII and COCO datasets and the results were verified on the validation set; this included the UViT block feature fusion approach, multi-scale contextual information module design, and input resolutions of 256×256 and 256×192, respectively.

4.3.1 Feature fusion

Regarding the UViT block, as shown in Fig. 2, three feature fusion methods were designed; the results are listed in Table 8. F_(in+out) indicates that the input and output are fused by a convolutional layer with a convolutional kernel size of 3×3 after performing a concatenation operation in the channel direction; F_out indicates that the feature map is directly output by the UViT block; F_{(local+global)} indicates the contextual local information at different scales and the global information extracted by the transformer block after performing the concatenation operation in the channel direction and summing with the input features. The results indicated that the first fusion method works best with a similar number of parameters and computational complexity, with 1.4 and 1.6 higher AP scores than the second and third methods, respectively, and PCKh 1.2 and 1.5 higher, respectively. This indicates that preserving the detailed information of shallow features effectively enhances pixel-level tasks, and the concatenation operation with the input features is more conducive to the acquisition of high-quality features.

Fig. 2

Structure of UViT block. Take the UViT block of En_2 as an example on the MPII val set.

Table 8

Ablation about feature fusion strategy on the COCO val and MPII val sets. Params and GFLOPs are calculated for the different feature fusion methods. The input resolution of COCO is 256×192 and that of MPII is 256×256

Method	#Params	COCO					MPII
		GFLOPs	AP	AP50	AP75	AR	GFLOPs	PCKh
F _(in+out)	1.52M	1.03	68.5	87.8	76.2	74.5	1.38	87.9
F _out	1.52M	0.96	67.1	87.2	74.5	73.3	1.28	86.7
F _{(local+global)}	1.53M	1.05	66.9	86.8	73.9	73.0	1.39	86.4

4.3.2 Multiscale context module

The U-shaped structural design has significant advantages in the feature extraction of contextual information. To further enhance the attention of important features, an improved attention mechanism was introduced before the summation operation of the context extraction module to enhance the local modeling capability. Table 9 shows that using the coordinate attention mechanism improves the AP score by 0.8 points compared to not applying it, and using the improved coordinate attention mechanism performance improves the AP score by 1.3 points. Ablation experiments showed that using a hybrid pooling operation can effectively improve the performance of pixel-level tasks, such as pose estimation.

Table 9
Ablation about attention mechanism on the COCO val and MPII val sets. Params and GFLOPs are calculated for the different attention mechanisms. The input resolution of COCO is 256×192 and that of MPII is 256×256

Method #Params COCO MPII

GFLOPs AP AP50 AP75 AR GFLOPs PCKh

ATT _(no) 1.50M 1.03 67.2 87.3 74.8 73.4 1.37 87.4

ATT _(co) 1.52M 1.03 68.0 87.5 75.9 74.1 1.37 87.6

ATT _(mixpool) 1.52M 1.03 68.5 87.8 76.2 74.5 1.38 87.9

Method	#Params	COCO					MPII
ATT _(no)	1.50M	1.03	67.2	87.3	74.8	73.4	1.37	87.4
ATT _(co)	1.52M	1.03	68.0	87.5	75.9	74.1	1.37	87.6
ATT _(mixpool)	1.52M	1.03	68.5	87.8	76.2	74.5	1.38	87.9

4.4 Inference speed

ShufflenetV2 [30] indicates that computational complexity cannot be focused only on FLOPs, which is an indirect metric used to measure model complexity, and that inference speed is the most direct evaluation metric. In this section, the inference speed of the UViT is compared with that of a mainstream network under the same hardware conditions. Figure 3 shows the measurements of AP scores, inference speed, and FLOPs on the COCO dataset.

Fig. 3

Measurement of AP score, speed, and GFLOPs on the COCO val2017 with the UViT and SOTA methods. Two distinct colors signify various input sizes, 384×288 and 256×192. The size of a bubble indicates the FLOPs’ scope.

The high-resolution network HRNet [11] has a slow inference speed owing to the presence of numerous parallel convolutions, and SwinT [12] remains computationally intensive, although it uses the Windows multi-head self-attention (W-MSA) approach. The minimal network inference speed was approximately twice as fast as that of Simple152 [5], HRNet-W32 [11], and SwinT [12]. Therefore, this highly efficient and lightweight network could be deployed on edge devices.

4.5 Discussion

To address the shortcomings of the CNN lacking a global view and transformers lacking spatial location information and the high demand for memory and computational resources, a hybrid architecture design of a CNN and Transformer was adopted in this study. The proposed U-shaped structure context module enhances the local feature extraction capability and the UViT-Residual block enhances the global modeling capability. Moreover, the proposed UViT network has more advantages in terms of parameters and FLOPs, with slightly lower accuracy and faster inference on human-pose estimation tasks compared to general-purpose networks, and it significantly improved accuracy with lower parameters and FLOPs compared to lightweight networks. It is assumed to be because the proposed model architecture is designed to accurately capture contextual information at different scales while effectively modeling globally, which is more advantageous for vision tasks similar to pose estimation that require larger receptive fields. Despite achieving a balance between efficiency and lightweight design in human-pose estimation, its generalizability to other visual tasks requires further validation. Additionally, evaluating the universality and robustness of UViT necessitates further assessment of diverse datasets to fully understand the limitations of the UViT network.

5 Conclusion

In this study, a lightweight hybrid architecture of a CNN and transformer is proposed for the effective deployment of pose estimation tasks in edge devices. Considering the importance of contextual information, the U-shaped network structure was improved and an enhanced attention mechanism was incorporated to achieve contextual information modeling at different scales. To further expand the receptive field and enhance the global modeling capability, a transformer structure was combined with this, and a lightweight UViT-Residual block was designed to capture global information. Compared with methods such as the better-performing HRNet series and SwinT, the proposed method achieved good results on the COCO dataset while being more efficient in terms of the number of parameters, computational complexity, and inference speed. A lightweight design often leads to a decrease in accuracy. Reducing the number of network parameters and FLOPs for effective deployment on edge devices with guaranteed accuracy is a key issue to be addressed in the future.

Footnotes

Acknowledgments

This work was supported by the National Key R&D Program of China (2017YFF0210600), Anhui Provincial Department of Education University Scientific Research Project (2023AH052233, 2022AH051361, 2022AH051368), Suzhou University Scientific Research Project (2021fzjj20, 2023yzd14), and Suzhou university center for design and research magnetic information materials (2021XJPT15)

Conflicts of interest

The authors declare no conflict of interest.

References

Yan

, Xiong

and Lin

, Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition, 32nd AAAI Conference on Artificial Intelligence, AAAI 2018 (2018), 7444–7452.

Duan

, Zhao

, Chen

, Lin

, Dai

Revisiting Skeleton-based Action Recognition, in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2022): pp. 2959–2968.

and Langari

, Myoelectric human computer interaction using CNN-LSTM neural network for dynamic hand gesture recognition, Journal of Intelligent & Fuzzy Systems 44 (2023), 4207–4221.

, Yu

, Wang

, Ou

Multi-View Human Pose Estimation in Human-Robot Interaction, in IECON Proceedings (Industrial Electronics Conference), IEEE, (2020): pp. 4769–4775.

Xiao

, Wu

, Wei

Simple baselines for human pose estimation and tracking, in Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), (2018): pp. 472–487.

Bazarevsky

, Grishchenko

, Raveendran

, Zhu

, Zhang

, Grundmann

BlazePose: On-device Realtime Body Pose tracking, ArXiv: Computer Vision and Pattern Recognition, ArXiv: Computer Vision and Pattern Recognition (2020).

Sun

, Xiao

, Wei

, Liang

, Wei

IntegralHuman Pose Regression, in Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), (2018): pp. 536–553.

Chen

, Wang

, Peng

, Zhang

, Yu

, Sun

Cascaded Pyramid Network for Multi-person Pose Estimation, in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, (2018): pp. 7103–7112.

Huang

, Zhu

, Huang

, Du

AID: Pushing the Performance Boundary of Human Pose Estimation with Information Dropping Augmentation, Computer Vision and Pattern Recognition (2020).

10.

Cheng

, Xiao

, Wang

, Shi

, Huang

T.S.

, Zhang

HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation, in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2020): pp. 5385–5394.

11.

Sun

, Xiao

, Liu

, Wang

Deep High-Resolution Representation Learning for Human Pose Estimation, in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2019): pp. 5686–5696.

12.

Liu

, Lin

, Cao

, Hu

, Wei

, Zhang

, Lin

, Guo

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, (2021): pp. 9992–10002.

13.

, Zhang

, Tao

ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation, (2022).

14.

Carion

, Massa

, Synnaeve

, Usunier

, Kirillov

, Zagoruyko

End-to-End Object Detection withTransformers, in Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), (2020): pp. 213–229.

15.

Yang

, Quan

, Nie

, Yang

TransPose:Keypoint Localization via Transformer, in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), (2021).

16.

, Zhang

, Wang

, Yang

, Xia

S.-T.

, Zhou

TokenPose: Learning Keypoint Tokens for Human Pose Estimation, in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), (2021).

17.

Mehta

, Rastegari

MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer, Learning (2021).

18.

Zhang

, Hu

, Wang

ParC-Net: Position Aware Circular Convolution with Merits from ConvNets and Transformer, in Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), (2022): pp. 613–630.

19.

Liu

, Luo

, Li

, Lu

, Yang

ConvTransformer: A Convolutional Transformer Network for Video Frame Synthesis, (2020).

20.

Guo

, Han

, Wu

, Tang

, Chen

, Wang

, Xu

CMT: Convolutional Neural Networks Meet Vision Transformers, in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2022): pp. 12165–12175.

21.

, Xiao

, Codella

, Liu

, Dai

, Yuan

, Zhang

CvT: Introducing Convolutions to Vision Transformers, in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, (2021): pp. 22–31.

22.

Qin

, Zhang

, Huang

, Dehghan

, Zaiane

O.R.

and Jagersand

, U2-Net: Going deeper with nested U-structure for salient object detection, , Pattern Recognition 106 (2020), 107404.

23.

Lin

T.-Y.

, Maire

, Belongie

, Hays

, Perona

, Ramanan

, Dollar

, Zitnick

C.L.

Microsoft COCO: Common Objects in Context, in Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), (2014): pp. 740–755.

24.

Andriluka

, Pishchulin

, Gehler

, Schiele

2D Human Pose Estimation: New Benchmark and State of the Art Analysis, in 2014 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, (2014): pp. 3686–3693.

25.

Newell

, Yang

, Deng

Stacked Hourglass Networks for Human Pose Estimation, in B. Leibe, J. Matas, N. Sebe, M.Welling (Eds.), COMPUTER VISION –ECCV 2016, PT VIII, (2016): pp. 483–499.

26.

Xiao

, Wu

, Wei

Simple Baselines for Human Pose Estimation andTracking, in Lecture Notes in Computer Science, (2018): pp. 472–487.

27.

Wang

, Zhang

, Liu

, Wu

, Yang

, Liu

, Chen

, Luo

, Lin

RIFormer: Keep Your Vision Backbone Effective But Removing Token Mixer, (2023), 14443–14452.

28.

Zhang

, Zhu

, Dai

, Ye

, Zhu

Distribution-Aware Coordinate Representation for Human Pose Estimation, in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2020).

29.

Sandler

, Howard

, Zhu

, Zhmoginov

, Chen

L.-C.

MobileNetV2: Inverted Residuals and Linear Bottlenecks, in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, (2018): pp. 4510–4520.

30.

, Zhang

, Zheng

H.-T.

, Sun

ShuffleNet V2: Practical Guidelines for EfficientCNNArchitecture Design, in Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), (2018): pp. 122–138.

31.

Sun

, Zhao

, Jiang

et al., High-Resolution Representations for Labeling Pixels and Regions, ArXiv: Computer Vision and Pattern Recognition (2019).

32.

Zhang

, Tang

, Wu

Simple and Lightweight Human Pose Estimation, Computer Vision and Pattern Recognition (2019).

33.

, Xiao

, Gao

, Yuan

, Zhang

, Sang

, Wang

Lite-HRNet: A Lightweight High-Resolution Network, in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE, (2021): pp. 10435–10445.

34.

Toshev

, Szegedy

DeepPose: Human Pose Estimation via Deep Neural Networks, in 2014 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, (2014): pp. 1653–1660.

35.

Zhang

, Zhu

, Ye

Fast Human Pose Estimation, in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2019): pp. 3512–3521.

36.

Bulat

, Tzimiropoulos

Binarized Convolutional Landmark Localizers for Human Pose Estimation and Face Alignment with Limited Resources, in 2017 IEEE International Conference on Computer Vision (ICCV), IEEE, (2017): pp. 3726–3734.

37.

Wang

, Zhang

, Ren

, Wei

, Gao

and Liu

, UULPN: An Ultra-lightweight Network for Human Pose Estimation Based on Unbiased Data Processing, Neurocomputing 480 (2022), 220–233.

38.

Zhang

, Qiang

, Yang

, Wei

, Chen

and Chen

, Human Pose Estimation via an Ultra-Lightweight Pose Distillation Network, , Electronics 12 (2023).

39.

Ronneberger

, Fischer

, Brox

U-Net: Convolutional Networks for Biomedical Image Segmentation, in Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), (2015): pp. 234–241.

40.

Dai

, Qi

, Xiong

, Li

, Zhang

, Hu

, Wei

Deformable Convolutional Networks, in 2017 IEEE International Conference on Computer Vision (ICCV), IEEE, (2017): pp. 764–773.

41.

, Liu

, Tian

, Li

, Bao

, Fang

, Lu

Dual Attention Network for Scene Segmentation, in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2019): pp. 3141–3149.

42.

Yuan

, Huang

, Guo

, Zhang

, Chen

and Wang

, OCNet: Object Context for Semantic Segmentation, , International Journal of Computer Vision 129 (2021), 2375–2398.

43.

Liu

, Chen

, Wang

Quantitative Evaluation Model of the Quality of Remanufactured Product, IEEE Transactions on Engineering Management (2023), 1–12.

44.

Liu

, Chen

and Cai

, Data-driven remanufacturability evaluation method of waste parts, , IEEE Transactions on Industrial Informatics 18 (2022), 4587–4595.

45.

Zhang

, Yang

, Li

and Jiang

, Human pose estimation based on parallel atrous convolution and body structure constraints, Journal of Intelligent & Fuzzy Systems 42 (2022), 5553–5563.

46.

Wang

, Girshick

, Gupta

, He

Non-local Neural Networks, Computer Vision and Pattern Recognition (2017).

47.

, Shen

, Albanie

, Sun

and Wu

, Squeeze-and-Excitation Networks, , IEEE Transactions on Pattern Analysis and Machine Intelligence 42 (2020), 2011–2023.

48.

, Guo

, Sansom

, McGuire

, Kalaani

, Chen

, Tang

, Yang

and Fu

, Spatial Pyramid Attention for Deep Convolutional Neural Networks, , IEEE Transactions on Multimedia 23 (2021), 3048–3058.

49.

Cao

, Xu

, Lin

, Wei

, Hu

GCNet: Non-Local Networks Meet Squeeze-Excitation Networks and Beyond, in 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), IEEE, (2019): pp. 1971–1980.

50.

Huang

, Wang

, Huang

, Wei

, Liu

CCNet: Criss-Cross Attention for Semantic Segmentation, International Conference on Computer Vision (2019).

51.

Hou

, Zhou

, Feng

Coordinate Attention for Efficient Mobile Network Design, in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2021): pp. 13708–13717.

52.

Dosovitskiy

, Beyer

, Kolesnikov

et al., an Image Is Worth 16X16 Words: Transformers for Image Recognition At Scale, ICLR 2021 - 9th International Conference on Learning Representations (2021).

53.

Howard

A.G.

, Zhu

, Chen

, Kalenichenko

, Wang

, Weyand

, Andreetto

, Adam

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications, Computer Vision and Pattern Recognition (2017).

54.

Howard

, Sandler

, Chen

et al., Searching for MobileNetV3, in 2019 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, (2019): pp. 1314–1324.

55.

Xiao

, Singh

, Mintun

, Darrell

, Dollar

, Girshick

Early Convolutions Help Transformers See Better, in Adv Neural Inf Process Syst, (2021): pp. 30392–30400.

56.

Chu

, Tian

, Wang

, Zhang

, Ren

, Wei

, Xia

, Shen

Twins: Revisiting the Design of Spatial Attention inVision Transformers, Neural Information Processing Systems,Neural Information Processing Systems (2021).

57.

Guo

, Han

, Wu

, Tang

, Chen

, Wang

, Xu

CMT: Convolutional Neural Networks Meet Vision Transformers, in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE, (2022): pp. 12165–12175.

58.

, Xiao

, Codella

, Liu

, Dai

, Yuan

, Zhang

CvT: Introducing Convolutions to Vision Transformers, in Proceedings of the IEEE International Conference on Computer Vision, IEEE, (2021): pp. 22–31.

59.

Rao

, Zhao

, Liu

, Lu

, Zhou

, Hsieh

C.J.

DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification, in Adv Neural Inf Process Syst, (2021): pp. 13937–13949.

60.

, Luo

, Zhou

, Si

, Zhou

, Wang

, Feng

, Yan

MetaFormer is Actually What You Need for Vision, in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, (2022): pp. 10809–10819.

61.

, Si

, Zhou

, Luo

, Zhou

, Feng

, Yan

, Wang

MetaFormer Baselines for Vision, (2022).

62.

Sun

, Yu

, Song

, Liu

, Yang

, Zhou

MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices, in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, (2020).

63.

Jiao

, Yin

, Shang

, Jiang

, Chen

, Li

, Wang

, Liu

TinyBERT: Distilling BERT for Natural Language Understanding, in Findings of the Association for Computational Linguistics: EMNLP 2020, (2020).

64.

Woo

, Park

, Lee

J.-Y.

, Kweon

I.S.

CBAM: Convolutional Block Attention Module, in Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), (2018): pp. 3–19.

65.

Chen

, Kalantidis

, Li

, Yan

, Feng

A2-Nets: Double attention networks, in Adv Neural Inf Process Syst, (2018).

66.

Liu

, Hou

, Cheng

, Wang

, Feng

Improving Convolutional Networks With Self-Calibrated Convolutions, Computer Vision and Pattern Recognition (2020), 10093–10102.

67.

, Koltun

Multi-scale context aggregation by dilated convolutions, in 4th International Conference on Learning Representations, ICLR 2016 –Conference Track Proceedings, (2016).

68.

Wang

, Chen

, Yuan

, Liu

, Huang

, Hou

, Cottrell

Understanding Convolution for Semantic Segmentation, in 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), (2018).

69.

Shelhamer

, Long

and Darrell

, Fully Convolutional Networks for Semantic Segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence 39 (2017), 640–651.