Design of a multi-sensor information acquisition system for mannequin reconstruction and human body size measurement under clothes

Abstract

Although the measurement of human sizes and the reconstruction of mannequins have been extensively studied, there has been little research on systems applicable for measurement and reconstruction under clothes. This work presents a method for three-dimensional (3D) mannequin reconstruction and human body size measurement based on information fusion of multiple sensors. For this purpose, a multi-sensor information acquisition system (MIAS) is developed allowing one to acquire 11 key size data of the human body under clothes. The MIAS is based on the proposed parametric mannequin method and the Laplace mesh deformation technique, thus allowing one to reconstruct the mannequin conforming to the MIAS measurements. All mannequins reconstructed by this method have the same topology, which will greatly facilitate subsequent processing, such as garment design, customization and virtual fitting. The results of the reconstruction experiments show that our system can reconstruct a mannequin that matches the size of the feature parts very well. In addition, performed comparison experiments with manual and laser measurements shows that the measurement results of our system are close to those of manual and laser measurements. This provides a viable method for measuring key body sizes under clothes and reconstructing the corresponding 3D mannequin.

Keywords

Anthropometric data acquisition mannequin reconstruction mesh deformation

Obtaining human body sizes under clothes and the corresponding mannequin is a daunting task. For many applications, such as garment design, virtual fitting and tailoring, mannequins have an important role.^1–3 In the garment manufacturing industry, for instance, large-scale machine production has replaced manual production, which has greatly increased output and reduced manufacturing costs, but at the same time has created overproduction.⁴ Only garment customization can solve the production capacity, achieve zero inventory, guide accurate consumption and realize accurate manufacturing.⁵ The growing commercial value of garment design, manufacturing and sales have driven the rapid development of anthropometric and mannequin reconstruction technologies.⁶ At present, human body measurement can be roughly divided into contact measurement and contactless measurement. Contact measurement mainly relies on the experience of the measurer and the measurement method. When the measurer is different, the stability of the measured value is low.⁷ What is needed in the measurement is the net size so that the amount of relaxation can be added during the process of making the garment according to the use of the garment. Therefore, it is necessary to measure the subject without wearing clothes during the measurement, which makes the measurement difficult to perform. Anthropometric measurements are numerous and time-consuming. There is also potential for human error in data recording, collation and input. The above problems are not conducive to the collection and analysis of data from contact anthropometric techniques. In addition, with the development of technology, designers also increasingly prefer to use digital tools such as computer-aided design (CAD) to design garments, wherein three-dimensional (3D) mannequins reflecting real human body sizes are essential.⁸ However, with the increasing awareness of privacy protection, acquiring human body sizes and reconstructing the corresponding mannequin has been a painful and challenging problem for the garment industry. In this paper, we develop a new system allowing contactless measurement of the data of the human body under clothes and reconstruct the corresponding 3D mannequin.

Related work

The contactless anthropometric method has been developed since the 1980s, and has been greatly developed in the clothing industry in some countries. At present, there are mainly two-dimensional (2D) and 3D measurement methods.

The basic idea of the measurement method based on 2D human images is first to extract contour lines from 2D images of the human body in the front and side directions, and then to obtain the human body sizes based on the sizes in the contour map. Hung et al.⁹ performed key point extraction from images of the front, side and back of the human body, combining a square reference of known size to estimate the corresponding measured sizes in multiple views. The method is feasible but has a large measurement error. Kohlschütter and Herout¹⁰ used two calibrated images as input, redesigning the contour extraction algorithm and locating key points on the contours to finally calculate the neck, chest, waist and hip girths. Similarly, Lin et al.¹¹ use a combined image and contour method, where they first encoded human contours and later identified key points by comparing the differences in the encoded sequences. Li et al.¹² reconstructed a 3D human body by 2D multi-view and cut the 3D human body with a plane at a specified position to obtain the relevant perimeter length. The accuracy of 2D image-based anthropometry is often closely related to the quality of the image, the accuracy of the extracted body contour lines and the key points.

In the field of customization of clothing, a more common method is 3D anthropometry, which relies heavily on 3D body scanners. Raw scans typically contain millions of vertices, consume huge computational and memory resources, and may contain excessive noise.¹³ Many algorithms have been proposed by several scholars to handle 3D body scans. Zhong et al.,¹⁴ for instance, cut the 3D mannequin obtained from the real scan from top to bottom using a circular cutting method to obtain a large amount of anthropometric data. Dorin and Hurwitz¹⁵ used the a priori knowledge that the human body maintains a specific posture as a key point extraction. Then they used the found key points to roughly estimate the size of the dressed human body. Liu et al.¹⁶ proposed a GPU-Steger line detector to quickly detect the center and sub-pixel levels of laser patterns in order to obtain the 3D shape of the human body. Łukasz et al.¹⁷ provided a fully documented solution for automatic extraction of measurements from 3D body scans, test results and their validation, with excellent results. They found that the repeatability error of these 3D measurements can typically be controlled to within 1 cm.¹⁴

However, for a human body wearing heavy clothes, measurement methods based on 2D pictures and 3D scans fail because they cannot extract the body contour features under the clothes. In recent years, there has been an increasing interest in reconstructing mannequins. The 3D scanning of naked people (using commercial laser-based or structured light 3D body scanners, such as Cyberware and Human Solutions Vitus) can yield accurate data to generate accurate mannequins. However, commercial 3D body scanning systems are relatively expensive and unaffordable for average malls, while the process of scanning is inconvenient and unwelcome with increased awareness of privacy protection. It can also lead to mannequin errors and vulnerabilities. For example, shoppers in stores often feel offended when having their bodies measured according to a 3D scanner. Wearing a straitjacket is a good alternative, but still not feasible in many cases. Currently, there are two main methods of reconstructing mannequins under clothes: picture-based mannequin reconstruction and 3D scan-based mannequin reconstruction. Note that there is not really such a classification. Our classification is only for the reader to better understand the proposed method.

Picture-based mannequin reconstruction has always been a research topic where the parameters of statistical mannequins (such as SCAPE¹⁸ and SMPL¹⁹) are often estimated from images or videos. Bogo et al.,²⁰ for instance, used the static model to have relevance to the real human body. The positions of different joint points can reflect different body shapes and postures, and map the corresponding joint points of the SMPL¹⁹ to the image. Omran et al.²¹ predicted 12 semantic body segments based on the image. Then they processed the semantic part probability mapping to predict the parameters of the mannequin and finally mapped the pre-defined posture points in the SMPL¹⁹ to the 2D image. Kanazawa et al.²² sent the convolutional features of the image to the iterative 3D regression module. The goal of the iterative 3D regression module is to infer the 3D human body and the camera, and to project the 3D joints onto the labeled 2D joints. Choutas et al.²³ introduced the ExPose (EXpressive POse and Shape rEgression) method, which directly regresses the SMPL-X²⁴ format body, face and hand from the red-green-blue (RGB) image, and more accurately estimates the expressive 3D mannequin with a lower computational overhead. However, reconstruction results based on pictures are difficult to adopt for clothing customization, because some critical sizes of the reconstructed mannequin, such as the hip girth, waist girth, chest girth and even height, are highly unconstrained and uncontrollable.

The 3D scan-based method of mannequin reconstruction under clothes can often produce more accurate postures and shapes of mannequins than vision-based methods. The input is an incomplete human body point cloud with noise and holes. The output is a naked mannequin. Yu et al.²⁵ combined volumetric dynamic reconstruction with data-driven template fitting to simultaneously reconstruct detailed geometry, non-rigid motion and internal body shape from a single depth camera with good real-time performance. Yang et al.²⁶ used motion cues by encouraging the estimated body shape to be within the observation range to estimate the 3D body shape in motion from a series of unstructured directional 3D point clouds. Zhang et al.²⁷ used high-quality four-dimensional (4D) data and visual shell sequences extracted from multi-view images to estimate the body shape under clothes from a series of 3D scans. Neophytou and Hilton²⁸ introduced shape and posture spatial deformation (SPSD) technology to model the deformation caused by a specific posture from a 3D scan of the whole body registration. The 3D scanning-based mannequin reconstruction usually shows good accuracy when wearing a small number of clothes, but no naked part is still under-constrained. Wearing loose or heavy clothes, or having too many defects in the input 3D point cloud, can easily lead to failure or distortion in the reconstruction of the mannequin, which cannot be reliably applied to garment customization.

Nowadays, millimeter wave technology is widely used in airport security screening to detect objects hidden under clothes²⁹ and for obstacle recognition in autonomous driving technology.³⁰ Regarding the interaction with objects, although millimeter waves can easily penetrate dielectric materials, the photons do not have enough energy to break chemical connections in molecules or remove atoms. Therefore, the adverse health effects of millimeter wave radiation are considered to be negligible. Common clothing materials are transparent in this frequency range.³¹ As technology develops, the cost of millimeter wave sensors is gradually decreasing. In this context, we propose a new method to measure the sizes of the human body under clothes and reconstruct a mannequin. Specifically, we design a multi-sensor information acquisition system (MIAS) to obtain the sizes of a human body (including height, weight, shoulder width, arm length, chest width, chest thickness, waist width, waist thickness, hip width, hip thickness and leg length) under clothes by using weight sensors, infrared sensors, millimeter wave sensors and vision sensors. The corresponding mannequin is reconstructed based on a parametric mannequin and Laplace mesh deformation technique. It should be noted that the millimeter wave sensors used in this paper are narrow-beam and are used for distance measurements. It should also be emphasized that our system is safe, with less than one-thousandth of the sensor radiation of a cell phone.

Measuring sizes and reconstructing mannequins in this study may be useful for designers and industry, allowing them to consider the constraints of body size limitations during the initial garment design. For example, when customers buy clothes, they can quickly find the style and the size they want with convenient body measurements. They can also upload their reconstructed 3D mannequin and size data for clothing customization, or virtual fitting online.

Methodology

As previously mentioned, the MIAS can measure 11 anthropometric data. This data is selected here because some of the data can be directly measured by existing sensors and some of the data can be obtained with relatively accurate and stable values by using existing deep learning algorithms. Therefore, principal component analysis (PCA) dimensionality reduction is performed on the mannequin data set, and then the relationship between the PCA parameters of the male and female mannequins and the 11 anthropometric data are respectively learned to construct two parametric models of the human body for a male and a female. The measurement data of the MIAS is the input into the parametric model that is necessary to reconstruct the mannequin. Then, the reconstructed mannequin is used as the initial mannequin. Comparing the initial mannequin, except weight, with 10 MIAS measurements, the initial mannequin feature points are extracted to determine the modified positions, which are modified by the Laplace mesh deformation algorithm. The corresponding sizes are extracted in the modified mannequin. Instead of reconstructing the mannequin as in traditional 3D scanning, the method in this paper acquires the anthropometric data under the clothes through multiple sensors, calculates the parametric mannequin parameters and makes local modifications, thus eliminating the effect of clothing on the reconstruction of the mannequin. Some basic implementation details are explained in the following part of this section. Figure 1 illustrates the entire flow of the proposed method.

Figure 1.

Pipeline of the proposed three-dimensional mannequin reconstruction and size measurement method. PCA: principal component analysis.

MIAS and structure

The focus of this paper is the acquisition of human sizes under clothes based on the MIAS, which consists of three main parts: a bracket, a controller and a measurement module. Among them, the bracket is the overall mechanical frame of the test system, which is mainly used to support the measurement module. The controller mainly achieves the operation flow control of the entire test system and is installed on the bottom bracket. The measurement module is used to measure sizes of the human body and is mounted on two guide rails in an opposite arrangement, as shown in Figure 2.

Figure 2.

Physical structure of the multi-sensor information acquisition system: (a) three-dimensional model picture; (b) physical picture.

The measurement module is the core of the MIAS, which is divided into two parts: the moving up-and-down part and the fixed part. The fixed part is the weight sensor at the bottom of the bracket and the millimeter wave sensor at the top of the bracket. The internal structure of the up-and-down moving part is shown in Figure 3(a). The infrared sensor, millimeter wave sensor and vision sensor are mounted on the slider base. Four polyoxymethylene (POM) wheels are installed on the slider base to slide up and down in the groove of the guide rails. To ensure that the plane measured by the measurement module is at the same height, a timing pulley and a timing belt are used to establish the power transmission. The installation of the timing pulley and synchronous belt is shown in Figure 3(b). It should be noted that the vision sensor is installed in the measurement module on one side only. Both millimeter wave sensors and infrared sensors are used here for distance measurement, a weight sensor for weight measurement and a vision sensor for obtaining a full-body image of the subject. These sensors are selected based on cost and available materials, so there are bound to be better options. Some of the parameters of the sensors are shown in Table 1.

Figure 3.

Three-dimensional model of the up-and-down moving part of the measurement module: (a) front view; (b) exploded view of the back. POM: polyoxymethylene.

Table 1.

Detailed parameters of the sensors

Sensor name	Related performance parameters
60 GHz millimeter wave sensor (with antenna)	Range: 0.2–7.0 m, accuracy: 3 mm, beam angle: about 18°, sampling frequency: 60 ms
Binocular vision sensor	Resolution: 640 × 480, frame rate: 30 fps, Sampling frequency: 100 ms
Time of flight (TOF) infrared sensor	Range: 0.1–1.8 m, accuracy: 3 mm
Weight sensor	Resolution: 50 g

Measurement method and steps

In the measurement system (Figure 4) $H$ is the height of the platform; $l_{1}$ is the distance between the measurement plane of the millimeter wave sensor and the guide rail; $L$ is the distance between the guide rails on both sides; and $h_{0}$ is the starting position of the measurement module. When the height of the characteristic section is $h_{m}$ , the moving distance of the measurement module to the characteristic section is ${h_{m} - h}_{0}$ . At this time, the millimeter wave sensor measurement results on the front and rear sides are $d_{2}$ and $d_{3}$ , respectively. The width or thickness measured by the millimeter wave sensor of the corresponding section is $s_{1} = L - (2 l_{1} + d_{2} + d_{3})$ . Similarly, $l_{2}$ is the distance between the measurement plane of the infrared sensor and the guide rail. At this time, the measurement results of the infrared sensors on the front and rear sides are $d_{4}$ and $d_{5}$ , respectively, and the width or thickness measured by the infrared sensor of the corresponding section is $s_{2} = L - ({2 l}_{2} + d_{4} + d_{5})$ . Since the actual millimeter wave sensor and infrared sensor are installed very close to each other, $s_{2}$ must be larger than $s_{1}$ . The height is ${H - (d}_{1} + l_{1})$ . The approximate location of the characteristic height section is derived from the measured height ratio. A rectangle object of fixed width is used to calibrate millimeter wave sensors and infrared sensors before the measurement begins. Then the measurement module moves up and down in feature position to measure where the smallest value of width and thickness is taken for the waist, and the largest one is selected for the chest and the hip. The vision sensor takes multiple images of the human body during the up-and-down movement and stitches them together.

Figure 4.

Schematic picture of the measurement system.

The line segment scale invariant feature transform (SIFT) feature-based image stitching method proposed by Zhu and Ren³² is robust in terms of resolution, illumination, rotation and scaling. The (convolutional pose machines) CPM algorithm³³ for bone point detection merges convolutional neural networks into the pose machine framework to learn image features and image-based spatial models. Excellent results are achieved on its standard benchmark and the position of the skeletal point is closest to the actual measurement position. Therefore, we use the method proposed by Zhu and Ren³² to stitch the images collected by the vision sensor. Then the CPM algorithm³³ is used to detect the bone points of the stitched image. The Euclidean distance between the bone points is calculated by the ratio of the measured height to the height corresponding to the human body in the image, where the height of the human body in the picture is extracted with reference to He et al.³⁴ From this, the shoulder width, arm length and leg length can be obtained. Weight is measured directly by the weight sensor at the bottom of the bracket.

The measurement is divided into two simple steps (Figure 5). Step 1: with legs together and back straight, face the side of the rail with the visual sensor and move the position of the measurement module to complete the measurement of height, weight, hip thickness, waist thickness, chest thickness, leg length, shoulder width and arm length, as show in Figure 5(a). Step 2: hold the head with both hands and elbows pointing parallel to the body, then move the position of the measurement module to complete the measurement of chest width, waist width and hip width, as show in Figure 5(b). It should be noted here that (i) the measured height of the measurement module is estimated based on the proportion of the height at which the body features are located and (ii), due to the presence of the millimeter wave sensor beam angle, slight height errors will not have an effect on the measurement results.

Figure 5.

Measurement steps diagram: (a) frontal measurement; (b) side measurement.

Theoretically, the up-and-down moving part of the measurement module can be arranged in four directions, front and back, left and right, so that it can complete the measurement in one time. However, based on cost and development cycle considerations, we favor the method of two measurements. For the thickness measurement of the subject, the opening and closing of the arms do not have a significant impact on the measurement, while the width measurement of the subject requires the subject to hold his head with both hands and elbows pointing parallel to the body in order not to block the width measurement. It is found, through many measurement experiments, that this posture has a smaller effect on the human width. The simultaneous measurement of sensors arranged in the front and back can reduce the influence of unstable poses on the measurement during this process.

MIAS measurement data processing

Figure 6 illustrates the processing of millimeter wave sensor data for measuring a given position. The repeated measurement data for the measurement points are three sets, each with a size of 20, namely the set of distance values $X$ measured by the millimeter wave sensor, the set of millimeter wave sensor amplitude values $P$ and the set of distance values $Y$ measured by the infrared sensor. The same subscript indicates the same number of measurements. So, the data can be expressed as

\{\begin{array}{l} X = {x_{1}, x_{2}, \dots, x_{20}} \\ P = {p_{1}, p_{2}, \dots, p_{20}} \\ Y = {y_{1}, y_{2}, \dots, y_{20}} \end{array}

(1)

Figure 6.

Schematic diagram of measurement data processing for millimeter wave sensors.

Measurement data processing

(a) Measurement range filtering. Use the infrared sensor measurement data set $Y$ as the upper limit of the millimeter wave measurement data. The data larger than the upper limit in the millimeter wave sensor measurement values is filtered out. The value in the triangular box in Figure 6 is filtered.

(b) Reflective intensity low value filtering. In general, high amplitude measurements have a high degree of confidence. Therefore, set an amplitude threshold $z$ , which can be selected by the actual situation. Then filter the data in the $X$ that is lower than this amplitude threshold. The values contained in the rectangular boxes in Figure 6 are filtered.

(c) Coarse error filtering. Use standard deviation $σ$ as the discriminant criterion for coarse errors. The data in $X$ that exceed the discriminant criterion are filtered out. The value in the circle in Figure 6 is filtered.

The average of the final filtered data is used as the final measurement result.

Building the parametric mannequin

We learn parametric mannequins for males and females from the CAESAR³⁵ data set (approximately 2000 scans per gender), where each mannequin has the same number of vertices. Referring to the method proposed by Allen et al.³⁶ and assuming that each mannequin has $n$ vertices, the geometry of a human body can be represented by a vector composed of 3 $n$ floating-point numbers. This vector is denoted as $s_{i}$ . If there are $m$ mannequins, let the average vector of $\{s_{i}\}$ be $E_{s}$ , $u_{i} = s_{i} - E_{s}$ , where $\{u_{i}\}$ describes the geometric change of these $m$ mannequins. This is a high-dimensional vector and PCA can be used to reduce the size of $\{u_{i}\}$ , for example, down to $k$ size. Suppose the main directions of PCA are $D_{1}, D_{2}, {\dots, D}_{k}$ . Thus, the human body geometry can be expressed as $S = E_{s} + ω_{1} D_{1} + ω_{2} D_{2} + \dots + ω_{k} D_{k}$ . Consequently, a set of parameters $ω = \{ω_{1}, ω_{2}, \dots, ω_{k}\}$ can represent the human body geometry. By setting $ω$ , a new human body geometry can be synthesized. Figure 7 demonstrates the process of parameter changes on the mannequin. The texture of the skin and the detailed information of the face are removed after dimensional reduction.

Figure 7.

Deformation between the parametric mannequins. The black outline marks the mannequins obtained by dimensionality reduction in the CAESAR³⁵ data set. The two mannequins in the middle of the black outlined mannequins are created by the linear difference of the parameters.

Backpropagation neural network for parameter prediction

The setting of mannequin parameter $ω$ is not intuitive. Assuming that we can measure $p$ data from the actual human body, that is, the data that can be measured from the MIAS, let these meaningful parameters be $C = \{c_{1}, c_{2}, \dots, c_{p}\}$ . Since the mapping from $C$ to $ω$ is obviously nonlinear, we used the backpropagation (BP) neural network³⁷ to build a $C$ to $ω$ mapping $M$ from this human body data, denoted as $M_{C} = ω$ .

The BP neural network is a multilayer feed-forward neural network trained according to the error back propagation algorithm. In the forward transmission, the input signal is transmitted from the input layer to the hidden layer up to the output layer. The state of neurons in each layer affects only the next layer. If the output layer does not obtain the expected output, the output layer continues to transmit in the reverse direction until the expected output is obtained. The strength and threshold of the connections between the input and hidden layer nodes and between the hidden layer nodes and output nodes are adjusted according to the prediction error so that the error decreases along the gradient direction. After repeated learning training, the network parameters corresponding to the minimum error are determined.

The inputs are the $p$ mannequin measurements ( $c_{1}, \dots, c_{p}$ ) and the outputs are the predicted values of the $k$ mannequin parameters ( $y_{1}, \dots, y_{k}$ ). The topology of the BP neural network is shown in Figure 8. Since there is a large difference in the size of data between different anthropometric data, such as height and weight, it is necessary to perform normalization operations on the input data. The hidden layer threshold and the output layer threshold are a and b, respectively, $W_{i, j}$ is the connection weight of the input layer and the hidden layer and $W_{j, t}$ is the connection weight of the hidden layer and the output layer. The number of neurons in the hidden layer can be determined by the empirical formula (2)

m = \sqrt{p + k} + l

(2)

where

m

is the number of hidden layer nodes,

p

is the number of input layer nodes,

k

is the number of output layer nodes and

l

is an adjustment constant between 1 and 10.

Figure 8.

Backpropagation neural network structure diagram.

Training process

The BP neural network with nonlinear prediction output capability needs to be trained before making predictions (Figure 9).

Figure 9.

Backpropagation neural network training processes.

The training process includes the following seven steps.

Step 1: network initialization. Initializing the relevant parameters of the BP neural network according to the variable sequence of the input and output ( $X, ω$ ). For ease of description, here $X$ is the set of normalized mannequin measurement data ( $x_{1}, \dots, x_{p}$ ) and $ω$ is the set of output mannequin parameters. The relevant parameters of the BP neural network are shown in Table 2.

Table 2.

Parameters of the backpropagation neural network

Name of the parameter	Parameter symbol
Input neurons	$p$
Output neurons	$k$
Hidden neurons	$m$
Input-hidden weights	$W_{i, j}$
Hidden-output weights	$W_{j, t}$
Hidden threshold	$a$
Output threshold	$b$
Learning rate	$η$
Activation function	$f$

Step 2: calculation of the value of the hidden layer outputs. Inputs variable $X$ , connection weights $W_{i, j}$ and hidden threshold $a$ are needed to calculate the value of hidden layer $H_{j}$

H_{j} = f (\sum_{i = 1}^{p} W_{i j} x_{i} - a_{j}) j = 1, 2, \dots, m

(3)

where

m

represents the hidden layer neurons and

f

is the activation function of the hidden layers. The rectified linear unit (ReLU) function is chosen as the activation function in this paper, which takes the form

f (x) = \{\begin{matrix} 0, x \leq 0 \\ x, x > 0 \end{matrix}

(4)

Step 3: calculation of the values of the output layer. The hidden layer output $H_{j}$ , the connection weights $W_{j, t}$ and the output layer threshold $b$ can calculate the predicted outputs $O_{t}$ . The latter are compared with mannequin parameters $ω$

O_{t} = \sum_{j = 1}^{m} H_{j} W_{j, t} - b_{t} t = 1, 2, \dots, k

(5)

Step 4: error calculation. The prediction error $e_{t}$ is calculated by subtracting the predicted output $O_{t}$ from the expected output $ω_{t}$ . When the error is small enough, it indicates that the parameters of the network are already good. This is the basis for stopping the training

e_{t} = ω_{t} - O_{t} t = 1, 2, \dots, k

(6)

Step 5: update weights. Update the network connection weights ( $W_{i, j}, W_{j, t}$ ) with network prediction errors $e_{t}$

W_{j, t} = W_{j, t} + η H_{j} e_{t} j = 1, 2, \dots, m; t = 1, 2, \dots, k

(7)

W_{i j} = W_{i j} + η H_{j} (1 - H_{j}) x (i) \sum_{t = 1}^{k} W_{j, t} e_{t} i = 1, 2, \dots, p; j = 1, 2, \dots, m

(8)

Step 6: update the thresholds. The thresholds ( $a, b$ ) of the network nodes are updated according to the network prediction error $e_{t}$

a_{j} = a_{j} + η H_{j} (1 - H_{j}) \sum_{t = 1}^{k} W_{j, t} e_{t} j = 1, 2, \dots, m

(9)

b_{t} = b_{t} + e_{t} t = 1, 2, \dots, k

(10)

Step 7: judge whether the iterative algorithm has reached convergence, that is, the error $e_{t}$ is no longer decreasing. If not, go back to step 2.

As previously mentioned, the MIAS provides 11 anthropometric data. The number of parameters of the parametric mannequin is five, which retains about 86% of the original mannequin accuracy. The initial mannequin is reconstructed by predicting these five parameters from the MIAS measurements by the constructed BP neural network. (The number of nodes of the BP neural network is 11 nodes in the input layer, 20 nodes in the hidden layer and five nodes in the output layer.) In this way, we can reconstruct a mannequin from a set of anthropometric data and all the resulting mannequins have the same topology. Since this is a regression prediction task, it is inevitable that the anthropometric data of the reconstructed mannequin will have an error with the expected data. Therefore, we use the reconstruction results as the initial mannequin. The next section presents the modifications of the initial mannequin.

Modification of the mannequin based on Laplace mesh deformation

This section focuses on the correction of the initial mannequin reconstructed in the previous section. A flow chart of the modification process is shown in Figure 10. Firstly, the measurement data of the MIAS is compared with the measurement data of the reconstructed initial mannequin to get the error between them. Then, the feature points are extracted from the initial mannequin and used with its surrounding points as control points to obtain the modified mannequin by Laplace mesh deformation.

Figure 10.

Frame diagram of the modified initial mannequin based on Laplace mesh deformation. MIAS: multi-sensor information acquisition system.

Mesh editing algorithms are dedicated to achieve overall changes in the mesh while maintaining its local characteristics. The Laplace mesh deformation algorithm³⁸ is suitable for this task because it can encode the local geometric properties of the mesh. Laplacian coordinates are the basis of the Laplacian mesh editing framework.

Denote the triangular mesh model as $M = (E, V)$ , where $E$ is the set of edges and $V$ is the set of vertices. The set of vertices $V = (v_{1}, \dots, v_{n})$ is the 3D position information of all vertices in the mesh model, where $n$ is the number of vertices in the model. The Laplacian coordinate $δ_{i}$ of point $v_{i}$ can be expressed as

δ_{i} = (δ_{i}^{x}, δ_{i}^{y}, δ_{i}^{z}) = v_{i} - \frac{1}{d_{i}} \sum_{j \in N (i)} v_{j}

(11)

where

N (i) = {j ∣ (i, j) \in E}

is the set of all adjacent vertices of

v_{i}

;

d_{i} = | N (i) |

is the number of adjacent vertices of vertex

v_{i}

, that is, the degree of vertex

v_{i}

Let $V$ be the Cartesian coordinate matrix of the model vertices and $δ$ the Laplace coordinate matrix of the model vertices. The Cartesian coordinate matrix of the model needs to be transformed into the Laplace coordinate matrix before deformation. The linear transformation between the Cartesian and Laplace coordinate systems can be expressed in the form of a matrix, $δ = L V$ , where L is the Laplace operator, which is the transformation matrix between the Cartesian and Laplace coordinates. It can be expressed as $= I - D^{- 1} A$ , where $I$ is the unit matrix, $D$ is the diagonal matrix ( $D_{i i} = d_{i}$ ) and $A$ the adjacency matrix

A_{i j} = \{\begin{matrix} 1, (i, j) \in E \\ 0, o t h e r w i s e \end{matrix}

(12)

Let the set of control points consisting of $t$ control points $c_{j}$ be $C$ , that is, $c_{j} \in C, j \in [1, t]$ . These control points are then used as reference points for the model deformation and also as constraints for Equation (13)

(\frac{L}{ω I_{t \times t} ∣ 0}) V = (\begin{matrix} δ \\ ω c_{1 : t} \end{matrix})

(13)

The system matrix in Equation (13) is denoted by $L^{'} = (\frac{L}{ω I_{t \times t} ∣ 0})$ , where the weight $ω$ denotes the weights of the control points, which characterize the strength of the constraints. To smooth the mesh deformation, the value of $ω$ can be 1. The equation with added constraints makes the number of equations in the linear system greater than or equal to the number of unknowns, so this linear system is full rank. From Equation (13), an approximate least squares solution of the following equation can be obtained

V^{'} = argmin (∥ L V - δ ∥^{2} + \sum_{j \in C} {|v_{j} - c_{j}|}^{2})

(14)

where

V^{'}

is the new Cartesian coordinate value of the vertices on the deformed mesh.

We refer to the algorithm proposed by Zhong et al.¹⁴ to extract the sizes of the initial mannequin and to locate its 21 feature points (the extraction positions are shown in Figure 11), and to set the feature points and points in their neighborhoods as control points we then use the actual measurement values as constraints to finally deform the initial mannequin into a modified mannequin that conforms to the measurement values.

Figure 11.

Schematic diagram of mannequin feature point extraction: (a) front view; (b) rear view.

Figure 12 demonstrates the results of the mesh deformation for each feature area. The black parts are the deformed parts. It is seen that the deformation of one feature part does not affect the others.

Figure 12.

The Laplacian mesh deformation illustration of the characteristic part of the initial mannequin: (a) and (a)* represent the front and side of the original mannequin, respectively; (b) the arm lengthened mannequin; (c) the shoulder width and length mannequin; (d) the chest width and width mannequin; (e) the waist width mannequin; (f) the hip width mannequin; (g) the thickened chest mannequin; (h) the thickened waist mannequin; (i) the thickened hip mannequin.

Experimental results and related discussion

Subjects and their distribution

The test study was conducted on 100 unpaid volunteers between the ages of 20 and 65 years from different regions of China. Due to the nature of the MIAS, the thickness and amount of clothing worn do not affect the measurement, so subjects are only asked to wear clothing that does not contain metallic objects. The distribution of age, gender and body mass index (BMI) of these subjects are shown in Figure 13.

Figure 13.

Age, gender and body mass index (BMI) distributions of the 100 subjects.

Mannequin reconstruction experiment

Figure 14 shows the process of reconstructing the final modified mannequin from the MIAS measurement values. The final modified mannequin is compared with the initial one after best-fit alignment in the form of a color map.

Figure 14.

Multi-sensor information acquisition system (MIAS) measurement value reconstruction process for different samples (a) and (b).

After Laplace mesh deformation modification, the feature part model measurements correspond exactly to the MIAS measurements. Therefore, the sizes of the parts of interest can be extracted on this modified mannequin. However, the magnitude of the initial mannequin modification corresponds to the errors. When the error is large, Laplace mesh deformation will cause more obvious bulging or sinking of the flat area, such as the hip thickness position of the female modified mannequin in Figure 14(b), while for the width direction, the mesh deformation between different feature parts is still relatively smooth. Therefore, this modification algorithm is not very suitable for making substantial modifications to the initial mannequin.

Anthropometric comparison experiment

The manual measurement tools are as follows: (i) a measuring tape, which is used to measure the subject's shoulder width, arm length, leg length and girth sizes; (ii) an F-ruler, modified by a right-angle ruler and slider itself, with an accuracy of 1 mm, which can measure the width and thickness of the human body; (iii) a common height and weight scale. The subjects’ measured items are the height, weight, shoulder width, arm length, chest width, chest thickness, waist width, waist thickness, hip width, hip thickness, leg length, chest girth, waist girth and hip girth. The manual measurement of the human body method is referenced to the standard ISO 8559-1:2017.³⁹ To minimize human error, we perform three measurements on the same subject and take the average value as the final one.

The difference between the manual measurements and the feature sizes extracted from the mannequin reconstructed from the MIAS measurements is defined as an error. We extract the height, shoulder width, arm length, chest width, chest thickness, waist width, waist thickness, hip width, hip thickness, leg length, chest girth, waist girth and hip girth from the mannequin, because these indicators are constrained in the initial measurement. The comparison results are shown in Table 3, and the systematic measurements for males and females are generally close to the manual measurements. Note that the measurement errors for the shoulder width, arm length and leg length are relatively large. These measurement values are calculated after feature point recognition by picture stitching. In our opinion, these relatively large errors are due to three main causes: one is the distortion generated by the picture stitching acquired by the vision sensor; the second is that the feature point recognition algorithm has certain errors in the recognition of human feature points; and the third is a certain deviation between the position of the extracted sizes from the human model and the actual manually measured position.

Table 3.

Errors in manual and systematic measurements (length error in cm, weight error in kg, numbers are listed as male/female)

	Mean	Maximum	Minimum	Standard deviation
Height error	0.3/0.4	0.9/0.7	0.1/0.1	0.6/0.5
Weight error	0.2/0.2	0.3/0.3	0.0/0.0	0.1/0.1
Shoulder width error	1.2 /1.1	2.1/2.2	0.5/0.2	1.7/1.7
Arm length error	1.2/0.9	2.3/2.0	0.2/0.1	1.9/1.8
Chest width error	0.4/0.5	1.2/1.0	0.0/0.1	0.7/0.8
Chest thickness error	0.5/0.4	1.0/1.1	0.1/0.0	0.9/0.8
Waist width error	0.4/0.2	1.2/0.6	0.0/0.1	1.0/0.5
Waist thickness error	0.2/0.3	1.0/0.9	0.1/0.0	0.8/0.7
Hip width error	0.3/0.3	0.9/1.1	0.2/0.0	0.7/1.0
Hip thickness error	0.4/0.3	1.1/0.7	0.1/0.1	0.9/0.6
Leg length error	1.0/0.9	2.2/2.4	0.2/0.2	1.8/2.2
Chest girth error	1.1/1.4	2.3/2.1	0.2/0.2	1.8/1.8
Waist girth error	1.4/1.0	1.9/1.8	0.0/0.1	1.5/1.6
Hip girth error	1.2/1.4	2.1/2.5	0.1/0.2	1.7/0.7

The measurement results of 100 subjects wearing tighter clothing using the MIAS are statistically analyzed and compared with a high-precision laser sensor to measure the characteristic areas. Since the laser here uses point-by-point measurement, only the data of the height, shoulder width, chest width, chest thickness, waist width, waist thickness, hip width, hip thickness, chest girth, waist girth and hip girth are compared. Table 4 shows the differences between laser-based measurements (A) and MIAS-based measurements (B) on the same part of the mean value.

Table 4.

Comparison of laser measurement and multi-sensor information acquisition system (MIAS) measurement data (A is laser-based, B is MIAS-based, length in cm, numbers are listed as male/female)

Measurement area	A	B	Difference (A–B)
Height	174.3/165.5	173.1/164.5	1.2/1.0
Shoulder width	43.5/40.1	40.3/37.2	3.2/2.9
Chest width	28.8/30.6	30.2/31.4	–1.4/–0.8
Chest thickness	20.4/24.6	20.8/23.9	–0.4/0.7
Waist width	30.0/26.8	30.3/27.2	–0.3/–0.4
Waist thickness	20.4/18.0	20.2/18.3	0.2/–0.3
Hip width	33.4/35.8	33.1/35.4	0.3/0.4
Hip thickness	24.2/24.2	23.4/24.5	0.8/–0.3
Chest girth	83.2/95.5	85.8/98.0	–2.6/–2.5
Waist girth	80.3/71.5	78.9/73.3	1.4/–1.8
Hip girth	92.0/97.5	90.3/95.4	1.7/2.1

As can be seen in Tables 3 and 4, the overall difference between the MIAS measurements and the manual or laser measurements is not significant. The difference between the two measurements of shoulder width in Table 4 is large, and there may be errors in the determination of body parts. The error between the MIAS and manual measurement or laser measurement of both is around 2 cm, or less than 2 cm, which is within the allowable error range. During the measurements, we notice that the millimeter wave sensors are close to the manual and laser measurements in flat areas (e.g., waist thickness, hip width) and their data fluctuations are less during data processing. It is also noted that body leaning can also have a large effect on the measurement results, so there is a relatively high requirement for the subject to stand during the measurement.

Time consumption

The proposed MIAS takes about 11 minutes to measure the body sizes and reconstruct the mannequin, of which about 2 minutes is taken to measure the body data and 9 minutes is taken to reconstruct and modify the mannequin. We notice that the process of reconstructing the initial mannequin is rapid enough, about 10 seconds, while modifying the initial mannequin is more time-consuming. It takes 2 minutes to construct the Laplace matrix and Laplace coordinates, and about 7 minutes to solve Equation (14). All the algorithms for data processing, reconstruction and modification of the mannequin are written based on the Python programming language and implemented on a laptop with a 1.9 GHz octa-core processor and 8 GB of RAM.

Methodology discussion

By arranging multiple sensors, 11 key body data are obtained directly or indirectly and input into the constructed parametric mannequin to reconstruct the initial mannequin. Finally, the modified mannequin is obtained by adjusting the initial mannequin with the Laplace deformation algorithm based on the measured body data. Each item of body data is derived from the modified mannequin. From the above reconstruction results and measurement data, the proposed method can restore the body shape of the human body under clothes to a certain extent, and can relatively accurately obtain the size of human body features. Since the original input to the proposed method is the measured net body sizes and the mannequin is modified according to these sizes, there are more constraints on the feature parts than in the mannequin reconstructed by deep learning, and the sizes of these parts are closer to those of the real human body.

Conclusion

The study is stimulated by the limitations of existing anthropometric techniques that do not allow accurate measurement of body sizes under clothes. In comparison to the analyzed literature, different methods for measuring and reconstructing the mannequin under clothes are proposed. Our work develops a new system for under clothes body size measurement and corresponding mannequin reconstruction. For this purpose, a MIAS is designed to acquire the body size under the clothes. The MIAS can obtain 11 key body size data by scanning twice from the front and side and inputting the measured body size data into the constructed parametric mannequin to get the initial mannequin. After the initial mannequin is compared with the MIAS measurements, the feature points are extracted to determine the modification parts and the modification magnitude. Then the feature parts are modified using the Laplace mesh deformation algorithm. Finally, the various measurements are obtained from the modified mannequin. Mannequin reconstruction experiments show that the MIAS can reconstruct a mannequin that meets the size of the characteristic part. By comparing with manual measurement and laser measurement, the results show that the proposed system can obtain the size of the characteristic parts of the human body relatively accurately and quickly. It can measure key body sizes and reconstruct the corresponding mannequin without the subject taking off clothes, and has considerable accuracy. The system is currently in the prototype system stage, and supplementary constraints on the human body under clothing can be added in the future to make the reconstructed mannequin of the human body closer to the real human body. Therefore, our system is expected to be promoted in a number of situations where body sizes can be measured and corresponding mannequins reconstructed without taking off clothes.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship and/or publication of this article.

ORCID iD

Guiqin Li

References

Xie H and Zhong Y. Structure-consistent customized virtual mannequin reconstruction from 3D scans based on optimization. Text Res J 2019; 90: 937–950.

Sonntag

Zebrowski

Neidlin

, et al. Virtual fitting and hemodynamic simulation of the EVAHEART 2 left ventricular assist device and double-cuff tipless inflow cannula. ASAIO J 2019; 65: 1.

Shi

Gao

Wang

, et al. Automatic 3D virtual fitting system based on skeleton driving. Vis Comput 2021; 37: 1075–1088.

Wilson

Building apparel manufacturing competitiveness through policy–a system dynamics approach. J Fash Market Manag. Epub ahead of print 26 May 2020. DOI: 10.1108/JFMM-10-2019-0243.

, et al. Personalized 3D mannequin reconstruction based on 3D scanning. Int J Clothing Sci Technol 2018; 30: 159–174.

Barboza

Carvalho

Ferreira

, et al. Main characteristics and anthropometrics of people with down syndrome – impact in garment design. In: G Di Bucchianico and PF Kercher (eds.) International Conference on Applied Human Factors and Ergonomics, Los Angeles, USA, 17-21 July 2017, paper no.587, pp.417–427. Cham: Springer.

Vitali

Rizzi

Acquisition of customer’s tailor measurements for 3D clothing design using virtual reality devices. Virt Phys Prototyp 2018; 13: 131–145.

Singh

Motif placement on garment pattern: comparison and development of a CAD tool. Int J Eng Comput Sci 2017; 6: 20658–20661.

Hung

PC-Y

Witana

Goonetilleke

RS.

Anthropometric measurements from photographic images. Comput Syst 2004; 29: 764–769.

10.

Kohlschütter

Herout

Automatic human body parts detection in a 2D anthropometric system. In: G Bebis et al. (eds.) international symposium on visual computing, Crete, Greece, 16-18 July 2012, paper no. LNCS 7432, pp. 536–544. Berlin: Springer.

11.

Lin

Y-L

, et al. Automated body feature extraction from 2D images. Exp Syst Appl 2011; 38: 2585–2591.

12.

Jia

Mao

, et al. Anthropometric body measurements based on multi-view stereo image reconstruction. In: D Panescu and D Haemmerich (eds.) proceedings of the annual international conference of the IEEE Engineering in Medicine and Biology Society, Osaka, Japan, 3-7 July 2013, pp. 366–369. Piscataway: IEEE.

13.

Zhang

Liu

, et al. DIMNet: dense implicit function network for 3D human body reconstruction. Comput Graph 2021; 98: 1–10.

14.

Zhong

, et al. Automatic body measurement based on slicing loops. Int J Clothing Sci Technol 2018; 30: 380–397.

15.

Dorin

Hurwitz

Automatic body part measurement of dressed humans using single RGB-D camera. In: proceedings of the 2016 CHI conference extended abstracts on human factors in computing systems, San Jose, USA, 7-12 May 2016, pp. 3042–3048, New York: ACM.

16.

Liu X , Zhao H , Zhan G , et al. Rapid and automatic 3D body measurement system based on a GPU–Steger line detector. Appl Opt 2016; 55: 5539–5547.

17.

Markiewicz Ł, Witkowski M, Sitnik R, et al. 3D anthropometric algorithms for the estimation of measurements required for specialized garment design. Exp Syst Appl 2017; 85: 366–385.

18.

Anguelov

Srinivasan

Koller

, et al. SCAPE: shape completion and animation of people. ACM Trans Graph 2005; 24: 408–416.

19.

Loper

Mahmood

Romero

, et al. SMPL: a skinned multi-person linear model. ACM Trans Graph 2015; 34: 1–16.

20.

Bogo

Kanazawa

Lassner

, et al. Keep it SMPL: automatic estimation of 3D human pose and shape from a single image. In: Leibe et al. (eds.) European conference on computer vision, Amsterdam, Netherlands, 8-16 October 2016, paper no. LNCS 9909, pp. 561–578. Cham: Springer.

21.

Omran

Lassner

Pons-Moll

, et al. Neural body fitting: unifying deep learning and model-based human pose and shape estimation. In: JE Guerrero (ed.) international conference on 3D vision, Verona, Italy, 5-8 September 2018, pp. 484–494. Piscataway: IEEE.

22.

Kanazawa

Black

Jacobs

, et al. End-to-end recovery of human shape and pose. In: ed L O’Conner (ed.) 2018 IEEE/CVF conference on computer vision and pattern recognition, Salt Lake City, USA, 18-23 June 2018, pp. 7122–7131. Piscataway: IEEE.

23.

Choutas

Pavlakos

Bolkart

, et al. Monocular expressive body regression through body-driven attention. In: A Vedaldi et al. (eds.) European conference on computer vision, Glasgow, US, 23-28 August 2020, pp. 20–40. Cham: Springer.

24.

Pavlakos

Choutas

Ghorbani

, et al. Expressive body capture: 3D hands, face, and body from a single image. In: L O’Conner (ed.) proceedings/CVPR of the IEEE Computer Society conference on computer vision and pattern recognition, Long Beach, USA, 16-20 June 2019, pp. 10975–10985. Piscataway: IEEE.

25.

Zheng

Guo

, et al. DoubleFusion: real-time capture of human performances with inner body shapes from a single depth sensor. In: KM Lee (ed.) IEEE Transactions on Pattern Analysis and Machine Intelligence, Salt Lake City, USA, 18-22 June 2018, paper no. 10, pp. 2523–2539. Piscataway: IEEE.

26.

Yang

Franco

Hétroy-Wheeler

, et al. Estimation of human body shape in motion with wide clothing. In: B Leibe et al. (eds.) European conference on computer vision, Amsterdam, Netherlands, 8-16 October 2016, paper no. LNCS 9908, pp. 439–454. Cham: Springer.

27.

Zhang

Pujades

Black

, et al. Detailed, accurate, human shape estimation from clothed 3D scan sequences. In: L O’Conner (ed.) 2017 IEEE conference on computer vision and pattern recognition (CVPR), Honolulu, USA, 21-26 July 2017, pp. 5484–5493. Piscataway: IEEE.

28.

Neophytou

Hilton

Shape and pose space deformation for subject specific animation. In: JE Guerrero (ed.) 2013 international conference on 3D vision-3DV 2013, Seattle, USA, 29 June-1 July 2013, pp. 334–341. Piscataway: IEEE.

29.

Mcmakin

Sheen

Hall

TE.

Millimeter-wave imaging for concealed weapon detection. Proc SPIE 2003; 5048: 52–62.

30.

Martin-Vega

Aguayo-Torres

Gomez

, et al. Key technologies, modeling approaches, and challenges for millimeter-wave vehicular communications. IEEE Commun Mag 2018; 56: 28–35.

31.

Wikner

Zouaoui

Luukanen

, et al. Multi-sensor millimeter-wave system for hidden objects detection by non-collaborative screening. In: DA Wikner and AR Luukanen (eds.) passive millimeter-wave imaging technology XIV, Orlando, USA, 28 April 2011, pp.1–9. Washington: SPIE.

32.

Zhu

Ren

Image mosaic method based on SIFT features of line segment.

Computat Math Meth Med 2014; 2014: 1–12.

33.

Wei

Ramakrishna

Kanade

, et al. Convolutional pose machines. In: L O’Conner (ed.) Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, USA, 26 June – 1 July 2016, pp. 4724–4732. Piscataway: IEEE.

34.

Gkioxari

Dollár

, et al. Mask R-CNN. In: L O’Conner (ed.) Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22-29 October 2017, pp. 2980–2988. Piscataway: IEEE.

35.

Robinette

Blackwell

Daanen

, et al. Civilian American and European Surface Anthropometry Resource (CAESAR), Final Report. Volume 1. Summary. 2002.

36.

Allen

Curless

Popovic

The space of human body shapes: reconstruction and parameterization from range scans. ACM Trans Graph 2003; 22: 587–594.

37.

Zhuo

Han

Bp neural network implementation on real-time reconfigurable FPGA system for a soft-sensing process. In: MS Zhao and ZZ Shi (eds.) international conference on neural networks & brain, Beijing, China, 13-15 October 2005, pp. 959–963. Piscataway: IEEE.

38.

Alexa

Differential coordinates for local mesh morphing and deformation. Vis Comput 2003; 19: 105–114.

39.

ISO 8559-1:2017. Size designation of clothes — part 1: anthropometric definitions for body measurement.