Prediction of seismic response of building structures using a CNN-LSTM-ATT network with transfer learning

Abstract

Accurate prediction of the seismic response of buildings is crucial for their structural assessment and performance evaluation. To this end, leveraging recent advancements in deep learning, this study introduces a convolutional long short-term memory neural network with attention mechanism (CNN-LSTM-ATT) for predicting the seismic response of moment frame and shear wall-frame structures. Through ablation experiments, the effectiveness of the convolutional and attention blocks was validated. Furthermore, employing transfer learning, the CNN-LSTM-ATT model was fine-tuned to predict seismic response across different target buildings. Two distinct transfer learning scenarios were investigated: 1) transfer from finite element models with various parameters of the same structure; and 2) transfer from finite element models to same actual structures. These scenarios demonstrate that model-based transfer learning significantly enhances the prediction accuracy of CNN-LSTM-ATT. Compared to the finite element models, the model based on transfer learning (i.e., with fine-tuning) in various scenarios, accurately predicted the nonlinear behaviors of structures. Thus, the proposed method is applicable for easy modeling and rapid prediction of dynamic response in various building structures under earthquakes.

Keywords

deep learning convolutional neural network long short-term memory neural network attention mechanism seismic response prediction transfer learning

Introduction

Earthquakes, as destructive and hazardous natural disasters, present significant challenges to the safety and reliability of buildings. To accurately assess the response of structures under earthquakes, various effective prediction models and analytical methods have been developed. Generally, the dynamic response analysis of structures under seismic actions involves using physics-based models, namely nonlinear time-history analysis (NLTHA) based on mechanical principles. Considerable research has been conducted over the past few decades in this area, resulting in numerous successful computational implementations in the field of civil engineering (Ayoub et al., 2022; Çavdar and Bayraktar, 2013), among which the finite element analysis (FEA) stands as one of the most popular numerical methods in structural dynamic analysis. Finite element models, commonly used in structural analysis, provide detailed structural response information. However, finite element models used for nonlinear dynamic analysis often require finely discretized mesh elements and smaller computational time steps, which result in high computational costs for nonlinear time-history analysis. Moreover, relying solely on finite element models for earthquake response prediction poses limitations due to model imperfections and parameter uncertainties.

To address challenges, such as high computational costs and numerical convergence difficulties, proxy models have been widely adopted in engineering design. These proxy models encompass various techniques, including generalized linear regression models (Guo et al., 2020), radial basis functions (Das and Choudhury, 2019), polynomial response surface models (Moodi et al., 2018), Kriging models (Ghosh et al., 2019; Hoang et al., 2021; Zhang and Wu, 2017), Chebyshev Polynomials Method (Zeng et al., 2024), and artificial neural networks (ANN) (Jahangir and Rezazadeh Eidgahee, 2021; Kalakonas and Silva, 2021; Mangalathu et al., 2018; Tran and Kim, 2020; Wang et al., 2021). However, these proxy models also exhibit the following drawbacks: 1) unbalanced sample distributions and imposed assumptions can diminish the predictive capabilities of these models; and 2) with an increase in problem dimensions, the construction of proxy models requires significantly more sample points and time, potentially compromising their accuracy and affecting their reliability.

In recent years, the rapid development of deep learning technologies has offered new insights and solutions for predicting the earthquake response of structures. The emergence of deep learning models, such as convolutional neural networks (CNN), long short-term memory networks (LSTM), attention mechanisms (ATT), and transfer learning, has significantly improved the modeling and prediction for the earthquake response of structures (Ahmed et al., 2022; Huang and Chen, 2021; Oh et al., 2020a; Perez-Ramirez et al., 2019; Wang et al., 2022, 2023; Zhang et al., 2020). Oh et al. (2020c) proposed a method for predicting the seismic response of building structures using CNN, based on acceleration and displacement time histories, and validated its effectiveness through numerical and experimental analyses. Xu et al. (2022) utilized LSTM to predict nonlinear structural earthquake response under arbitrary lengths and sampling rates. Zhang et al. (2019) introduced two LSTM network schemes that replaced traditional physics-based numerical analysis methods in a data-driven manner, to accurately predict the nonlinear earthquake response of buildings. Liao et al. (2023) proposed a recursive neural network based on deep learning and attention mechanisms (AttLSTM) to accurately predict bridge response under dynamic loads, and demonstrated superior accuracy and reliability compared to traditional LSTM models. Ning et al. (2023) utilized three different deep learning models (i.e., LSTM, WaveNet, and 2D CNN) to predict the time history response of various structures under earthquakes. Sadeghi Eshkevari et al. (2021) designed a physics-based recurrent neural network for estimating the dynamic characteristics of linear and nonlinear multi-degree-of-freedom systems subjected to ground motions. Recent studies, such as Zhang et al. (2024) and Zhao et al. (2024), have developed CNN-LSTM and LSTM-based models to improve seismic response prediction for railroads and train-bridge systems, respectively.

Although deep learning models, including CNN and LSTM, have made significant strides in predicting earthquake response of structures, several crucial challenges are still encountered for practical applications. Firstly, deep learning models typically require large-scale labeled data for training. However, obtaining such data in the field of earthquake response prediction is highly challenging and costly. Experimental and observational data in earthquake engineering usually provide limited information on seismic events, which requires an improved training technique for deep learning models for accurate predictions with limited data. Secondly, the diversity in structural characteristics and seismic environments poses challenges to the generalization performance of the trained models. Each building has unique geometric shapes, material properties, and structural parameters, while seismic events vary based on location and time. As deep learning models need robust generalization capabilities to adapt to the diversity of different structures and seismic events, further research is required to enhance the generalization performance. Despite the significant potential of deep learning models in predicting the earthquake response of structures, overcoming challenges, such as data acquisition and generalization performance, is crucial to better satisfy the practical requirements for earthquake engineering.

This study utilized a finite element model as a source model for pre-training to establish a deep learning model (i.e., CNN-LSTM-ATT). Subsequently, fine-tuning was performed using actual earthquake response data from structures to enhance the prediction performance of the deep learning model. In this model, CNN effectively extracts the local feature patterns and frequency components from structural vibration signals (Oh et al., 2020b), LSTM captures the temporal relationships and long-term dependencies in sequential data (Li et al., 2022), and ATT filters the crucial information for the current task from vast amounts of data. This study designed three different application scenarios to verify the robustness of this transfer learning method: 1) transfer between structures with different material strengths; 2) transfer from finite element models to the same actual structures. In each scenario, a CNN-LSTM-ATT model was established and then fine-tuned with a small amount of data.

This transfer learning approach is expected to address issues such as data scarcity and differences among various structures, enabling the model to better adapt to diverse structural earthquake response prediction tasks. By integrating the physical knowledge from finite element models and the advantages of deep learning models, this study aims to achieve more accurate and reliable earthquake response predictions. The finite element model provides an in-depth understanding of structural behavior, encompassing detailed information on the material properties, geometric shapes, and boundary conditions. This information can be utilized to initialize the deep learning model, aiding its comprehension of the fundamental characteristics of structures. This fusion of physical knowledge and deep learning holds the potential for significant breakthroughs in predicting the earthquake response of structures.

Structural information and modeling

To simulate the earthquake response of structures, two types of finite element models (i.e., shear wall-frame system and moment frame system) were established according to the Chinese seismic design code (GB50011-2010, 2010). The shear wall-frame structure model is labeled as SF-FEM, while the frame structure is used for finite element model (F-FEM) and physical model (F-AS). Based on the different concrete strength grades, SF-FEM is divided into SF-FEM(C30) and SF-FEM(C50). Table 1 summarizes the basic information of these three models.

Table 1.

Information of models.

Model	Model type	Structure type	Number of stories
SF-FEM(C30)	Finite element model	Shear wall-frame	12
SF-FEM(C50)	Finite element model	Shear wall-frame	12
F-FEM	Finite element model	Moment frame	2
F-AS	Actual structure	Moment frame	2

Shear wall-frame structure model

Figure 1 shows a twelve-story shear wall-frame structure. The story height is 4.5 m at the first floor and 3.5 m at the floors above (i.e., total height of 43.0 m). The dead load is 5.0 kN/m², and the live load is 2.0 kN/m². OpenSees was used to establish a finite element model for the shear wall-frame structure with a damping ratio of 0.05, using a Rayleigh damping model. The Concrete01 constitutive model (Kent and Park, 1971) was used for the unconfined concrete, and the Concrete02 model (Yassin, 1994) was used for confined core concrete. The Steel02 constitutive model was used for reinforcing bars. The compressive strength of concrete is 30 MPa, and the yield strength of both longitudinal reinforcement and stirrups is 388 MPa. The moment frame and beam-column connections were modeled using force-based fiber elements (forceBeamColumn Element), and the shear wall was modeled using multiple vertical line element model (MVLEM). P-Delta effect was also considered. For nonlinear analysis, the Newton-Raphson algorithm was employed. Component section information is provided in Table 2. The material constitutive relationships for concrete and reinforcement are shown in Figure 2.

Figure 1.

Analysis model for shear wall-frame structure (unit: mm) (a) structural floor plan (b) equivalent overall model of shear wall-frame structure.

Table 2.

Component dimensions and reinforcement details.

Component type	Cross-sectional dimensions	Longitudinal reinforcement	Longitudinal distribution reinforcement	Horizontal distribution reinforcement
Column	600 mm × 600 mm	4D25 + 8D18	—	—
Beam & Coupling beam	300 mm × 600 mm	6D22 + 4D25 + 4D12	—	—
Shear wall	250 mm × 6000 mm	8D16	D10@200	D12@200

Figure 2.

Material constitutive relationships (a) concrete01 constitutive model (b) concrete02 constitutive model (c) steel02 constitutive model.

To improve the efficiency of analysis, the model is simplified by selecting one frame in Y-direction and one segment of coupled shear wall from the three-dimensional structure. An equivalent planar model in Y-direction is established, where the principle of equivalence maintains the relative stiffness of the frame and shear wall (i.e., the structural stiffness characteristic value $λ$ remains unchanged). The thickness and reinforcement of the wall are reduced using equation (1), and rigid hinge links were used between the frame and shear wall to establish the planar model, as shown in Figure 1. The stiffness characteristic value ( $λ$ ) is defined as follows (GB50010-2010, 2015):

λ = H \sqrt{\frac{C_{F}}{E_{C} I_{e q}}}

(1)

where

H

is the height of the structure;

E_{c} I_{e q}

is the equivalent stiffness of the shear wall; and

C_{F}

is the lateral stiffness of the moment frame.

Moment frame structure model

This model represents a two-story frame structure, selected from the literature (Ma et al., 2013). The frame is regular both horizontally and vertically, with plan dimensions of 1.6 m × 1.6 m and a floor height of 1.2 m, as illustrated in Figure 3. The scale ratio is 1:4. The dead load is 5.0 kN/m², and the live load is 2.0 kN/m². The column section dimensions are 100 mm × 100 mm, beam section dimensions are 100 mm × 70 mm, and the slab thickness is 60 mm. The mass of each floor in the model is 630 kg (including the weight of the reinforcement in concrete). The compressive strength of concrete at 28 days is 12 MPa. The yield strength of the longitudinal reinforcement and stirrups is 335 MPa and 190 MPa, respectively. For the finite element model of this structure, nonlinear beam-column elements (nonlinearBeamColumn) from OpenSees were employed to simulate the frame beams and columns, using fiber sections. The unconfined concrete adopted the Concrete01 (Kent and Park, 1971) material model from OpenSees, while the confined concrete used the Concrete02 material (Yassin, 1994). The reinforcing bars used the Steel02 constitutive model.

Figure 3.

Analysis model for moment frame structure (unit: mm).

Ground motion data

In this study, 20 seismic event records were selected from the Pacific Earthquake Engineering Research Center (PEER) earthquake database. The peak ground acceleration (PGA) of these selected seismic records was scaled to 310 cm/s², corresponding to a rare earthquake under 7-degree seismic fortification based on the Chinese seismic design code (GB50011-2010, 2010). Details of the ground motion data are provided in Table 3.

Table 3.

Ground motion data.

ID	Event	Station	Magnitude	Epicentral distance (km)	Peak acceleration (g)
1	EI Alamo-02	EI Centro Array#9	6.8	2.9	0.214
2	Borrego Mtn-04	LB-Terminal ISland	6.6	199.84	0.365
3	Imperial Valley-01	EI Centro Array#9	7.0	34.9	0.014
4	Northwest Calif-01	Ferndale City Hall	6.5	53.6	0.144
5	Imperial Valley-02	El Centro Array#9	6.9	6.1	0.262
6	Northwest Calif-02	Ferndale City Hall	6.6	91.2	0.061
7	Northern Calif-01	Ferndale City Hall	6.8	44.7	0.113
8	Kobe-01	0 KJMA	6.9	20.9	0.343
9	Kern County	LA-Hollywood Stor FF	7.3	117.8	0.041
10	Kern County	Pasadena-CIT Athenaeum	7.3	125.6	0.047
11	Imperial Valley-05	El Centro Array#9	7.0	6.09	0.254
12	Imperial Valley-05	Hollister City Hall	6.9	25.8	0.046
13	El Alamo	El Centro Array#9	6.8	121.7	0.336
14	Coyote Lake	Gilroy Array#2	6.7	8.5	0.255
15	Managua Nicaragua-05	Managua ESSO	6.8	4.33	0.532
16	Westmorland	Parachute Test Site	6.9	16.5	0.258
17	Northrn Calif-04	Cape Mendocino	6.2	28.73	0.606
18	San Fernando	2516 Via Tejon PV	6.6	55.2	0.023
19	Managua Nicaragua-01	Managua ESSO	6.7	4.1	0.335
20	San Fernando	UCSB-Fluid Mech Lab	6.6	124.4	0.017

CNN-LSTM-ATT deep learning model

This study adopted a deep learning model (CNN-LSTM-ATT) that combined CNN, LSTM, and ATT to predict the nonlinear earthquake response of structures. In this model, CNN is utilized to capture the spatial features of time-series data, followed by passing these features to LSTM for time-series analysis and modeling. Finally, ATT is employed to enhance the representational and generalization capabilities of the trained model.

Convolutional neural networks (CNN)

CNN is a type of deep learning model particularly well-suited for image processing and pattern recognition tasks. CNN, based on convolutional operations, extracts features from input data by sliding a window (convolutional kernel). In time series data, this is similar to identifying local patterns and trends (i.e., small-scale patterns and overall changes in the data). This aids the model in better understanding the local structure of the input data. By employing CNN, the model can more effectively capture the local features and patterns in time series data, rather than treating the entire sequence as a whole. This enhances the model’s sensitivity to the internal complexity of the data and allows it to better adapt to local variations, thereby improving its ability to model time series data. The principles of CNN can be summarized as two main steps of convolution operation and pooling operation (Figure 4).

Convolution operation is the core step used to extract local features from input data. The core idea of convolutional operations is to detect and extract specific patterns throughout the entire time series by applying filters (or convolutional kernels) to different parts of the input data. Specifically, for a given input data, the calculation formula for the convolution operation is as follows:

C = ReLu (X * W + b)

(2)

ReLu = \max (0, x)

(3)

where * represents the convolution operation;

X

is the input data;

W

is the convolutional kernel;

b

is the bias term;

x

is the real number; and

ReLu

represents the Rectified Linear Unit activation function. Typically, after the convolution operation, a nonlinear activation function (e.g., ReLu) is applied to introduce nonlinear mapping.

Pooling operation is used to reduce the size of feature maps, to extract the essential features, and to reduce computational complexity. The most widely used pooling operations are the max pooling and average pooling. The max pooling (equation (4)) selects the maximum value within a local region as the pooled result, whereas the average pooling (equation (5)) calculates the mean value within a local region.

Y (i, j) = \max (X (i * K : (i + 1) * K, j * K : (j + 1) * K))

(4)

Y (i, j) = mean (X (i * K : (i + 1) * K, j * K : (j + 1) * K))

(5)

where

Y (i, j)

is the elements in the pooled result;

X (i * K : (i + 1) * K, j * K : (j + 1) * K)

is the local region in the input feature map; and

K

is the pooling window size.

Figure 4.

Architecture of 1d CNN block.

Long short-term memory neural networks (LSTM)

LSTM is a type of recurrent neural network structure primarily used for handling sequential data. In the CNN-LSTM-ATT model, LSTM is employed to perform sequential modeling on the local feature sequences extracted by CNN, enabling the learning of dynamic evolution and temporal characteristics within sequences. Through memory units and gate mechanisms in LSTM, the model can effectively handle long sequences and prevent issues, such as vanishing or exploding gradients during training. By modeling the sequence data, LSTM integrates contextual information of input data into the model, which enables a better understanding of the inherent patterns within sequential data and consequently provides more accurate prediction results. Therefore, within the CNN-LSTM-ATT model, LSTM plays a crucial role by enabling the model to perform temporal modeling of sequence data and modeling long-term dependencies, further enhancing the model’s predictive performance on time-series data. As seen in Figure 5(a), the LSTM architecture consists of a series of self-connected LSTM units. In Figure 5(b), each comprises three interconnected components including a forget gate, an input gate, and an output gate. The specific computational process of the unit is defined as follows:

f_{t} = sigmoid (W_{f} \cdot [h_{t - 1}, x_{t}] + b_{f})

(6)

i_{t} = sigmoid (W_{i} \cdot [h_{t - 1}, x_{t}] + b_{i})

(7)

o_{t} = sigmoid (W_{o} \cdot [h_{t - 1}, x_{t}] + b_{o})

(8)

{\tilde{C}}_{t} = \tan h (W_{C} \cdot [h_{t - 1}, x_{t}] + b_{C})

(9)

C_{t} = f_{t} \cdot C_{t - 1} + i_{t} \cdot {\tilde{C}}_{t}

(10)

h_{t} = o_{t} \cdot \tanh (C_{t})

(11)

where

W

is the weight matrices;

b

is the biases;

x_{t}

is the input vector at the current time step;

h_{t - 1}

is the hidden output from the previous time step;

f_{t}

is the output of the forget gate;

i_{t}

is the output of the input gate;

o_{t}

is the output gate output;

{\tilde{C}}_{t}

is the candidate memory cell;

C_{t - 1}

and

C_{t}

are the memory cells at previous and current time steps, respectively; and

h_{t}

is the hidden output of the current LSTM unit.

Figure 5.

Architecture of LSTM block: (a) architecture of LSTM neural network with multiple hidden layers; (b) architecture of a LSTM cell.

Attention mechanisms (ATT)

ATT layer is an implementation of the attention mechanism within deep learning models, which is used to dynamically adjust the focus of the model for different positions within input sequences. The ATT layer calculates the attention weights to encode the importance of different time steps. These weights are then used to perform a weighted summation across the input sequence, thereby generating representations of key information within the sequence. The role of the ATT layer is to provide a degree of focus on the input sequence, enabling the model to pay attention to parts meaningful for the current task. Specifically, given a query vector ( $Q$ ) and a set of key-value pairs ( $K - V$ ), ATT computes similarity scores between the query and keys, normalizes the scores using softmax, and obtains attention weights. In this study, the attention scores are computed using the scaled dot-product model.

Attention (Q, K, V) = s oftmax (\frac{Q K^{T}}{\sqrt{d_{k}}}) V

(12)

softmax (X) = \frac{e^{X_{i}}}{\sum_{j = 1}^{n} e^{X_{i}}}

(13)

where the

softmax

function normalizes the similarity scores into a probability distribution. The attention weights are used to perform a weighted summation with values (

V

) to generate the final output. T represents the transpose of the matrix; and

d_{k}

is the dimensionality of the input vectors (i.e., the scaling factor), which effectively prevents excessively large dot products and reduces the chances of gradient vanishing, aiding in stability and convergence during training.

In this study, a self-ATT is employed (refer to Figure 6). For an input sequence ( $X$ ) with feature values, queries ( $Q$ ), keys ( $K$ ), and values ( $V$ ) are computed by separately multiplying it with three trainable weight matrices ( $W_{Q}$ , $W_{K}$ , and $W_{V}$ ). This mechanism allows CNN-LSTM-ATT model to adaptively adjust attention according to different parts of the input sequence, better capturing the relationship and importance between inputs. The specific calculation process is defined as follows:

Attention (X) = s oftmax (\frac{X W_{Q} X^{T} W_{K}}{\sqrt{d_{k}}}) X W_{V}

(14)

Figure 6.

Architecture of attention block.

CNN-LSTM-ATT for seismic response of structures

CNN-LSTM-ATT model was utilized for predicting the seismic response of building structures, taking seismic excitations (i.e., accelerations) and structural response (i.e., displacements) within any fixed time as input and producing the structural response for the next time step. For instance, it might use the ground acceleration and structural displacement for the previous n time steps as input and predict the displacement for the n + 1 time step. In this context, n represents the length of the input vector for the CNN-LSTM-ATT model, and can be any integer less than the total number of time steps in the seismic event. Therefore, the CNN-LSTM-ATT model is not constrained by the overall duration of the seismic motion or the length of time steps for individual samples. Generally, the CNN-LSTM-ATT model performs single-step predictions of the structure response, forecasting only one time step at a time. If multiple time steps need to be predicted, the predicted values are used as known values for subsequent predictions in chronological order.

The CNN-LSTM-ATT is a deep learning architecture derived from the LSTM model. Figure 7 shows the architecture of the CNN-LSTM-ATT that comprises the input layer, convolutional layers, LSTM blocks, attention blocks, and output layer. The convolutional layers consist of two one-dimensional convolutional layers (Conv1d) and two max-pooling layers (MaxPool1d). The LSTM block consists of three stacked LSTM layers. The attention block comprises three fully connected (FC) layers used to generate vectors ( $Q$ , $K$ , and $V$ ). The output layer utilizes a single FC layer.

Figure 7.

Architecture of CNN-LSTM-ATT.

Rectified Linear Unit (ReLU) activation functions were employed in both the CNN and FC layers, to reduce the computational load and mitigate gradient vanishing to some extent. Dropout layers were applied after the attention block and FC layers during the training phase to prevent overfitting. Given a dropout probability of p (equal to 0.1 in this study), each neuron was dropped out with a probability of p during each iteration. During the training of the CNN-LSTM-ATT model, the Adam optimizer was selected, and the mean squared error (MSE) widely used in regression tasks was chosen as the loss function. The Adam optimizer is a gradient descent-based optimization algorithm that integrates momentum and adaptive learning rate features. The algorithm maintains the first and second moment estimates of gradients for each parameter, incorporating bias correction for efficient parameter updates. Due to the large volume of data, the training data are fed to the model in batches using batch size. The batch size represents the number of samples selected for a single training iteration and affects the optimization level and speed of the CNN-LSTM-ATT model. In this study, a batch size of 64 was used for all subsequent models, trained over 100 epochs. The entire training process was executed in a Python environment, utilizing PyTorch as the deep learning library.

Performance validation of CNN-LSTM-ATT model

To verify the feasibility of the proposed CNN-LSTM-ATT model in predicting the dynamic response of structures, the sixth floor of the SF-FEM structure model with small residual displacements was taken as an example. Twenty sets of earthquake ground motion data, along with the corresponding displacement data from finite element analysis, are split into training, validation, and testing sets in an 8:1:1 ratio for the deep learning model. Therefore, among the 20 datasets, there are 2 test sets, numbered TS1 and TS2, respectively. Prior to inputting data, the training data need to be normalized, mapping all data into the range $[- 1, 1]$ . This normalization eliminates dimensional interference and ensures the model converges effectively.

Considering the requirement of consistent input sequence length in LSTM, a common approach is to use a fixed-length sliding window to obtain samples and labels. In all the models presented in this paper, the sliding window length is set to 100, corresponding to a time duration of 1 second. Models using sliding windows of varying lengths exhibit significant performance differences. Typically, longer windows result in higher model performance compared to smaller window lengths. However, longer window lengths require more input information, which can diminish the model’s generalizability. To address this, this study proposes a data processing strategy: padding the first data point of seismic waves and displacement data with a fixed length of zeros, matching the sliding window’s length. This strategy is crucial as it allows the model to predict the entire displacement sequence without requiring any displacement information when the seismic wave is known.

This study selected four different model structures. Firstly, the basic LSTM model was used as the baseline, followed by the introduction of the attention mechanism (ATT) to enhance the representational capacity of the LSTM model, resulting in the LSTM-ATT model. Subsequently, an attempt was made to integrate the CNN module into the LSTM model, forming the CNN-LSTM model. Finally, both the CNN and ATT modules were integrated into the LSTM model simultaneously, resulting in the CNN-LSTM-ATT model. To validate the performance improvement of these two modules in structural seismic response prediction, an ablation study was conducted. In the field of machine learning, especially in complex deep neural networks, ablation study is used to describe the process of removing certain parts of the network to better understand the behavior of the network. During the training phase, all network architectures utilized the same training dataset, employed the Adam optimizer, and adopted the mean squared error (MSE) loss function, with the maximum of 100 epochs. The initial learning rate was set to $10^{- 3}$ , and at the 10th, 40th, and 60th epochs, the learning rate was adjusted to $10^{- 4}$ , $10^{- 5}$ , and $10^{- 6}$ , respectively. The model accuracy was evaluated using the coefficient of determination ( $R^{2}$ ), root mean square error (RMSE), and mean absolute error (MAE), as follows:

R^{2} = 1 - \frac{\sum_{i = 1}^{n} {(y_{i} - {\hat{y}}_{i})}^{2}}{\sum_{i = 1}^{n} {(y_{i} - \bar{y})}^{2}}

(15)

R MSE = \sqrt{\frac{\sum_{i = 1}^{n} {({\hat{y}}_{i} - y_{i})}^{2}}{n}}

(16)

MAE = \frac{\sum_{i = 1}^{n} | {\hat{y}}_{i} - y_{i} |}{n}

(17)

where a higher

R^{2}

indicates a better fit of the model to the data;

y_{i}

is the actual value;

{\hat{y}}_{i}

is the prediction;

\bar{y}

is the average of actual values; and n is the number of actual values. RMSE quantifies the deviation between the predictions and actual values; and MAE represents the average of the absolute errors between the predictions and actual values. Similar to RMSE, a lower MAE signifies higher predictive accuracy. Typically, RMSE is greater than or equal to MAE because RMSE is more sensitive to outliers.

Figure 8 shows the training processes of four models: LSTM, LSTM-ATT, CNN-LSTM, and CNN-LSTM-ATT, with minimum losses of $4.13 \times 10^{- 5}$ , $4.22 \times 10^{- 5}$ , $4.15 \times 10^{- 5}$ , and $4.60 \times 10^{- 5}$ , respectively. All four models converge to the same loss order of magnitude after 100 epochs. The CNN-LSTM-ATT model exhibits the highest loss, while the LSTM model has the lowest loss. This is because the CNN-LSTM-ATT model combines multiple complex neural network structures, thus having more parameters and higher complexity. During training, such multiple structures would lead to more challenging convergence for the model, resulting in higher training loss.

Figure 8.

Comparison of the loss functions in four networks for predicting the seismic response of the shear-wall frame structure.

Figure 9 demonstrates the prediction performance of the four models on the test set with a time step of 0.01 seconds. Observing the waveforms, it is evident that both the CNN-LSTM and LSTM-ATT curves exhibit superior fitting results, compared to LSTM. The predictions of the CNN-LSTM-ATT model closely matched the actual displacement, achieving the highest prediction accuracy. As the system exhibited a strongly nonlinear phase, LSTM generated significant displacement deviations due to its inability to accurately capture the dynamic response characteristics of the structure.

Figure 9.

Prediction results of different models for SF-FEM(C30): (a) time history curves for the TS1; (b) scatter plot for the TS1; (c) time history curves for the TS2; (d) scatter plot for the TS2.

Table 4 presents the prediction metrics of various models for the shear-wall frame structure. As seen in Figure 10, LSTM-ATT and CNN-LSTM achieved higher accuracy in terms of

R^{2}

, MAE, and RMSE, compared to LSTM. However, the CNN-LSTM-ATT model, benefitting from the incorporation of both CNN and ATT, was able to capture the dynamic features of the structure more effectively during the prediction process, showcasing its stronger feature extraction capabilities. The CNN-LSTM-ATT model consistently outperformed both LSTM-ATT and CNN-LSTM across all metrics. In TS1, CNN-LSTM-ATT achieved the highest

R^{2}

of 0.946, a significant improvement over LSTM (0.794) and CNN-LSTM (0.926). It also achieved the lowest MAE (0.695) and RMSE (0.931), showing an improvement of approximately 41% in MAE and 50% in RMSE, compared to LSTM. In TS2, CNN-LSTM-ATT reached an

R^{2}

of 0.971, a 26% improvement over LSTM (0.702), and reduced MAE to 0.379, significantly outperforming LSTM (1.238) and CNN-LSTM (0.673). The RMSE of CNN-LSTM-ATT (0.674) was also much lower than that of the other models.

Table 4.

Prediction metrics of different models for the shear-wall frame structure.

Test set	Model	$R^{2}$	MAE	RMSE
TS1	LSTM	0.794	1.185	1.862
	LSTM-ATT	0.836	1.057	1.629
	CNN-LSTM	0.926	0.753	1.095
	CNN-LSTM-ATT	0.946	0.695	0.931
TS2	LSTM	0.702	1.238	2.158
	LSTM-ATT	0.764	1.117	1.920
	CNN-LSTM	0.921	0.673	1.108
	CNN-LSTM-ATT	0.971	0.379	0.674

The Bold values indicate the highest prediction accuracy among the compared models.

Figure 10.

Prediction metrics of different models for the shear-wall frame structure: (a) TS1; (b) TS2.

This ablation study highlights the contribution of both the CNN and ATT mechanisms in improving model performance. By integrating CNN for local feature extraction and ATT for focusing on important time steps, CNN-LSTM-ATT is able to better handle complex sequential data, improving both prediction accuracy and reliability. The incorporation of the ATT mechanism enables the model to focus on key seismic features, leading to stronger prediction capabilities in earthquake response tasks.

Prediction methods based on transfer learning

Transfer learning

Transfer learning is a machine learning method that leverages a model trained in one domain (Task 1) and applies it to another related domain (Task 2) to enhance the model’s performance in the target domain (Task 2). This approach finds applications across various fields, such as computer vision, natural language processing, recommendation systems, etc. While the transfer learning offers numerous advantages, considerations include the similarity between the source and target domains, as well as how to select or adapt models to address challenges in new tasks.

In practical engineering scenarios, it is often prohibitively expensive and challenging to obtain extensive and authentic seismic response data of structures. This is a realistic issue in many engineering fields, such as structural design and assessment of buildings, bridge engineering, safety assessment of nuclear power plants, and design of oil and gas pipelines and tanks, as well as underground structures and tunnel engineering. Due to the difficulty in obtaining a large amount of seismic response data for evaluating the stability and safety of structures, the training of seismic response models relies on limited samples. Because traditional machine learning methods rely on statistical characteristics among samples, the uncertainty of statistical estimation would decrease the trained model performance in small sample contexts. In particular, the unstable parameter estimation would further result in overfitting or underfitting issues. The transfer learning, through mechanisms including knowledge transfer, feature sharing, and domain adaptation, helps overcome the challenges of learning with small samples, thereby enhancing the generalization and performance of the trained model. Leveraging existing data and knowledge, the transfer learning renders machine learning tasks in small sample scenarios more feasible and effective. Thus, in situations of limited data availability or high costs, the transfer learning is effective to mitigate the impacts of data constraints.

Strategy of transfer learning

The strategy employed in this study involves a finite element model-based transfer learning approach, utilizing a well-trained model from the source domain and adapting it through fine-tuning for application in the target domain. The source domain is determined where the model is initially trained and performs well. In the transfer learning, a well-trained model from this domain is adjusted or fine-tuned for the target domain. The target domain is the new domain where the model is transferred. The goal is to adapt the model to the target domain, improving its performance. Fine-tuning is a specific method of transfer learning. This methodology provides a pragmatic solution to the scarcity of structural response data in practical engineering, enabling the acquisition of a higher-performing surrogate model at a reduced cost. Typically, pre-trained models would exhibit robust feature extraction capabilities. Consequently, during the fine-tuning process, adjustments are primarily made to the weights and parameters at the final layers. Additionally, compared to training models from scratch, the fine-tuning necessitates fewer epochs. However, the learning rate during this phase should be carefully controlled to prevent excessive modifications to the learned features that could lead to diminished performance. This approach optimally uses the rich information from the source domain data and the acquired features, which allows more efficient and feasible application in the target domain.

The robustness of transfer learning in predicting structural response was verified through two combinations of source models and prediction models. These combinations include SF-FEM(C30) and SF-FEM(C50), F-FEM and F-AS, where C30 and C50 represent the compressive strengths of concrete of 30 MPa and 50 MPa, respectively. In these combinations, SF-FEM(C30) and F-FEM indicate the source domain models, while SF-FEM(C50) and F-AS indicate the target domain models. The model is pre-trained using data generated from the source domain models, and then fine-tuned with a small amount of data from the target domain models to achieve large-scale predictions in the target domain. Considering the scenario with the limited data available, this study only considers the first 4 seconds of data as known data.

Prediction of finite element model response

In the ablation study, the prediction performance of four models was compared. For the same structure, the CNN-LSTM-ATT model was able to fully reconstruct the structural response under seismic excitations that were not included in the training set. However, changes in structural parameters affect the corresponding seismic response. To adapt the model to new structures with varying strength parameters, a detailed fine-tuning strategy was employed. This involved collecting a small amount of response data from the new structure and fine-tuning the model by adjusting the weights and parameters of the final layers to better capture the response characteristics of the new structure. Note that careful control of the learning rate is necessary during this process to avoid excessive modifications to the learned features.

In this transfer learning process, the response data from F-FEM(C30) was used to pre-train the model, with SF-FEM(C50) as the target prediction structure. The difference between these two models lies in the concrete strength: SF-FEM(C30) has a concrete strength of 30 MPa, while SF-FEM(C50) has a concrete strength of 50 MPa. As a result, the stiffness of the two finite element models differs, which would affect the dynamic response of the structure. Figure 11 compares the prediction results of different models including 400 input time steps. As seen in Table 5, the $R^{2}$ of the CNN-LSTM-ATT model exceeded 0.9 in both test sets, demonstrating the highest prediction accuracy. Figure 12 compares the prediction performance of models in accordance with input time steps during fine-tuning. An input time step of 0 indicates no transfer learning, and 100 time steps represent 1 second. When the number of input time steps was small, the performance of models including CNN-LSTM-ATT was lower than that of the LSTM model. However, as the number of input time steps increased, the prediction performance of the CNN-LSTM-ATT model significantly improved compared to the other models.

Figure 11.

Prediction results of different models with 400 input time steps for SF-FEM(C50): (a) time history curves for TS1; (b) scatter plot for TS1; (c) time history curves for TS2; (d) scatter plot for TS2.

Table 5.

Prediction metrics of different models with 400 input time steps for SF-FEM(C50).

Test set	Model	$R^{2}$	MAE	RMSE
TS1	LSTM	0.869	1.002	1.366
	LSTM-ATT	0.856	1.053	1.432
	CNN-LSTM	0.868	1.026	1.371
	CNN-LSTM-ATT	0.922	0.783	1.052
TS2	LSTM	0.863	0.760	1.044
	LSTM-ATT	0.852	0.778	1.085
	CNN-LSTM	0.866	0.756	1.031
	CNN-LSTM-ATT	0.932	0.549	0.734

The Bold values indicate the highest prediction accuracy among the compared models.

Figure 12.

Prediction metrics of different models according to input time steps: (a) $R^{2}$ on TS1; (b) $R^{2}$ on TS2; (c) MAE on TS1; (d) MAE on TS2; (e) RMSE on TS1; (f) RMSE on TS2.

The advantage of transfer learning in this section lies in its ability to significantly reduce the reliance on target-domain data while maintaining high prediction accuracy. By leveraging the pre-trained model from F-FEM(C30) and fine-tuning it with a small amount of response data from the new structure SF-FEM(C50), the CNN-LSTM-ATT model was able to adapt to the new structure with varying concrete strength, demonstrating its ability to generalize across different structural configurations. This approach minimizes the need for large amounts of target-domain data, making it a cost-effective and efficient method for predicting the seismic response of structures with varying parameters.

Prediction of actual structure response

In this section, F-FEM is the source domain model, and F-AS is the target domain model. F-AS represents the real structure from the shaking table test, while F-FEM is the corresponding finite element model of F-AS. The difference between the two lies in the fact that F-FEM is an imprecise fit of F-AS, resulting in a significant discrepancy in the dynamic response of the structure, including phase and amplitude. Figure 13 compares time history curves and scatter plot of each prediction model including 400 input time steps in the F-AS structure. As summarized in Table 6, the CNN-LSTM-ATT model consistently outperformed the other models in terms of prediction accuracy, showing an $R^{2}$ of 0.949, a MAE of 0.331, and a RMSE of 0.404. In contrast, the LSTM model exhibited the lowest $R^{2}$ value of 0.732, highlighting its inferior predictive performance. The CNN-LSTM and LSTM-ATT models demonstrated intermediate performance, but neither matched the robustness of CNN-LSTM-ATT, which leveraged attention mechanism and integrated learning to capture the structural response characteristics more effectively. Note that the value of $R^{2}$ generally ranges between 0 and 1. However, in nonlinear relationships, if there is a certain phase shift between the predicted and actual values, $R^{2}$ can decrease rapidly and may even become negative. For this reason, the $R^{2}$ for FEM, primarily used to understand the nonlinear behavior of structures, became negative (Table 6).

Figure 13.

Prediction results of different models for F-AS: (a) time history curves; (b) scatter plot.

Table 6.

Prediction metrics of different models with 400 input time steps for F-AS.

Model	$R^{2}$	MAE	RMSE
FEM (without input)	−0.530	0.986	1.363
LSTM	0.732	0.732	0.928
LSTM-ATT	0.941	0.362	0.435
CNN-LSTM	0.892	0.461	0.589
CNN-LSTM-ATT	0.949	0.331	0.404

The Bold values indicate the highest prediction accuracy among the compared models.

Figure 14 compares the performance of these models in accordance with input time steps, specifically evaluating $R^{2}$ , MAE, and RMSE. As the number of input time steps increased, the CNN-LSTM-ATT model’s performance significantly improved, achieving an $R^{2}$ of nearly 1.0 at 400 time steps. Moreover, in terms of error metrics, both MAE and RMSE decreased as the input time steps increased, indicating better prediction accuracy. This result indicates that the low prediction accuracy can be improved through fine-tuning. In contrast, the other models including LSTM exhibited less improvement according to additional time steps, indicating that CNN-LSTM-ATT’s architecture is better suited for complex time-series predictions with larger datasets.

Figure 14.

Prediction metrics of different models according to input time steps: (a) $R^{2}$ ; (b) MAE; (c) RMSE.

In this section, the advantage of transfer learning lies in its ability to effectively reduce reliance on target-domain data while maintaining high prediction accuracy. Despite certain dynamic response differences between the F-FEM model and the F-AS structure, transfer learning successfully improved prediction accuracy through fine-tuning the pre-trained F-FEM model with a small amount of F-AS response data. The CNN-LSTM-ATT model outperformed other models, especially in handling complex nonlinear time-series predictions, showing stronger robustness. Through transfer learning, the model can quickly adapt to new structures or experimental data without the need for large amounts of target-domain data, thus greatly reducing data collection and experimental costs.

Conclusions

This study employed a convolutional long short-term memory neural network model with attention mechanism (CNN-LSTM-ATT), which integrates the feature extraction capabilities of convolutional neural networks (CNN), the information identification ability of attention mechanism (ATT), and sequential modeling proficiency of long short-term memory networks (LSTM). The model aims to simulate the nonlinear dynamic response of building structures under earthquakes. The efficacy of the CNN block and ATT block was validated through ablation experiments and numerical model integration. Additionally, transfer learning was utilized to fine-tune models for predicting the seismic response of different target building structures. The main conclusions can be summarized as follows:

(1) The performance of CNN-LSTM-ATT was validated using a numerical model that simulated a twelve-story frame shear wall structure. Ground acceleration was selected as the input, while inter-story displacements of the sixth level were used as the output. Results from ablation experiments demonstrated that LSTM-ATT and CNN-LSTM outperformed LSTM, with CNN-LSTM-ATT exhibiting the highest predictive accuracy. This affirms the model’s capability to predict the nonlinear seismic response with low error, demonstrating its proficiency in capturing nonlinear features.

(2) Two scenarios were devised, including transfer from finite element models with various parameters of the same structure, and transfer from finite element models to the same actual structures. Additionally, the impact of different input time steps on the prediction results of model was compared. As the number of input time steps increased, the prediction accuracy of the model also gradually improved. The outcomes highlight the effectiveness of model-based transfer learning strategies in enhancing the predictive accuracy, validating the robustness of fine-tuning in seismic response prediction.

(3) Compared to finite element models, the model after fine-tuning effectively transferred the structural nonlinear behaviors trained from numerical models to actual structures, resulting in more accurate predictions of the target structure’s response. The fine-tuned models demonstrated higher accuracy in various scenarios compared to finite element models, highlighting their robust applicability and versatility.

(4) In the field of seismic engineering, the proposed CNN-LSTM-ATT model can be used for rapid assessment of structural nonlinear responses under seismic actions, providing data support for seismic design optimization and structural health monitoring. Although this study primarily focused on a specific type of structure (i.e., a twelve-story shear wall-frame structure), the proposed method is also applicable to other types of structures, such as bridges, high-rise buildings, and underground structures, making the findings potentially generalizable across different structural types.

A limitation of this study is its reliance on numerical models for initial training, which may not fully capture the complexities and variations of real-world structures. Additionally, the fine-tuning process was conducted using a limited amount of real structural data, which was effective for adapting the model to the target domain, but it would not fully encompass all possible behaviors of structures under various seismic conditions. Although pre-training on finite element simulations significantly reduces the reliance on costly experimental or field data, a small amount of target-domain data (e.g., only 1–2 s) is still required for fine-tuning and adapting to specific structural configurations. In the current small-scale trained model, transfer learning cannot be realized without any input data. Future improvements could include integrating data augmentation techniques (e.g., simulating different types of seismic waves or varying structural parameters) to generate more diverse training samples. Further, the application of unsupervised domain adaptation methods could reduce the dependency on target domain data, allowing the model to better adapt to new domains without the need for labeled structural data. Therefore, further studies are needed to enhance the generalization of the proposed method across various structural types and seismic scenarios.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by the National Natural Science Foundation of China (Grant No. 52278498 and No. 51878268), the Huxiang Youth Talent Support Program of Hunan Province, China (Grant No. 2021RC3041), the Natural Science Foundation of Hunan Province, China (Grant No. 2020JJ4195), a grant (RS-2022-00143493) from Digital-Based Building Construction and Safety Supervision Technology Research Program funded by Ministry of Land, Infrastructure and Transport of Korean Government, and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2024-00333882 and RS-2024-00455788).

ORCID iDs

Gao Ma

Zhenhao Zhang

References

Ahmed

Mangalathu

Jeon

J-S

(2022) Seismic damage state predictions of reinforced concrete structures using stacked long short-term memory neural networks. Journal of Building Engineering 46: 103737.

Ayoub

Deü

J-F

Larbi

, et al. (2022) Application of the POD method to nonlinear dynamic analysis of reinforced concrete frame structures subjected to earthquakes. Engineering Structures 270(12): 114854.

Çavdar

Bayraktar

(2013) Pushover and nonlinear time history analysis evaluation of a RC building collapsed during the Van (Turkey) earthquake on October 23, 2011. Natural Hazards 70(1): 657–673.

Das

Choudhury

(2019) Evaluation of effective stiffness of RC column sections by support vector regression approach. Neural Computing and Applications 32(11): 6997–7007.

GB50010-2010 (2015) Code for Design of Concrete Structures (GB50010-2010). Beijing: Ministry of Housing and Urban-Rural Development of the People's Republic of China.

GB50011-2010 (2010) Code for Seismic Design of Buildings (GB50011-2010). Beijing: Ministry of Housing and Urban-Rural Development of the People's Republic of China.

Ghosh

Roy

Chakraborty

(2019) Kriging metamodeling-based Monte Carlo simulation for improved seismic fragility analysis of structures. Journal of Earthquake Engineering 25(7): 1316–1336.

Guo

Alam

Wang

, et al. (2020) Optimal intensity measures for probabilistic seismic demand models of a cable-stayed bridge based on generalized linear regression models. Soil Dynamics and Earthquake Engineering 131: 106024.

Hoang

Phan

Nguyen

, et al. (2021) Kriging metamodel-based seismic fragility analysis of single-bent reinforced concrete highway bridges. Buildings 11(6): 238.

10.

Huang

Chen

(2021) Deep learning for nonlinear seismic responses prediction of subway station. Engineering Structures 244: 112735.

11.

Jahangir

Rezazadeh Eidgahee

(2021) A new and robust hybrid artificial bee colony algorithm – ANN model for FRP-concrete bond strength evaluation. Composite Structures 257: 113160.

12.

Kalakonas

Silva

(2021) Seismic vulnerability modelling of building portfolios using artificial neural networks. Earthquake Engineering & Structural Dynamics 51(2): 310–327.

13.

Kent

Park

(1971) Flexural members with confined concrete. Journal of the Structural Division 97(7): 1969–1990.

14.

Chen

(2022) Fast seismic response estimation of tall pier bridges based on deep learning techniques. Engineering Structures 266: 114566.

15.

Liao

Lin

Zhang

, et al. (2023) Attention-based LSTM (AttLSTM) neural network for seismic response modeling of bridges. Computers & Structures 275: 106915.

16.

Wang

(2013) Experimental study of the seismic behavior of an earthquake-damaged reinforced concrete frame structure retrofitted with basalt fiber-reinforced polymer. Journal of Composites for Construction 17(6): 04013002.

17.

Mangalathu

Heo

Jeon

J-S

(2018) Artificial neural network based multi-dimensional fragility development of skewed concrete bridge classes. Engineering Structures 162: 166–176.

18.

Moodi

Mousavi

Ghavidel

, et al. (2018) Using response surface methodology and providing a modified model using whale algorithm for estimating the compressive strength of columns confined with FRP sheets. Construction and Building Materials 183: 163–170.

19.

Ning

Xie

Sun

(2023) LSTM, WaveNet, and 2D CNN for nonlinear time history prediction of seismic responses. Engineering Structures 286: 116083.

20.

Glisic

Park

, et al. (2020a) Neural network-based seismic response prediction model for building structures using artificial earthquakes. Journal of Sound and Vibration 468: 115109.

21.

Lee

Park

(2020b) Damage localization method for building structures based on the interrelation of dynamic displacement measurements using convolutional neural network. Structural Control and Health Monitoring 27(8): e2578.

22.

Park

(2020c) Seismic response prediction method for building structures using convolutional neural network. Structural Control and Health Monitoring 27(5): e2519.

23.

Perez-Ramirez

Amezquita-Sanchez

Valtierra-Rodriguez

, et al. (2019) Recurrent neural network model with Bayesian training and mutual information for response prediction of large buildings. Engineering Structures 178: 603–615.

24.

Sadeghi Eshkevari

Takáč

Pakzad

, et al. (2021) DynNet: physics-based neural architecture design for nonlinear structural response modeling and prediction. Engineering Structures 229(1): 111582.

25.

Tran

V-L

Kim

S-E

(2020) A practical ANN model for predicting the PSS of two-way reinforced concrete slabs. Engineering with Computers 37(3): 2303–2327.

26.

Wang

Liu

Xin

(2021) Bond strength prediction of concrete-encased steel structures using hybrid machine learning method. Structures 32: 2279–2292.

27.

Wang

Noori

, et al. (2022) Probabilistic seismic response prediction of three-dimensional structures based on Bayesian convolutional neural network. Sensors (Basel) 22(10): 3775.

28.

Wang

Noori

, et al. (2023) Seismic response prediction of structures based on Runge-Kutta recurrent neural network with prior knowledge. Engineering Structures 279: 115576.

29.

Chen

Shen

, et al. (2022) Recursive long short-term memory network for predicting nonlinear structural seismic response. Engineering Structures 250: 113406.

30.

Yassin

MHM

(1994) Nonlinear Analysis of Prestressed Concrete Structures under Monotonic and Cyclic Loads. Berkeley: University of California.

31.

Zeng

Zhao

, et al. (2024) A fuzzy computational framework for the train-bridge system based on Chebyshev polynomials method. Structures 70: 107771.

32.

Zhang

(2017) Seismic vulnerability analysis of RC bridges based on Kriging model. Journal of Earthquake Engineering 23(2): 242–260.

33.

Zhang

Chen

, et al. (2019) Deep long short-term memory networks for nonlinear structural seismic response prediction. Computers & Structures 220: 55–68.

34.

Zhang

Liu

Sun

(2020) Physics-guided convolutional neural network (PhyCNN) for data-driven seismic response modeling. Engineering Structures 215: 110704.

35.

Zhang

Xie

Tang

, et al. (2024) High-speed railway seismic response prediction using CNN-LSTM hybrid neural network. Journal of Civil Structural Health Monitoring 14(5): 1125–1139.

36.

Zhao

Wei

Zhang

, et al. (2024) Safety analysis of high-speed trains on bridges under earthquakes using a LSTM-RNN-based surrogate model. Computers & Structures 294: 107274.