Association Between Heart Rate at Diagnosis and Long-Term Recurrence Risk of Pulmonary Embolism in a Historical Cohort Study of Elder Women

Abstract

To investigate the association between heart rate (HR) at diagnosis and long-term pulmonary embolism (PE) recurrence among elderly (≥ 50 year-old) female patients after acute PE (APE). Hospitalized patients with APE were grouped separately according to whether they experienced recurrent PE and whether the HR was < 80 beats/min. Logistic regression and COX regression analysis were employed to assess the risk of PE recurrence. Kaplan–Meier method was applied to compare the recurrence-free survival of PE recurrence. Eighty-five patients were included, including 24 ones with HR < 80 beats/min and 11 recurrent PE cases. The mean time of PE recurrence were 71.7 ± 26.9 months (n = 6) and 27.7 ± 25.2 months (n = 5) among the patients with low HR and with high HR, respectively (P < .001). The HR (< 80 beats/min) was a negative predictor of PE recurrence (OR 0.071 (0.090-0.572), P = .013; HR 0.091 (0.016-0.523), P = .007), even after the adjustment for age, BMI, albumin, risk stratification, surgery, immobility ≥ 4 days, the blood cells counts, bilirubin and complications. The cumulative recurrence-free rates of PE recurrence at the 1^st-, 2^nd-, 5^th-, and 10^th-years for the low HR group were 100%, 100%, 87.5%, and 58.3%, compared to the 1^st-, 2^nd-, and 3^rd-years of 94.0%, 93.4%, and 48.0% for the high HR group (log-rank = 0.019). The low HR (< 80 beats/min at diagnosis) among elderly (≥ 50 years old) female patients at APE diagnosis would benefit to the long-term PE recurrence. But limited recurrent cases should be noted.

Keywords

acute pulmonary embolism heart rate long-term PE recurrence elder women

Introduction

Acute pulmonary embolism (APE) is a fatal disease with high mortality, of whom nearly 1 in 5 patients die within 1 year.^1–5 The cost of anticoagulation therapy and the complications would increase patients’ financial burden and decline the quality of life.^3,6 A significant part of patients with pulmonary embolism (PE) may progress to chronic thromboembolic pulmonary hypertension, along with other functional impairments such as post-thrombotic syndrome, and in severe cases, mortality.^2,7–10

Patients with APE face enduring risk of PE recurrence. Some studies suggested the PE recurrence was 1.7% to 6.7%.^11,12 The risk of PE doubles among patients over 40 years old, which means that PE is primarily a disease of the old-age patients.^13–16 PE recurrence associates with many factors, including age, estrogen-related therapy, family history, obesity, and others. Moreover, several studies have observed a higher recurrence rate of PE among elderly women compared to elderly men, whereas among younger patients, this recurrence appears to be less frequent in women than in men.^15,17–20 In general, the elder women would have a higher risk of PE suffering rather than men at same age. Thus, it is important to focus on the elder women with PE.

As a rapidly available risk stratification parameter, the heart rate (HR) at diagnosis of APE attracted our attention again for relating to the mortality and short-term recurrence, usually accompanying with the cut-off value of HR > 100 beats/min in some scores.^21–23 However, a recent retrospective study uncovered the lower HR indicating the lower in-hospital death of patients with APE.²² Meanwhile, the results from the Registro Informatizado de la Enfermedad TromboEmbólica registry showed that the lower HR (HR < 80 beats/min) was significantly connected to the lower 30-day PE recurrence status in a large population.²³ Regarded as one important component of the sPESI/PESI, the relationship between HR and the prognosis of APE is not that new here. But few study focused on the effect of the low HR on long-term recurrence among the elderly female patients with APE.

Therefore, we performed this retrospective study to identify the association between HR and the long-term recurrence rate among the elderly female patients after APE.

Methods

Study Design and Participants

The patients were gathered from January 2012 to June 2018 when they were firstly admitted into the Second Xiangya Hospital of Central South University because of APE. The diagnosis of PE adhered to the 2019 European Society of Cardiology (ESC) guideline.²⁴ This study received approval from the Ethics Committee of the Second Xiangya Hospital of Central South University, the Ethics number: (020) Ethical review [Clinical research] No.107 and according to the Declaration of Helsinki. As this was a retrospective study, informed consent was exempted.

The inclusion criteria include: (1) Firstly confirmation of APE through computed tomography pulmonary angiography (CTPA) or pulmonary angiography at our center; (2) Patients were biological women with age ≥ 50 years old (because there were several studies that used the cut-off age of 50 years old,^25,26 we followed the cut-off). The exclusion criteria followed by: (1) Lacking the data of HR at diagnosis; (2) Cases of in-hospital death because of first APE; (3) Death after discharge in 3 months; (4) Loss to follow-up.

Data Collection and Follow-up

We collected demographic and clinical data, including age, sex, body mass index (BMI), medical history, vital signs, comorbidities, blood routine, coagulation function, and biochemical indices. The patients were followed up until December 2023 through telephone interviews and outpatient clinic visits. Recurrent PE was tracked as the observational endpoint. Overall non-recurrence duration was defined from the date of diagnostic APE to recurrent PE. Non-recurrent patients were censored on the date of the last clinical contact. There were 6 patients died after discharge in 3 months and the family members did not know the cause, so we excluded those cases.

Statistical Analysis

Multiple imputation was used for missing values. The variables were expressed as mean ± standard deviation or median (P25, P75) for continuous variables with normal distribution or non-normal distribution respectively. Categorical variables were expressed as the number of patients and relative frequencies (n, %). The statistical analysis of the differences in groups was performed by t-test or Mann-Whitney U-test for continuous indices and χ² test for categorical indices. The maximum likelihood ratio forward logistic regression and the maximum likelihood ratio COX proportional hazards were applied to examine the association between HR and the recurrence of PE. The Kaplan–Meier method was used to estimate the proportion of patients without recurrence compared by The Log Rank test. All comparisons were employed with a 2-sided test through SPSS (Statistical Package for Social Science, Chicago, IL) version 25.0, and a P-value less than .05 was considered significant. All figures were concluded via GraphPad Prism (San Diego, CA, USA) version 9.4.1 software.

Results

Clinical Features of This Cohort at Baseline

A total of 541 patients with APE were screened in this study, and 85 elderly female cases were firstly confirmed as APE at our center according to the inclusion and exclusion criteria (all patients received anticoagulant therapy). The flowchart is illustrated in Figure 1. The median time of follow-up was 21.8 (7.9, 39.1) months (range 1.0-126.0). The mean age of those patients at diagnosis was 66 ± 9 years old. Of those, 1 case had smoking history (1.2%) and 1 had history of alcoholism (1.2%). Twenty-four patients (28.2%) had a low HR (<80 beats/min at diagnosis). The systolic blood pressure (SBP) and diastolic blood pressure (DBP) were 125 (111, 140) mm Hg and 75 (67, 82) mm Hg, respectively. There were 10 cases (11.8%) that experienced surgery (1 month before diagnosis) and 13 cases (15.3%) had a history of immobility ≥ 4 days (1 week before diagnosis). Over half ones were with simplified PE severity index (sPESI) ≥ 1 (50.6%). Most patients were at low (30.6%) and medium (65.9%) ESC risk stratification. The most comorbidity was hypertension (37.2%), followed by cancer (27.1%), diabetes (10.6%), and coronary heart disease (8.2%).

Figure 1.

The flowchart of APE inclusion, the inclusion and the exclusion.

In the aspect of blood routine, the white blood cell (WBC) counts, neutrophil (N) cell counts, and monocyte (M) counts were 8.66 (6.20, 11.63)×10⁹/L, 6.39 (4.22, 9.45)×10⁹/L, and 0.47 (0.33, 0.60)×10⁹/L, respectively. Neutrophil/lymphocyte ratio (NLR), platelet/lymphocyte ratio (PLR), and monocyte/lymphocyte ratio (MLR) were 4.65 (2.85, 7.75), 155.36 (116.39, 216.35), and 0.33 (0.23, 0.49), respectively. Other data were shown in Table 1.

Table 1.

Clinical Characteristics of Elder Female Patients After APE at Diagnosis.

Variables	n = 85
Demographics
Age (years old)	66 ± 9
BMI (kg/m²)	23.37 ± 2.52
Previous history, n (%)
Smoking history	1 (1.2%)
History of alcoholism	1 (1.2%)
HR (<80 beats/min at diagnosis)	24 (28.2%)
SBP (mm Hg)	125 (111, 140)
DBP (mm Hg)	75 (67, 82)
sPESI ≥ 1	43 (50.6%)
ESC risk stratification, n (%)
Low risk	26 (30.6%)
Medium risk	56 (65.9%)
High risk	3 (3.5%)
Comorbidity, n (%)
Coronary heart disease	7 (8.2%)
Diabetes	9 (10.6%)
Hypertension	32 (37.6%)
Atrial fibrillation	4 (4.7%)
Cancer	23 (27.1%)
COPD	1 (1.2%)
Risk of PE
Surgery (1 month before diagnosis)	10 (11.8%)
Immobility ≥ 4 days (1 week before diagnosis)	13 (15.3%)
Blood routine
WBC (10⁹/L)	8.66 (6.20, 11.63)
RBC (10¹²/L)	3.93 (3.58, 4.21)
PLT (10⁹/L)	199 (169, 253)
N (10⁹/L)	6.39 (4.22, 9.45)
L (10⁹/L)	1.31 (1.05, 1.69)
NLR	4.65 (2.85, 7.55)
E (10⁹/L)	0.04 (0.01, 0.10)
M (10⁹/L)	0.47 (0.33, 0.60)
PLR	155.36 (116.39, 216.35)
MLR	0.33 (0.23, 0.49)
MPV (fl)	10.7 (10.0, 11.4)
Coagulation function
D-Dimer increased	84 (98.9%)
APTT (s)	35.12 ± 7.31
PT (s)	12.9 (11.4, 14.1)
fibrinogen (g/L)	3.40 (2.89, 4.89)
INR	1.00 (0.93, 1.12)
Biochemical index
Albumin (g/L)	34.9 ± 4.7
TBil (µmol/l)	10.70 (7.25, 13.50)
Uric acid (µmol/l)	266.9 (182.2, 344.7)
TNT (pg/ml)	9.83 (6.58, 12.10)
NT-proBNP (pg/ml)	287 (145, 959)
PE recurrence	11 (12.9%)

Abbreviations: APE, acute pulmonary embolism; PE, pulmonary embolism; BMI, body mass index; HR, heart rate; SBP, systolic blood pressure; DBP, diastolic blood pressure; sPESI, simplified pulmonary embolism severity index; ESC, European Society of Cardiology; COPD, chronic obstructive pulmonary disease; WBC, white blood cell; RBC, red blood cell; PLT, platelet; N, neutrophil; L, lymphocyte; NLR, neutrophil/lymphocyte ratio; E, eosinophil; M, monocyte; PLR, platelet/lymphocyte ratio; MLR, monocyte/lymphocyte ratio; MPV, mean platelet volume; APTT, activated partial thromboplastin time; PT, prothrombin time; INR, international standardized ratio; TBil, total bilirubin; TNT, troponin T; NT-proBNP, N terminal pro-brain natriuretic peptide.

Almost patients had an increased D-Dimer value (98.9%) and the mean activated partial thromboplastin time (APTT) was 35.15 ± 7.31 s. The mean/median values of albumin, total bilirubin (TBil), uric acid, troponin T (TNT), and NT-proBNP were 34.9 ± 4.7 g/L, 10.70 (7.25, 13.50) µmol/l, 266.9 (182.2, 344.7) µmol/l, 9.83 (6.58, 12.10) pg/ml and 287 (145, 995) pg/ml, respectively. The overall rate of recurrent PE was 12.9% (11 cases). The data are illustrated in Table 1.

Demographic and Clinical Indices at Baseline Between Recurrent and Non-Recurrent Subgroups

Dividing the patients into PE recurrent and PE non-recurrent subgroups, we found that the recurrent patients had a higher percentage with HR < 80 beats/min at diagnosis relative to the non-recurrent patients (6 (54.5%) vs 18 (24.3%), P = .039). As for the risk of PE, there were 6 cases (54.5%) that had a history of surgery and 8 cases (72.7%) of immobility ≥ 4 days before diagnosis, and also 2 cases (18.2%) had a history of surgery and 5 cases (45.5%) of immobility ≥ 4 days before recurrence (all P < .001 when compared to the non-recurrent group). In the aspect of blood routine, coagulation function and biochemical indices, APTT (41.00 ± 4.70 s vs 34.16 ± 7.24 s, P = .002) were significantly higher while N counts (4.38 (3.59, 6.72)×10⁹/L vs 6.76 (4.35, 9.45)×10⁹/L, P = .032) were significantly lower among recurrent patients than the non-recurrent patients. Of note, the WBC counts (7.21 (5.39, 6.72)×10⁹/L vs 8.70 (6.29, 11.63)×10⁹/L, P = .071) and TBil (7.60 (4.23, 10.45) µmol/l vs 10.85 (7.50, 14.15) µmol/l, P = .077) were obviously lower among the recurrent patients than the non-recurrent patients. The median time of PE recurrence was 38.5 (8.5, 54.5) months. Other signs, comorbidity, risk stratification, and blood variables had no significant difference. Above were generated in Table 2.

Table 2.

Comparison of Clinical Characteristics Between the Recurrent and Non-Recurrent PE Groups.

Variables	Recurrent PE (n = 11)	Non-Recurrent PE (n = 74)	P-value
Demographics
Age (years old)	66 ± 7	65 ± 9	.570
BMI (kg/m²)	23.44 ± 2.94	23.20 ± 2.42	.114
Previous history, n (%)
Smoking history	0 (0%)	1 (1.4%)	.700
History of alcoholism	0 (0%)	1 (1.4%)	.700
HR (<80 beats/min at diagnosis)	6 (54.5%)	18 (24.3%)	.039
Median time of PE recurrence (months)	38.5 (8.5, 54.5)	-	-
SBP (mm Hg)	127 (112, 138)	125 (112, 141)	.896
DBP (mm Hg)	76 (71, 86)	75 (64, 82)	.291
sPESI ≥ 1	3 (25.0%)	40 (54.8%)	.099
ESC risk stratification, n (%)			.059
Low risk	6 (50.0%)	20 (27.4%)
Medium risk	6 (50.0%)	50 (68.5%)
High risk	0 (0%)	3 (4.1%)
Comorbidity, n (%)
Coronary heart disease	0 (0%)	7 (9.6%)	.290
Diabetes	1 (8.3%)	8 (11.0%)	.863
Hypertension	5 (41.7%)	27 (37.0%)	.569
Atrial fibrillation	0 (0%)	4 (5.5%)	.432
Cancer	3 (27.3%)	20 (27.0%)	.986
COPD	0 (0%)	1 (1.4%)	.700
Risk of PE
Surgery (1 month before diagnosis)	6 (54.5%)	4 (5.4%)	<.001
Immobility ≥ 4 days (1 week before diagnosis)	8 (72.7%)	5 (6.8%)	<.001
Surgery (1 month before PE recurrence)	2 (18.2%)	0 (0%)	<.001
Immobility ≥ 4 days (1 week before PE recurrence)	5 (45.5%)	6 (8.1%)	<.001
Blood routine
WBC (10⁹/L)	7.21 (5.39, 9.96)	8.70 (6.29, 11.63)	.071
RBC (10¹²/L)	3.93 (3.85, 4.22)	3.95 (3.55, 4.21)	.538
PLT (10⁹/L)	202 (198, 228)	192 (168, 259)	.605
N (10⁹/L)	4.38 (3.59, 6.72)	6.76 (4.35, 9.45)	.032
L (10⁹/L)	1.44 (1.26, 1.64)	1.30 (0.98, 1.72)	.283
NLR	3.17 (2.13, 5.71)	5.20 (2.98, 7.95)	.299
E (10⁹/L)	0.09 (0.02, 0.16)	0.04 (0.01, 0.09)	.108
M (10⁹/L)	0.35 (0.26, 0.58)	0.48 (0.34, 0.60)	.132
PLR	127.04 (105.12, 178.57)	157.76 (116.39, 223.10)	.329
MLR	025 (0.20, 0.37)	0.34 (0.24, 0.52)	.195
MPV (fl)	10.1 (9.7, 11.1)	10.7 (10.1, 11.6)	.074
Coagulation function
D-Dimer increased	11 (100%)	73 (98.6%)	.700
APTT (s)	41.00 ± 4.70	34.16 ± 7.24	.002
PT (s)	12.9 (12.6, 13.5)	12.8 (11.3, 14.1)	.377
Fibrinogen (g/L)	4.03 (3.00, 6.22)	3.31 (2.87, 4.80)	.455
INR	1.00 (0.94, 1.06)	1.01 (0.93, 1.12)	.932
Biochemical indices
Albumin (g/L)	36.2 ± 3.5	34.7 ± 4.9	.292
TBil (µmol/l)	7.60 (4.23, 10.45)	10.85 (7.50, 14.15)	.077
Uric acid (µmol/l)	289 (185, 339)	262 (182, 345)	.504
TNT (pg/ml)	9.83 (4.26, 9.98)	9.83 (6.54, 13.0)	.690
NT-proBNP (pg/ml)	237 (153,2505)	310 (134, 950)	.979

Demographic and Clinical Indices at Baseline Between High HR and Low HR Subgroups

Compared to the patients with higher HR (≥ 80 beats/min), the patients with lower HR (< 80 beats/min) had a higher percentage with coronary heart disease (5 (21.7%) vs 2 (3.2%), P = .008) but a lower percentage with diabetes (0 (0%) vs 9 (14.5%), P = .047). The WBC counts (6.33 (5.33, 9.99)×10⁹/L vs 9.30 (7.00, 12.00)×10⁹/L, P = .011), the N counts (4.51 (3.83, 7.87)×10⁹/L vs 6.78 (4.65, 9.80)×10⁹/L, P = .032), and the M counts were (0.40 (0.27, 0.50)×10⁹/L vs 0.49 (0.34, 0.61)×10⁹/L, P = .033) significantly lower among the patients with lower HR (< 80 beats/min) rather than the patients with higher HR (≥ 80 beats/min). Relative to the patients with higher HR (≥ 80 beats/min), the PLT counts (198 (131, 202)×10⁹/L vs 223 (176, 271)×10⁹/L, P = .060) was obviously lower but albumin (36.5 ± 4.4 g/L vs 34.3 ± 4.8 g/L, P = .055) was obviously higher among the patients with low HR (< 80 beats/min). The mean time of PE recurrence were 71.7 ± 26.9 months (n = 6) and 27.7 ± 25.2 months (n = 5) among the patients with low HR (< 80 beats/min) and the patients with high HR (≥ 80 beats/min), respectively, which was statistically significant (P < .001). Other signs, comorbidity, risk stratification, and blood variables had no significant difference. Data were generated in Table 3.

Table 3.

Comparison of Clinical Characteristics Between High Heart Rate and Low Heart Rate Groups.

Variables	HR < 80 beats/min (n = 24, at diagnosis)	HR ≥ 80 beats/min (n = 61, at diagnosis)	P-value
Demographics
Age (years old)	67 ± 9	65 ± 10	.534
BMI (kg/m²)	23.76 ± 2.58	23.04 ± 2.39	.284
Previous history, n (%)
Smoking history	1 (4.2%)	0 (0%)	.109
History of alcoholism	0 (0%)	1 (1.6%)	.528
PE recurrence
Cases, n (%)	6 (26.1%)	5 (8.1%)	.038
Mean time of PE recurrence (months)	71.7 ± 26.9	27.7 ± 25.2	<.001
SBP (mm Hg)	128 (115, 135)	125 (110, 140)	.545
DBP (mm Hg)	76 (71, 80)	74 (64, 84)	.529
sPESI ≥ 1	12 (52.2%)	31 (50.0%)	.946
ESC risk stratification, n (%)			.268
Low risk	5 (21.7%)	21 (33.9%)
Medium risk	18 (78.3%)	38 (61.3%)
High risk	0 (0%)	3 (4.8%)
Comorbidity, n (%)
Coronary heart disease	5 (21.7%)	2 (3.2%)	.008
Diabetes	0 (0%)	9 (14.5%)	.047
Hypertension	12 (52.2%)	20 (32.3%)	.140
Atrial fibrillation	1 (4.2%)	3 (4.9%)	.883
Cancer	5 (21.7%)	18 (29.0%)	.418
COPD	0 (0%)	1 (1.6%)	.528
Risk of PE
Surgery (1 month before diagnosis)	2 (8.3%)	8 (13.1%)	.538
Immobility ≥ 4 days (1 week before diagnosis)	3 (12.5%)	10 (16.4%)	.653
Surgery (1 month before PE recurrence)	0 (0%)	2 (3.3%)	.369
Immobility ≥ 4 days (1 week before PE recurrence)	3 (12.5%)	8 (13.1%)	.939
Blood routine
WBC (10⁹/L)	6.33 (5.33, 9.99)	9.30 (7.00, 12.00)	.011
RBC (10¹²/L)	3.88 (3.71, 4.05)	3.94 (3.50, 4.24)	.693
PLT (10⁹/L)	198 (131, 202)	223 (176, 271)	.060
N (10⁹/L)	4.51 (3.83, 7.87)	6.78 (4.65, 9.80)	.032
L (10⁹/L)	1.28 (0.91, 1.51)	1.36 (1.06, 1.93)	.488
NLR	3.98 (3.08, 5.85)	5.13 (2.72, 9.20)	.243
E (10⁹/L)	0.05 (0.01, 0.09)	0.04 (0.02, 0.11)	.856
M (10⁹/L)	0.40 (0.27, 0.50)	0.49 (0.34, 0.61)	.033
PLR	153.08 (120.48, 187.84)	155.43 (112.84, 225.74)	.426
MLR	0.31 (0.22, 0.37)	0.33 (0.24, 0.56)	.186
MPV (fl)	11.0 (10.0, 11.5)	10.5 (9.9, 11.6)	.535
Coagulation function
D-Dimer increased	24 (100%)	60 (98.4%)	.528
APTT (s)	35.15 ± 5.75	34.72 ± 7.94	.712
PT (s)	13.3 (12.3, 14.0)	12.8 (11.1, 14.2)	.343
Fibrinogen (g/L)	3.00 (2.68, 4.55)	3.44 (3.08,5.19)	.096
INR	1.04 (0.98, 1.06)	1.00 (0.92, 1.12)	.781
Biochemical index
Albumin (g/L)	36.5 ± 4.4	34.3 ± 4.8	.055
TBil (umol/l)	10.85 (7.30, 14.85)	10.70 (7.48, 13.50)	.409
Uric acid (umol/l)	275.2 (208.5, 358.7)	249.2 (180.8, 344.2)	.259
TNT (pg/ml)	9.83 (5.57, 14.80)	9.83 (7.78, 10.82)	.831
NT-proBNP (pg/ml)	304 (145, 1180)	287 (142, 956)	.996

Association Between HR and PE Recurrence

Table 4 illustrates the association between HR and the PE recurrence. Patients with low HR (< 80 beats/min) at APE diagnosis was associated with increased odds of having the PE recurrence among the elderly female patients after APE when taking age and BMI into consideration (odds ratio 0.299, 95% confidence interval: 0.089, 0.999, P = .049) or taking age, BMI, albumin, monocyte, platelet, neutrophil, WBC, NLR, coronary heart disease (yes or no), sPESI ≥ 1, total bilirubin, surgery before diagnosis or recurrence, immobility ≥ 4 days before diagnosis or recurrence, diabetes, atrial fibrillation, risk stratification and cancer into consideration (odds ratio 0.268, 95% confidence interval: 0.173, 0.983, P = .047). Still, the adjusted COX proportional hazards model showed that patients with low HR (< 80 beats/min) at APE diagnosis had declined hazard of the PE recurrence when taking age and BMI into consideration (hazard ratio 0.142, 95% confidence interval: 0.002, 0.813, P = .033), or even after adjustment for age, BMI, albumin, monocyte, platelet, neutrophil, WBC, NLR, coronary heart disease (yes or no), sPESI ≥ 1, total bilirubin, surgery before diagnosis or recurrence, immobility ≥ 4 days before diagnosis or recurrence, diabetes, atrial fibrillation, risk stratification and cancer (hazard ratio 0.053, 95% confidence interval: 0.004, 0.656, P = .022).

Table 4.

Effects of Heart Rate (<80 Beats/min, at Diagnosis) on PE Recurrence Among the Elderly Female Patients After APE.

	Odds ratio			Hazard ratio
	Standardized β	95% Confidence Interval	P	Standardized β	95% Confidence Interval	P
Model A	0.299	0.089, 0.999	.049	0.142	0.002, 0.813	.033
Model B	0.268	0.173, 0.983	.047	0.053	0.004, 0.656	.022

Notes: Model A, adjusted for age and BMI; Model B, adjusted for age, BMI, albumin, monocyte, platelet, neutrophil, white blood cell, coronary heart disease (yes or no), sPESI ≥ 1 (yes or no), total bilirubin, surgery before diagnosis or recurrence, Immobility ≥ 4 days before diagnosis or recurrence, diabetes, atrial fibrillation, risk stratification and cancer.

Abbreviations: APE, acute pulmonary embolism; PE, pulmonary embolism.

PE Recurrence-Free Assessment Between High HR and Low HR Subgroups

The 1^st, 2^nd, 5^th, and 10^th-year cumulative PE recurrence-free rates were found to be 100%, 100%, 87.5%, and 58.3% in the low HR (< 80 beats/min at diagnosis) group, respectively. But The 1^st, 2^nd, and 3^rd-year cumulative PE recurrence-free rates were found to be 94.0%, 93.4%, and 48.0% in the high HR (≥ 80 beats/min at diagnosis) group, respectively. With respect to PE recurrence-free assessment, it showed better PE recurrence-free survival in the low HR (< 80 beats/min at diagnosis) group than the high HR (≥ 80 beats/min at diagnosis) group among elderly female patients after APE (log-rank = 0.019, Figure 2).

Figure 2.

Comparison of estimated recurrence-free survival between elderly female patients after APE with HR < 80 beats/min at diagnosis and with HR ≥ 80 beats/min at diagnosis.

Discussion

Our study demonstrated some findings as follows: (I) Low HR (HR < 80 beats/min at diagnosis) served as a negative predictor for the increase in PE recurrence, as evidenced by both odds ratio and hazard ratio analyses, which still persisted even after adjusting for pertinent variables including age, BMI, albumin levels, monocyte count, platelet count, neutrophil count, WBC count, NLR, presence of coronary heart disease, sPESI score (≥ 1, yes or no), total bilirubin levels, and cancer diagnosis; (II) The low HR (HR < 80 beats/min at diagnosis) group demonstrated significantly higher recurrence-free survival compared to the high HR (HR ≥ 80 beats/min at diagnosis) group; (III) In female patients aged 50 years or older following APE, those with a HR less than 80 beats/min at diagnosis experienced a delayed recurrence time.

Despite the relatively small number of recurrent PE cases, particularly in the low HR group, our findings align with prior research. Chopard et al¹² observed 74 recurrent venous thromboembolism (VTE) cases among 858 patients undergoing extended anticoagulant therapy post-PE diagnosis during a follow-up period of up to 2.1 ± 0.3 years after the initial phase of 3 to 6 months. Similarly, a multicenter Swiss cohort reported a 13.1% cumulative incidence of VTE recurrence over a 3-year follow-up period.¹³ Long-term recurrence rates were also evident in studies with extended follow-up periods, such as a retrospective cohort spanning 2004 to 2013, which revealed a 22.3% overall recurrence rate among PE patients.²⁷ Coincidentally, the data from the National Readmission Data (NRD, USA) 2013 to 2014 showed overall 7.3% of the PE readmission after the last PE diagnosis.¹⁶ One respectively cohort study²⁸ showed that patients suffered from APE had a recurrent risk of APE. The authors gathered 69 cases with APE, of who 8 patients finally recurred APE during the 6-month follow-up, which was little higher than our study.

HR, as the marker of short-term death or recurrence due to APE, is well established alone or along with sPESI/PESI,^22,23 but stays few regarding to the PE recurrence alone, and warrants further exploration in the context of long-term recurrence risk. The COMMAND VTE (COntemporary ManageMent AND outcomes in patients with Venous ThromboEmbolism) Registry²² from Japan corroborates our findings, demonstrating a lower PE recurrence rate in patients with a lower HR within 30 days of diagnosis. The authors divided the large numbers of patients with APE into 5 groups, including HR < 80 beats/min, 80 ≤ HR < 100 beats/min, 100 ≤ HR <110 beats/min, 100 ≤ HR <110 beats/min, and HR ≥110 beats/min, and discovered that the HR < 80 beats/min group had the lowest PE recurrent rate (2.7%) in 30 days after APE diagnosis. We found that patients with low HR exhibited a lower risk of PE recurrence than the high HR group in 2 years (0% vs 6.6%). Moreover, our study highlights the potential utility of HR in risk stratification, as indicated by the absence of recurrent PE cases within 90 days among low-risk patients in a study by Bledsoe et al.²⁹ The cause might be the short-term observation and small sample size than the Japanese cohort study.

Due to low recurrence rate, the recurrent PE could not that easy be revealed in a short-term prognosis +/− small sample size, which means that the recurrent cases would be more if prolonging the follow-up or including more patients. Despite noting a notable discrepancy in HR between recurrent and non-recurrent VTE cohorts (84.5 IQR 25 vs 88 IQR 26 beats/min, P < .05) during a median follow-up of 30 months, the study did not establish HR as an independent predictor of VTE recurrence.²⁷ Conversely, our current investigation, featuring an extended follow-up period ranging from 1 to 126 months, identified 11 cases of recurrent PE. Remarkably, a diminished HR (< 80 beats/min) emerged as a potential marker of post-APE recurrence. These findings underscore the necessity of prolonged observation periods in uncovering recurrent PE instances and suggest a plausible correlation between HR and PE recurrence. It is imperative to acknowledge the inherent constraints of such analyses, encompassing sample size limitations and potential confounders. Nevertheless, these insights significantly enrich the nuanced comprehension of factors influencing recurrent PE, thus furnishing valuable guidance for clinical risk assessment and therapeutic interventions.

Sex-specific analyses revealed intriguing patterns, with younger men exhibiting higher recurrence rates than women of the same age, while elder women demonstrated higher recurrence rates than men of equivalent age.²³ Totally, women were more possible to recur PE than men.³⁰ The sex-specific phenotype is not well understood yet. But the estrogen could help us comprehend. Estrogen may have great benefit in the blood vessel protection as an anti-inflammatory agent,³¹ on the other hand the use of estrogen may promote PE as a medication.^32,33 This might be that extra dose of estrogen could promote PE than physiological dose. Meanwhile, PE also has an aging phenotype, elder patients would explore more odds of PE morbidity, in which most PE patients would diagnose at an elder age.^{7,10,12,17,29} For the purpose of reducing age and sex bias, we chose the female participants with age ≥ 50 years old as study subjects.

In addition to our observations regarding HR and recurrence, our study revealed a noteworthy disparity in total bilirubin levels between the recurrent and non-recurrent groups (7.60 (4.23, 10.45) vs 10.85 (7.50, 14.15) µmol/l, P = .077). Some studies found that a high expression level of bilirubin at diagnosis was associated with the occurrence of PE^34,35 or the severe phenotype.³⁶ This phenomenon might imply the body at the acute state. Interestingly, prior researches have indicated a propensity for lower total bilirubin expression in the recurrent VTE group, particularly among patients with milder symptoms.^36,37 Furthermore, there appears to be a 73% higher risk of PE among individuals with total bilirubin levels below 8.6 µmol/L.³⁸ Conversely, a comprehensive study involving extensive data sets revealed that patients within the normal range of bilirubin were more susceptible to respiratory diseases when bilirubin expression was elevated.³⁹ Also, there was an extensive, statin-treated cohort (without liver disease or cardiovascular disease) study that found the association between low expression of bilirubin and elevated risk of cardiovascular diseases.⁴⁰ As for Asian population, Korean cohort exhibited a trend similar to the before two.⁴¹ In fact, bilirubin was thought to have antioxidant and anti-inflammatory effects via a bilirubin-biliverdin cycling mechanism, in which oxygen free radical was consumed.^42–44 This might help understand the low expression of bilirubin in recurrent PE group.

Moreover, our investigation unveiled reduced counts of immune cells (WBC, M, and N) in the recurrent group. The findings from the RIETE Registry had a opposite trend. The authors found that compared to WBC counts ranging from 4000 to 11000 cells per µl in cancer patients with APE at diagnosis, those with WBC over 11000 cells per µl at diagnosis were more likely to have recurrent PE (OR: 1.7, CI: 1.1-2.6).⁴⁵ Meanwhile, the other prospective cohort study discovered that among 303 VTE cases with anticoagulation treatment, 16 cases had VTE recurrence during a follow-up of at least 1 year, in whom the levels of neutrophils and monocytes were higher at initial VTE diagnosis than those from the non-recurrent ones.⁴⁶ However, leukocytes also could modulate the fibrinolysis via the expression and activation of fibrinolytic agents, participate in the resolution of thrombi and facilitate the clearance of coagulation factors through phagocytosis.^47–49 In our study, the lower expression of immune cells in the recurrent ones (shown in Supplemental Table 1) might be likely to suggest the relation to the resolution of thrombosis by leukocytes or others. This comprehensive mechanism should be explored in the future.

Conclusion

In female patients aged 50 years or older following APE, those with a HR less than 80 beats/min at diagnosis experienced a delayed recurrence time. Furthermore, the presence of a HR below 80 beats/min (yes or no, at diagnosis) emerged as a negative predictor of PE recurrence. Notably, patients belonging to the low HR (< 80 beats/min at diagnosis) group exhibited superior recurrence-free survival compared to those in the high HR (≥ 80 beats/min at diagnosis) group. But, the limited recurrent cases should be noted.

Limitations

This study also had some limitations. Firstly, this was a single-center and small-size pilot study. It could explain the numbers of PE recurrence were more in the low HR group. However, the mean time of PE recurrence was much less in the low HR group. Secondly, we did not include the dynamic variables at the time of PE recurrence, especially the HR. Thirdly, this study focused on the specific group of elder women aged 50 or older, but whether the association between HR and PE recurrence would exist in other groups should be explored in the future.

Supplemental Material

sj-docx-1-cat-10.1177_10760296241268432 - Supplemental material for Association Between Heart Rate at Diagnosis and Long-Term Recurrence Risk of Pulmonary Embolism in a Historical Cohort Study of Elder Women

Supplemental material, sj-docx-1-cat-10.1177_10760296241268432 for Association Between Heart Rate at Diagnosis and Long-Term Recurrence Risk of Pulmonary Embolism in a Historical Cohort Study of Elder Women by Yuan Li, MD, PhD, Fang Li, PhD, Meizhi Li, MS, Qiong Yi, MS, and Shangjie Wu, MD in Clinical and Applied Thrombosis/Hemostasis

Footnotes

Abbreviations list

Acknowledgments

All the work in this study was completed at the Department of Pulmonary and Critical Care Medicine, the Second Xiangya Hospital, Central South University, Changsha, China. The authors would thank all colleagues for the supportive work and all the participants for their devotion to scientific improvement.

Author Contributions

Conceptualization was performed by Shangjie Wu, Meizhi Li, Qiong Yi, and Yuan Li. Data curation was carried out by Yuan Li, Qiong Yi, Meizhi Li, and Fang Li. Formal analysis was conducted by Yuan Li. Funding acquisition was performed by Shangjie Wu and Yuan Li. Investigation was done by Yuan Li, Qiong Yi, and Meizhi Li; Methodology was developed by Shangjie Wu, Meizhi Li, Qiong Yi, and Yuan Li. Supervision was carried out by Shangjie Wu, Meizhi Li. and Qiong Yi. Visualization was conducted by Yuan Li. Original draft was written by Yuan Li. Review and editing were carried out by Yuan Li, Qiong Yi, Meizhi Li, Fang Li, and Shangjie Wu.

Data Availability

Data used to support the findings of this study are available from the corresponding author upon reasonable request.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Scientific Research Launch Project for new employees of the Second Xiangya Hospital of Central South University, and the National Key Clinical Specialty Construction Projects of China.

ORCID iD

Yuan Li

Supplemental Material

Supplemental material for this article is available online.

References

Heit

Spencer

White

. The epidemiology of venous thromboembolism. J Thromb Thrombolysis. Jan 2016;41(1):3–14. doi:https://doi.org/10.1007/s11239-015-1311-6

Hepburn-Brown

Darvall

Hammerschlag

. Acute pulmonary embolism: A concise review of diagnosis and management. Intern Med J. Jan 2019;49(1):15–27. doi:https://doi.org/10.1111/imj.14145

Hojen

Nielsen

Overvad

, et al. Long-term management of pulmonary embolism: A review of consequences, treatment, and rehabilitation. J Clin Med. Oct 10 2022;11(19):5970. doi:https://doi.org/10.3390/jcm11195970

Marti

John

Konstantinides

, et al. Systemic thrombolytic therapy for acute pulmonary embolism: A systematic review and meta-analysis. Eur Heart J. Mar 7 2015;36(10):605–614. doi:https://doi.org/10.1093/eurheartj/ehu218

Martin

Molsberry

Cuttica

Desai

Schimmel

Khan

. Time trends in pulmonary embolism mortality rates in the United States, 1999 to 2018. J Am Heart Assoc. Sep 2020;9(17):e016784. doi:https://doi.org/10.1161/JAHA.120.016784

Hawkins

. Pharmacoeconomics of thrombosis management. Pharmacotherapy. Jul 2004;24(7 Pt 2):95S–99S. doi:https://doi.org/10.1592/phco.24.10.95s.36123

Agnelli

Becattini

. Acute pulmonary embolism. N Engl J Med. Jul 15 2010;363(3):266–274. doi:https://doi.org/10.1056/NEJMra0907731

Ploesteanu

Nechita

Andrucovici

, et al.

Is syncope a predictor of mortality in acute pulmonary embolism?

J Med Life. Jan-Mar. 2019;12(1):15–20. doi:https://doi.org/10.25122/jml-2018-0063

Spyropoulos

Lin

. Direct medical costs of venous thromboembolism and subsequent hospital readmission rates: An administrative claims analysis from 30 managed care organizations. J Manag Care Pharm. Jul-Aug. 2007;13(6):475–486. doi:https://doi.org/10.18553/jmcp.2007.13.6.475

10.

Albertsen

Nielsen

Sogaard

, et al. Risk of recurrent venous thromboembolism: A Danish nationwide cohort study. Am J Med. Sep 2018;131(9):1067–1074 e4. doi:https://doi.org/10.1016/j.amjmed.2018.04.042

11.

Kaymaz

Akbal

Tanboga

, et al. Ultrasound-assisted catheter-directed thrombolysis in high-risk and intermediate-high-risk pulmonary embolism: A meta-analysis. Curr Vasc Pharmacol. Jan 26 2018;16(2):179–189. doi:https://doi.org/10.2174/1570161115666170404122535

12.

Chopard

Albertsen

Ecarnot

, et al. Extended anticoagulation after pulmonary embolism: A multicenter observational cohort analysis. J Am Heart Assoc. Jul 5 2022;11(13):e024425. doi:https://doi.org/10.1161/JAHA.121.024425

13.

Lauber

Limacher

Tritschler

, et al. Predictors and outcomes of recurrent venous thromboembolism in elderly patients. Am J Med. Jun. 2018;131(6):703 e7–703 e16. doi:https://doi.org/10.1016/j.amjmed.2017.12.015

14.

Silverstein

Heit

Mohr

Petterson

O'Fallon

Melton

3rd . Trends in the incidence of deep vein thrombosis and pulmonary embolism: A 25-year population-based study. Arch Intern Med. Mar 23 1998;158(6):585–593. doi:https://doi.org/10.1001/archinte.158.6.585

15.

Brink

Elf

Svensson

Engstrom

Melander

Zoller

. Sex-Specific risk factors for deep venous thrombosis and pulmonary embolism in a population-based historical cohort study of middle-aged and older individuals. J Am Heart Assoc. Mar 7 2023;12(5):e027502. doi:https://doi.org/10.1161/JAHA.122.027502

16.

Shah

Arora

Kumar

, et al. Short-term outcomes of pulmonary embolism: A national perspective. Clin Cardiol. Sep 2018;41(9):1214–1224. doi:https://doi.org/10.1002/clc.23048

17.

Kitamukai

Sakuma

Takahashi

Kagaya

Watanabe

Shirato

. Incidence and characteristics of pulmonary thromboembolism in Japan 2000. Intern Med. Nov 2003;42(11):1090–1094. doi:https://doi.org/10.2169/internalmedicine.42.1090

18.

Jang

Bang

. Incidence of venous thromboembolism in Korea: From the health insurance review and assessment service database. J Thromb Haemost. Jan 2011;9(1):85–91. doi:https://doi.org/10.1111/j.1538-7836.2010.04108.x

19.

Jarman

Mumma

Singh

Nowadly

Maughan

. Crucial considerations: Sex differences in the epidemiology, diagnosis, treatment, and outcomes of acute pulmonary embolism in non-pregnant adult patients. J Am Coll Emerg Physicians Open. Feb. 2021;2(1):e12378. doi:https://doi.org/10.1002/emp2.12378

20.

Yoshikawa

Yamashita

Morimoto

, et al. Sex differences in clinical characteristics and outcomes of patients with venous thromboembolism - from the COMMAND VTE registry. Circ J. Jun. 25 2019;83(7):1581–1589. doi:https://doi.org/10.1253/circj.CJ-19-0229

21.

Keller

Beule

Coldewey

Dippold

Balzer

. Heart rate in pulmonary embolism. Intern Emerg Med. Sep 2015;10(6):663–669. doi:https://doi.org/10.1007/s11739-015-1198-4

22.

Yamashita

Morimoto

Takase

, et al. Impact of heart rate at diagnosis on clinical outcomes in patients with acute pulmonary embolism. Am J Cardiol. Jan 15 2023;2023(187):38–47. doi:https://doi.org/10.1016/j.amjcard.2022.10.042

23.

Jaureguizar

Jimenez

Bikdeli

, et al. Heart rate and mortality in patients with acute symptomatic pulmonary embolism. Chest. Feb. 2022;161(2):524–534. doi:https://doi.org/10.1016/j.chest.2021.08.059

24.

Konstantinides

Meyer

Becattini

, et al. ESC Guidelines for the diagnosis and management of acute pulmonary embolism developed in collaboration with the European Respiratory Society (ERS). Eur Heart J. Jan 21 2020;41(4):543–603. doi:https://doi.org/10.1093/eurheartj/ehz405

25.

Quinn

Thompson

Terrin

, et al. A prospective investigation of pulmonary embolism in women and men. JAMA. Oct 7 1992;268(13):1689–1696. doi:https://doi.org/10.1001/jama.1992.03490130077034

26.

Jara-Palomares

Alfonso

Maestre

, et al. Comparison of seven prognostic tools to identify low-risk pulmonary embolism in patients aged<50 years. Sci Rep. Dec 27 2019;9(1):20064.

27.

Anniccherico-Sanchez

Alonso-Martinez

Urbieta-Echezarreta

Villar-Garcia

Rojo-Alvaro

. Factors associated with pulmonary embolism recurrence and the benefits of long-term anticoagulant therapy. Cardiovasc Hematol Disord Drug Targets. 2017;17(3):205–211. doi:https://doi.org/10.2174/1871529X17666170918143459

28.

Qian

Huang

Yang

Zhang

. Heart-type fatty acid binding protein in the assessment of acute pulmonary embolism. Am J Med Sci. Dec 2016;352(6):557–562. doi:https://doi.org/10.1016/j.amjms.2016.08.018

29.

Bledsoe

Woller

Stevens

, et al. Management of low-risk pulmonary embolism patients without hospitalization: The low-risk pulmonary embolism prospective management study. Chest. Aug 2018;154(2):249–256. doi:https://doi.org/10.1016/j.chest.2018.01.035

30.

Olie

Zhu

Martinez

Scarabin

Emmerich

. Sex-specific risk factors for recurrent venous thromboembolism. Thromb Res. Jul 2012;130(1):16–20. doi:https://doi.org/10.1016/j.thromres.2011.10.026

31.

Knowlton

Lee

. Estrogen and the cardiovascular system. Pharmacol Ther. Jul 2012;135(1):54–70. doi:https://doi.org/10.1016/j.pharmthera.2012.03.007

32.

Anderson

Jr ., Spencer

. Risk factors for venous thromboembolism. Circulation. Jun 17 2003;107(23 Suppl 1):I9–16. doi:https://doi.org/10.1161/01.CIR.0000078469.07362.E6

33.

de Bastos

Stegeman

Rosendaal

, et al. Combined oral contraceptives: Venous thrombosis. Cochrane Database Syst Rev. Mar 3 2014;2014(3):CD010813. doi:https://doi.org/10.1002/14651858.CD010813.pub2

34.

Liu

Xiao

Liu

, et al. A new nomogram prediction model for pulmonary embolism in older hospitalized patients. Heliyon. Feb 15 2024;10(3):e25317. doi:https://doi.org/10.1016/j.heliyon.2024.e25317

35.

Yadollahi

Hosseinalipour

Karajizadeh

, et al. Investigating the prevalence of and predictive and risk factors for pulmonary embolism in patients with COVID-19 in Nemazee Teaching Hospital. Blood Res. Sep 30 2023;58(3):127–132. doi:https://doi.org/10.5045/br.2023.2023076

36.

Aslan

Meral

Akgun

, et al. Liver dysfunction in patients with acute pulmonary embolism. Hepatol Res. Mar. 2007;37(3):205–213. doi:https://doi.org/10.1111/j.1872-034X.2007.00014.x

37.

Duman

Ozyurt

Erdogan

Kara

Durakoglugil

. The role of serum bilirubin levels in determining venous thromboembolism. J Vasc Surg Venous Lymphat Disord. Sep 2019;7(5):635–639. doi:https://doi.org/10.1016/j.jvsv.2019.02.002

38.

Cervellin

Buonocore

Sanchis-Gomar

Lippi

. Low serum bilirubin values are associated with pulmonary embolism in a case-control study. Clin Chem Lab Med. Aug 1 2016;54(8):e229–e230. doi:https://doi.org/10.1515/cclm-2015-1156

39.

Horsfall

Rait

Walters

, et al. Serum bilirubin and risk of respiratory disease and death. JAMA. Feb 16 2011;305(7):691–697. doi:https://doi.org/10.1001/jama.2011.124

40.

Horsfall

Nazareth

Petersen

. Cardiovascular events as a function of serum bilirubin levels in a large, statin-treated cohort. Circulation. Nov 27 2012;126(22):2556–2564. doi:https://doi.org/10.1161/CIRCULATIONAHA.112.114066

41.

Suh

Cho

Park

Kim

Cho

Lee

. Relationship between serum bilirubin levels and cardiovascular disease. PLoS One. 2018;13(2):e0193041. doi:https://doi.org/10.1371/journal.pone.0193041

42.

Sedlak

Saleh

Higginson

Paul

Juluri

Snyder

. Bilirubin and glutathione have complementary antioxidant and cytoprotective roles. Proc Natl Acad Sci U S A. Mar 31 2009;106(13):5171–5176. doi:https://doi.org/10.1073/pnas.0813132106

43.

Stocker

Yamamoto

McDonagh

Glazer

Ames

. Bilirubin is an antioxidant of possible physiological importance. Science. Feb 27 1987;235(4792):1043–1046. doi:https://doi.org/10.1126/science.3029864

44.

McDonagh

. The biliverdin-bilirubin antioxidant cycle of cellular protection: Missing a wheel? Free Radic Biol Med. Sep 1 2010;49(5):814–820. doi:https://doi.org/10.1016/j.freeradbiomed.2010.06.001

45.

Trujillo-Santos

Di Micco

Iannuzzo

, et al. Elevated white blood cell count and outcome in cancer patients with venous thromboembolism. Findings from the RIETE Registry. Thromb Haemost. Nov 2008;100(5):905–911. doi:https://doi.org/10.1160/TH08-05-0339

46.

Posadas-Martinez

Torres Gomez

Mezzarobba

, et al. Predictors of recurrence of venous thromboembolic disease after discontinuing of anticoagulation: A prospective cohort study. Medwave. Dec 28 2021;21(11):e8504. Predictores de recurrencia de enfermedad tromboembolica venosa post suspension de anticoagulacion: estudio de cohorte prospectiva. doi:https://doi.org/10.5867/medwave.2021.11.002068

47.

Swystun

Liaw

. The role of leukocytes in thrombosis. Blood. Aug 11 2016;128(6):753–762. doi:https://doi.org/10.1182/blood-2016-05-718114

48.

Henke

Nicklas

Obi

. Immune cell-mediated venous thrombus resolution. Res Pract Thromb Haemost. Nov 2023;7(8):102268. doi:https://doi.org/10.1016/j.rpth.2023.102268

49.

Mukhopadhyay

Johnson

Duru

, et al. Fibrinolysis and inflammation in venous thrombus resolution. Front Immunol. 2019;2019(10):1348. doi:https://doi.org/10.3389/fimmu.2019.01348

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB