Voice features and deep learning models for identifying acute decompensated heart failure

Abstract

Background

Acute decompensated heart failure (ADHF) is a state of systemic congestion which frequently requires hospitalization. Early and noninvasive detection of decompensation remains challenging. This study evaluated whether voice characteristics differ between patients hospitalized with ADHF and after recovery to examine the potential of voice as a non-invasive biomarker for ADHF.

Methods

Patients requiring hospital admission for ADHF were prospectively enrolled. Patients with respiratory infection, sepsis, lung or vocal cord disease, acute coronary syndrome, or serum creatinine>3.0mg/dL were excluded. Voice was recorded at two time points: at admission for ADHF and at discharge after recovery. Low-level audio features were extracted, and Mel-spectrograms derived from the voice waveforms were used as the input feature for deep learning-based classification models. Model performance was primarily evaluated at the patient-state level using area under the receiver operating characteristic curve (AUC) with 95% confidence intervals estimated by patient-level bootstrap resampling.

Results

A total of 100 patients were included (mean age 70.5 years; 47% women) and randomly divided into a training group (N=88) and a test group (N=12). In the analysis with low-level audio features, harmonics-to-noise ratio and Shimmer demonstrated discriminative potential between decompensated and recovered states. In patient-state-level analysis, the best-performing model, ConvNeXt-large, achieved a mean AUC of 0.83 (95% CI 0.70-0.95). Segment-level analysis demonstrated higher discriminative performance (AUC up to 0.87), suggesting the presence of discriminative acoustic features.

Conclusions

Voice characteristics differed between admission and discharge states in patients hospitalized for ADHF, and voice analysis using deep learning models was able to distinguish acute decompensated and recovered states in patients with ADHF. Our results suggest that voice analysis may provide a novel and non-invasive approach for detection of ADHF.

Keywords

heart failure acute decompensation voice analysis biomarker deep learning

Introduction

Heart failure (HF) is a clinical syndrome characterized by impaired heart function leading to inadequate systemic perfusion and congestion.¹ More than 64 million individuals worldwide are affected by HF,² and its prevalence exceeds 15% among adults aged 80 years or older.^3,4 HF remains the most common cause of hospitalization in adults over 65 years of age⁵ and it remains as one of the most costly and disabling diseases causing frequent visits to the emergency room.⁶ Acute decompensated heart failure (ADHF), defined by pulmonary and systemic congestion resulting from elevated cardiac filling pressures,⁷ requires timely management. Since HF hospitalization is closely associated with an increase of readmission, in-hospital and post-discharge mortalities in short- and long-term,^8,9 early identification of acute decompensation is crucial.

Common strategies to monitor pulmonary and systemic congestion for patients with HF include serum biomarkers and imaging modalities. Measurement of serum biomarkers such as N-terminal pro-B-type natriuretic peptide (NT-proBNP) or B-type natriuretic peptide (BNP) is recommended by current guidelines.^1,10,11 Soluble suppression of tumorigenicity 2 (sST2), a novel biomarker reflecting inflammation and fibrosis, have demonstrated prognostic value in HF.¹² Imaging techniques such as chest x-ray and computed tomography (CT) provide valuable information on pulmonary congestion, pleural effusion, and cardiomegaly. However, both biomarkers and imaging techniques are difficult to repeat due to cost, discomfort associated with phlebotomy, and the inconvenience of hospital visits to undergo these tests. Imaging techniques such as chest CT could quantify the severity of pulmonary congestion; however, they cannot be repeated frequently due to risks associated with cumulative radiation exposure. Recently, a pulmonary pressure-guided therapy for HF patients was introduced which provided continuous monitoring and early intervention, but it has potential of complications, requires high initial and maintenance costs as well as invasive procedure.¹³ Non-invasive tests such as lung ultrasound or wearable monitoring systems either require hospital visits with trained medical professionals or remain insufficiently validated for routine clinical practice.¹⁴ Therefore, a non-invasive, safe, inexpensive, and easily accessible method to detect congestion and decompensation is urgently needed for patients with HF.

Body fluid overload in ADHF can influence voice characteristics through edema of the lungs, larynx, and vocal cords. Although body weight monitoring is recommended to detect congestion, edema-associated weight gain is only detectable after certain amount of time in disease progression, making it unreliable to be an early predictor of ADHF.¹⁵ Moreover, the amount of pulmonary and systemic congestion to cause a change in body weight may be greater than the amount of congestion to cause changes in voice. The phonation threshold pressure, an indicator of the pulmonary drive for phonation, was increased in healthy adults with systemic dehydration caused by administering a single 60-mg dose of furosemide.¹⁶ This raises the possibility that voice could be a more sensitive and useful marker of ADHF reflecting changes associated with congestion earlier than body weight assessment. Voice is a non-invasive and easy-to-obtain biomarker that can be easily monitored, even in telemedicine settings, to detect congestion in patients with HF; however, relevant studies are scarce. A pilot study of 10 patients with ADHF suggested that several voice measures could be early indicators of HF through acoustic speech analysis but limited by its small sample size.¹⁵ An uncontrolled, retrospective and observational study investigated the association between voice signal characteristics and adverse clinical outcomes in patients with HF but it lacked standardized voice analysis protocols.¹⁷ A small, prospective study of 40 ADHF patients found significant differences between baseline and discharge recordings, but was limited by not using clinical-grade detectors.¹⁸

In this prospective study, we evaluated whether voice characteristics differ between the decompensated and recovered states in patients hospitalized for ADHF. We further examined the ability of voice features and deep learning-based models to distinguish between these states to assess the potential of voice as a useful and non-invasive biomarker for detecting ADHF.

Methods

Study design and population

This prospective observational study was conducted at Korea University Guro Hospital, Seoul, Korea. Patients requiring admission for ADHF were eligible for enrollment. Patients were excluded if they had significant concurrent respiratory infection or sepsis; significant lung or vocal cord diseases which could affect voice (including chronic obstructive pulmonary disease or asthma); acute coronary syndrome; inability to speak Korean; or severe renal dysfunction defined as serum creatinine >3.0 mg/dL or receipt of renal replacement therapy. Patients were discharged when their treating physicians determined that recovery from ADHF was sufficient for home discharge, based on an integrated assessment of vital signs, symptoms, echocardiographic findings, laboratory results, and overall clinical status. Patients who left against medical advice or died during hospitalization were dropped out in the final analyses. A total of 112 patients were enrolled between July 1, 2020, and December 31, 2022 (Figure 1). This study was approved by the Institutional Review Board of Korea University Guro Hospital (N: 2020GR0356), and all patients signed a written informed consent. This study was conducted in accordance with the ethical principles of the Declaration of Helsinki.

Figure 1.

Study schema.

Voice acquisition and clinical data

The voice was recorded twice: at hospital admission for ADHF and at discharge after recovery from ADHF. Voice was recorded in a quiet education room in the cardiology ward using a Sony PCM-A10 recorder located approximately 20cm from patient’s mouth at a sampling frequency of 48 kHz and a scale of 24 bits. Patients were asked to phonate five fundamental Korean vowels (“[a], [e], [i], [ɔ], [u]”) for 3 seconds each and then to repeat the sentence “daehan minkook manse” five times. Serum NT-proBNP, sST2, BUN, creatinine, and body weight were measured at both admission and discharge.

Data preparation

A total of 100 patients were eligible for analysis after excluding 12 patients with incomplete voice recordings. The voice recordings were processed using frequency analysis algorithm with Fourier transform and Praat program to report 4 different low-level speech features-fundamental frequency (F0), harmonics-to-noise ratio (HNR), Jitter, and Shimmer. The study population was randomly assigned to two mutually exclusive groups: training (N=88) and test (N=12).

For deep learning analysis, voice recordings obtained at admission and discharge were converted into Mel-spectrograms for model input. Multiple recordings obtained from each patient at each clinical state were retained as repeated observations for model training.

The primary analysis was conducted at the patient-state level, where the unit of analysis was the patient-state (admission or discharge recording from an individual patient). For each patient-state, predicted probabilities obtained from individual recordings were averaged to derive a single patient-state probability for performance evaluation.

In addition, a secondary segment-level analysis was performed to evaluate the model’s ability to detect discriminative acoustic patterns within short voice segments. For this analysis, a sliding data augmentation strategy was applied, in which audio recordings were segmented into 1-second windows with a sliding step of 0.1 seconds across the signal. Each segment was treated as an individual sample, resulting in 9,442 training samples and 663 test samples. The window length and hop length were 1s and 21ms, respectively.

Low-level audio features

Low-level audio features were analyzed to identify voice characteristics associated with decompensated (‘wet’) and recovered (‘dry’) states. F0 is a fundamental frequency, HNR measures the ratio between periodic and non-periodic components of a speech sound. Jitter refers to consistency of period, and Shimmer reflects consistency of amplitude of voice oscillation. Jitter or Shimmer increases when irregularity of period or amplitude of oscillation increases.

Deep learning models

We trained the models to classify whether a certain voice belonged to the ADHF or recovered state. This was treated as a binary classification task. To train the deep learning-based binary classification models, the input features, model structure, and loss function were selected first. For the input feature, a Mel-Spectrogram, a time-frequency domain feature, was extracted from the voice recordings since it has the advantage of representing frequency domain characteristics of vocalizations. The hyperparameters of the Mel-Spectrogram were as follows: N-FFT = 1,024, hop_length = 256, and mel_bin = 128.

For the model structures, two broad types of the deep learning-based classification models are utilized: convolutional neural networks (CNNs) and Transformer (Supplemental Figure 1). CNN-style classification models, including ResNet50,¹⁹ ConvNeXt,²⁰ DenseNet201,²¹ ResNeXt,²² and EfficientNet-b7,²³ refine the local features of the Mel-Spectrogram to find the optimal features for ADHF classification during the gradient descent steps. The typical structure of the CNN-style classification model consists of the first convolution layer, CNN stem, and a classification layer. The first convolution layer initially adjusts the channels of the input features. Second, the stem extracts hidden features using CNN layers. For DenseNet201, the stem contains four dense layers and three transition layers. Other CNN-style classification models have different CNN structures in their stems. The hidden features are converted into ADHF classification results, enabling humans to understand them by the classification layer.

Transformer style classification models, such as ViT²⁴ and PVT,²⁵ find optimal attention maps between cropped patches of the Mel-Spectrogram. Features were optimized using attention maps to focus on the more important input features and to remove meaningless features. The input features were first cropped into patches and projected onto hidden feature tokens. Each hidden feature token represented a truncated patch. Moreover, a classification token was added to the hidden feature tokens to calculate the attention maps. The hidden feature tokens were refined using attention maps in the multi-head attention layers. Finally, the refined hidden features of the classification token were converted into ADHF classification results by the classifying layer.

We trained the deep learning-based classification models with AdamW optimizer²⁶ with a learning rate of 0.00001 and cross-entropy loss. Each model was trained five times with different learning rates ranging from 0.00001 to 0.00005, and performance was calculated with the average, maximum and minimum Area Under the Curve (AUC). All experiments for the deep learning models were performed using the PyTorch library of the Python programming language.

Statistical analysis

Continuous variables are presented as mean ± standard deviation or median (interquartile range), as appropriate. Categorical variables are presented as counts and percentages. Baseline clinical characteristics were compared between the training and test groups using the Wilcoxon rank-sum test. Paired comparisons between admission and discharge states were performed using the Wilcoxon signed-rank test. Linear regression analyses were performed to assess associations between selected serum biomarkers (NT-proBNP and sST2) and low-level audio features.

The performance of deep learning-based classification models for distinguishing acute decompensated heart failure (ADHF) from the recovered state was evaluated using receiver operating characteristic (ROC) curves and the area under the ROC curve (AUC) with 95% confidence intervals. Model performance was evaluated at two levels. The primary analysis was conducted at the patient-state level, where predicted probabilities obtained from individual recordings were averaged to generate a single probability per patient-state, reflecting the clinically relevant unit of inference. Confidence intervals were estimated using bootstrap resampling at the patient level within the independent test cohort. Segment-level analyses were performed to assess the model’s ability to detect discriminative acoustic features and were considered secondary.

All statistical analyses were conducted using Scipy and Matplotlib libraries of Python programming language. A two-sided P value <0.05 was considered statistically significant. This study was designed as a prospective exploratory proof-of-concept investigation, and no formal a priori sample size calculation was performed.

Results

Baseline characteristics

Baseline characteristics of the training group (N=88) and test group (N=12) are presented in Table 1. In the overall cohort, 47% of patients were women, and the mean age was 70.5 years. Patients in the training group were slightly older (70.7 ± 14.4 years vs. 69.4 ± 14.6 years; P = 0.040) and had higher initial serum BUN levels (24.1 ± 12.2 mg/dL vs. 16.5 ± 5.3 mg/dL; P = 0.024) compared to the test group. However, there were no significant differences between the two groups with respect to sex, left ventricular ejection fraction (LVEF), proportion of patients according to the HF subtypes based on EF, baseline SBP, history of diabetes mellitus or hypertension, serum Creatinine, sST2, NT-proBNP and body weight at admission and at discharge. The mean change in body weight between admission and discharge was 3.37 ± 3.55 kg in the training group and 4.18 ± 4.62 kg in the test group (P = 0.895). The change in serum NT-proBNP was 3931.2 ± 4671.7 pg/mL and 1619.4 ± 2058.8 pg/mL, respectively (P = 0.270).

Table 1.

Baseline characteristics of patients with ADHF.

	Training group (n=88)	Test group (n=12)	p-value
Age, years	70.7±14.4	69.4±14.6	0.040
Female, N (%)	38 (43.2)	9 (75.0)	0.783
HFpEF, N (%)	29 (33.0)	4 (33.3)	0.942
HFmrEF, N (%)	7 (8.0)	3 (25.0)	0.071
HFrEF, N (%)	52 (59.1)	5 (41.7)	0.239
LVEF, (%)	39.4±14.8	43.4±15.1	0.393
Body weight, kg
At admission	65.6±17.0	60.0±10.4	0.197
At discharge	62.2±15.8	55.8±10.3	0.133
Difference	3.37±3.55	4.18±4.62	0.895
NT-proBNP, pg/mL
At admission	6654.2 [5089.4-8219.0]	6593.7 [581.9-12605.4]	0.991
At discharge	2558.2 [1706.6-3409.8]	4974.3 [-1260.8-11209.3]	0.960
Difference	3931.17±4671.66	1619.42±2058.82	0.270
BUN, mg/dL
At admission	24.1±12.2	16.5±5.3	0.024
At discharge	24.3±11.1	20.9±4.4	0.560
Difference	-0.22±8.76	-4.38±7.76	0.158
Creatinine, mg/dL
At admission	1.13±0.70	0.95±0.36	0.467
At discharge	1.12±0.74	0.97±0.39	0.618
Difference	0.01±0.20	-0.02±0.19	0.671
sST2, ng/mL
At admission	56.2±53.6	44.8±19.9	0.806
At discharge	30.1±37.7	32.9±35.1	0.985
Difference	26.9±56.9	11.9±39.3	0.925
History of diabetes mellitus, N (%)	38 (43.7)	5 (41.7)	0.901
History of hypertension, N (%)	54 (62.1)	8 (66.7)	0.764
SBP, mmHg	130.9±21.5	121.2±21.7	0.158

Values are expressed as mean±SD or median with interquartile ranges, or number (%), as appropriate. Comparisons between the training and test groups were performed using the Wilcoxon rank-sum test. Paired comparisons between admission and discharge values were performed using Wilcoxon signed-rank test.

LVEF = left ventricular ejection fraction; NT-proBNP = N-terminal pro-B-type natriuretic peptide; sST2 = soluble suppression of tumorigenicity 2; SBP = systolic blood pressure; BUN = blood urea nitrogen.

Low-level audio feature analysis

We analyzed voice characteristics for all patients with low-level audio features. Analysis of low-level audio features for “daehan minkook manse” is presented in Supplemental Figure 2. Results for other pronunciations are shown in Supplemental Figure 3. HNR and Shimmer differed significantly between ‘wet’ and ‘dry’ states across all pronunciations (P <0.05 for all comparisons). Jitter showed a significant difference between ‘wet’ and ‘dry’ states only for “[u]”.

We also conducted a linear regression analysis with absolute values of ‘wet’ and ‘dry’ states by inputting audio features and classic serum biomarkers of HF (NT-proBNP and sST2) for “daehan minkook manse” (Supplemental Figure 4). No clear linear associations were observed between low-level audio features and serum biomarkers. Linear regression plots of other pronunciations are presented in Supplemental Figure 5.

The distributions of selected low-level audio features and deep learning-derived scores are illustrated in Figure 2. Paired comparisons between admission and discharge states are shown for each patient. While HNR and Shimmer demonstrated differences between states (Figure 2(a) and (b)), the separation between admission and discharge states was more pronounced for the deep learning-derived score (Figure 2(c)), indicating a clearer distinction between clinical states compared with individual low-level features.

Figure 2.

Distribution of low-level audio features and deep learning-derived scores according to patient state.

Deep learning-based classification

We further analyzed voice with Mel-Spectrogram and deep learning models. Model performance was evaluated at both the patient-state level and the segment level. At the patient-state level, ConvNeXt-large demonstrated the highest overall performance, achieving an AUC of 0.83 (95% CI, 0.70-0.95), whereas other models showed AUC values ranging from 0.60 to 0.81 (Figure 3(a) and Table 2). Accuracy ranged from 55.8% to 71.7% across models.

Figure 3.

Performance of deep learning models for distinguishing acute decompensated heart failure from recovered state.

Table 2.

Performance of deep learning models for distinguishing acute decompensated heart failure from recovered state.

Model (patient-state level)	Daehan minkook manse +[a], [e], [i], [ɔ], [u]		+ additional HF features
Model (patient-state level)	AUC (95% CI)	Accuracy (%)	AUC (95% CI)	Accuracy (%)
ConvNeXt-large	0.83 (0.70 – 0.95)	63.3	0.73 (0.57 – 0.92)	82.5
DenseNet201	0.68 (0.48 – 0.85)	58.3	0.75 (0.59 – 0.93)	77.5
EfficientNet-b7	0.60 (0.40 – 0.79)	55.8	0.73 (0.58 – 0.92)	79.2
ResNet50	0.73 (0.51 – 0.90)	60.0	0.75 (0.59 – 0.93)	75.8
ResNext-101	0.72 (0.48 – 0.93)	68.3	0.73 (0.56 – 0.92)	79.2
PVT	0.81 (0.62 – 0.95)	71.7	0.74 (0.58 – 0.93)	80.0
Swin Transformer	0.77 (0.59 – 0.92)	65.0	0.73 (0.57 – 0.92)	79.2
ViT-16-large	0.78 (0.62 – 0.93)	65.0	0.73 (0.56 – 0.92)	80.8
ViT-32-base	0.81 (0.64 – 0.95)	66.7	0.74 (0.58 – 0.93)	79.2

Model (segment-level)	Daehan minkook manse +[a], [e], [i], [ɔ], [u]		+ additional HF features
Model (segment-level)	AUC (95% CI)	Accuracy (%)	AUC (95% CI)	Accuracy (%)
ConvNeXt-large	0.77 (0.74 – 0.80)	69.2	0.86 (0.83 – 0.89)	81.1
DenseNet201	0.87 (0.84 – 0.90)	84.4	0.89 (0.87 – 0.91)	86.5
EfficientNet-b7	0.71 (0.68 – 0.74)	73.4	0.80 (0.77 – 0.83)	77.8
ResNet50	0.76 (0.73 – 0.79)	71.7	0.75 (0.72 – 0.78)	68.3
ResNext-101	0.75 (0.72 – 0.78)	67.7	0.78 (0.75 – 0.81)	70.8
PVT	0.78 (0.75 – 0.81)	72.9	0.82 (0.79 – 0.85)	75.5
Swin Transformer	0.85 (0.82 – 0.88)	78.3	0.81 (0.78 – 0.84)	79.8
ViT-16-large	0.84 (0.81 – 0.87)	77.5	0.87 (0.84 – 0.90)	77.7
ViT-32-base	0.83 (0.80 – 0.86)	76.5	0.85 (0.82 – 0.88)	76.9

AUC values are presented with 95% confidence intervals. Accuracy is presented as percentage. AUC = area under receiver operating characteristic curve; ADHF = acute decompensated heart failure; NT-proBNP = N-terminal pro-B-type natriuretic peptide; sST2 = soluble suppression of tumorigenicity 2.

Additional HF features are NT-proBNP, sST2, and body weight.

When evaluated at the segment level, models demonstrated higher apparent discriminative performance compared with patient-state-level analysis (Figure 3(b) and Table 2). DenseNet201 achieved the highest segment-level performance, with an AUC of 0.87 (95% CI, 0.84-0.90), while other several other models also showed good discriminative performance.

Performance was higher at the segment level than at the patient-state level, reflecting the non-independence of multiple recordings obtained from the same individual. The patient-state-level analysis therefore provides a more conservative and clinically relevant estimate of model performance. The incorporation of additional clinical HF features (NT-proBNP, sST2, and body weight) did not consistently improve AUC across models. At the patient-state level, modest increases in accuracy were observed in some models, whereas improvements were less consistent at the segment level (Table 2).

Model performance by pronunciation

Model performance for each pronunciation is presented in Table 3. At the patient-state level, classification performance varied across tasks. The sentence task (“daehan minkook manse”) showed relatively consistent discrimination across models, whereas performance for vowel phonations demonstrated greater variability. Among individual vowel sounds, “u” demonstrated the highest performance, while “a” showed comparatively lower performance. These patterns were consistent across different models. At the segment level, overall performance was higher, but heterogeneity across pronunciation tasks remained evident.

Table 3.

Performance of deep learning models for distinguishing acute decompensated heart failure from recovered state according to pronunciation.

Model (patient-state level)	Daehan minkook manse		[a]		[e]		[i]		[ɔ]		[u]
Model (patient-state level)	AUC (95% CI)	Accuracy (%)	AUC (95% CI)	Accuracy (%)	AUC (95% CI)	Accuracy (%)	AUC (95% CI)	Accuracy (%)	AUC (95% CI)	Accuracy (%)	AUC (95% CI)	Accuracy (%)
ConvNeXt-large	0.78 (0.62 – 0.92)	66.7	0.64 (0.37 – 0.89)	62.5	0.70 (0.51 – 0.87)	62.5	0.62 (0.40 – 0.83)	62.5	0.72 (0.52 – 0.91)	68.7	0.88 (0.70 – 1.00)	81.7
DenseNet201	0.66 (0.46 – 0.83)	64.2	0.43 (0.21 – 0.65)	49.2	0.57 (0.35 – 0.78)	58.3	0.50 (0.26 – 0.74)	52.5	0.65 (0.41 – 0.86)	64.3	0.64 (0.42 – 0.82)	64.2
EfficientNet-b7	0.58 (0.39 – 0.77)	54.2	0.47 (0.27 – 0.66)	48.3	0.57 (0.35 – 0.78)	55.0	0.59 (0.35 – 0.83)	52.5	0.57 (0.32 – 0.79)	57.4	0.58 (0.36 – 0.78)	51.7
ResNet50	0.64 (0.40 – 0.86)	60.0	0.46 (0.23 – 0.70)	44.2	0.73 (0.52 – 0.92)	70.8	0.52 (0.27 – 0.76)	55.8	0.70 (0.48 – 0.89)	68.7	0.72 (0.49 – 0.91)	69.2
ResNext-101	0.69 (0.42 – 0.92)	65.8	0.43 (0.21 – 0.68)	45.8	0.62 (0.40 – 0.83)	59.2	0.59 (0.34 – 0.82)	60.0	0.62 (0.40 – 0.83)	59.1	0.64 (0.41 – 0.85)	63.3
PVT	0.75 (0.56 – 0.92)	66.7	0.70 (0.48 – 0.89)	67.5	0.59 (0.36 – 0.82)	60.8	0.71 (0.49 – 0.91)	67.5	0.71 (0.49 – 0.92)	71.3	0.66 (0.40 – 0.88)	65.8
Swin Transformer	0.77 (0.58 – 0.93)	60.8	0.56 (0.31 – 0.81)	60.0	0.60 (0.39 – 0.80)	58.3	0.70 (0.47 – 0.88)	66.7	0.69 (0.48 – 0.89)	72.6	0.77 (0.60 – 0.92)	66.7
ViT-16-large	0.77 (0.58 – 0.92)	67.5	0.58 (0.32 – 0.82)	55.0	0.61 (0.41 – 0.82)	61.7	0.67 (0.41 – 0.90)	71.7	0.59 (0.36 – 0.82)	58.3	0.73 (0.50 – 0.93)	70.8
ViT-32-base	0.74 (0.55 – 0.91)	65.0	0.55 (0.32 – 0.77)	53.3	0.71 (0.52 – 0.89)	67.5	0.69 (0.44 – 0.90)	68.3	0.72 (0.51 – 0.90)	65.2	0.84 (0.66 – 0.96)	77.5

Model (segment-level)	Daehan minkook manse		[a]		[e]		[i]		[ɔ]		[u]
Model (segment-level)	AUC (95% CI)	Accuracy (%)	AUC (95% CI)	Accuracy (%)	AUC (95% CI)	Accuracy (%)	AUC (95% CI)	Accuracy (%)	AUC (95% CI)	Accuracy (%)	AUC (95% CI)	Accuracy (%)
ConvNeXt-large	0.59 (0.55 – 0.63)	58.4	0.60 (0.56 – 0.64)	65.7	0.77 (0.74 – 0.80)	78.9	0.49 (0.45 – 0.53)	61.9	0.84 (0.81 – 0.87)	83.1	0.89 (0.87 – 0.91)	75.7
DenseNet201	0.90 (0.88 - 0.92)	88.7	0.79 (0.76 – 0.82)	80.8	0.76 (0.73 – 0.79)	73.0	0.65 (0.61 – 0.69)	69.4	0.73 (0.70 – 0.76)	76.3	0.84 (0.81 – 0.87)	82.0
EfficientNet-b7	0.71 (0.68 – 0.74)	77.2	0.57 (0.53 – 0.61)	69.5	0.66 (0.62 – 0.70)	70.4	0.62 (0.58 – 0.66)	73.6	0.79 (0.76 – 0.82)	82.9	0.79 (0.76 – 0.82)	82.3
ResNet50	0.84 (0.81 – 0.87)	79.1	0.76 (0.73 – 0.79)	77.8	0.82 (0.79 – 0.85)	79.4	0.64 (0.60 – 0.68)	72.3	0.86 (0.83 – 0.89)	87.6	0.86 (0.83 – 0.89)	87.4
ResNext-101	0.74 (0.71 – 0.77)	75.4	0.70 (0.67 – 0.73)	76.8	0.77 (0.74 – 0.80)	76.3	0.72 (0.69 – 0.75)	65.0	0.90 (0.88 – 0.92)	89.7	0.86 (0.83 – 0.89)	82.8
PVT	0.67 (0.63 – 0.71)	62.5	0.66 (0.62 – 0.70)	66.8	0.73 (0.70 – 0.76)	74.9	0.69 (0.65 – 0.73)	68.5	0.75 (0.72 – 0.78)	77.0	0.86 (0.83 – 0.89)	78.7
Swin Transformer	0.81 (0.78 – 0.84)	69.3	0.76 (0.73 – 0.79)	74.7	0.84 (0.81 – 0.87)	84.7	0.87 (0.84 – 0.90)	86.0	0.83 (0.80 – 0.86)	78.3	0.90 (0.88 – 0.92)	87.6
ViT-16-large	0.76 (0.73 – 0.79)	73.2	0.71 (0.68 – 0.74)	75.3	0.80 (0.77 – 0.83)	78.3	0.77 (0.74 – 0.80)	81.3	0.84 (0.81 – 0.87)	85.8	0.73 (0.70 – 0.76)	78.2
ViT-32-base	0.77 (0.74 – 0.80)	71.5	0.69 (0.65 – 0.73)	64.8	0.84 (0.81 – 0.87)	84.5	0.84 (0.81 – 0.87)	83.1	0.83 (0.80 – 0.86)	78.7	0.76 (0.73 – 0.79)	72.9

AUC values are presented with 95% confidence intervals. Accuracy is expressed as percentage.

AUC = area under receiver operating characteristic curve; ADHF = acute decompensated heart failure.

Discussion

In this proof-of-concept study, we evaluated whether voice characteristics could serve as a non-invasive biomarker for detecting ADHF. By analyzing voice recordings obtained at hospital admission and recovered states in patients with ADHF, we demonstrated that specific low-level audio features differed according to volume status and that deep learning-based models were able to distinguish decompensated from recovered states at the patient-state level. These findings suggest that changes in voice during HF exacerbation may reflect volume states and support the potential role of voice as a novel, patient-centered and noninvasive approach to monitor HF status.

Although the long-term aim of our study is to enable telemedicine-based monitoring for early detection of ADHF while minimizing patient burden and hospital visits, the present study focused on voice recordings obtained at hospital admission and at discharge to capture distinct states of decompensation and recovery. To identify specific characteristics that reflect HF exacerbation and to discover deep learning models that can be adapted for telemedicine HF monitoring, patients requiring hospital admission could be an ideal model for testing because voice characteristics may reflect extremes of HF presentation. Voice is produced as a coordinated interaction of the lungs, larynx, and articulators.²⁷ Patients with ADHF have pulmonary congestion and systemic edema, which can affect the conditions of the articulatory organs. Swelling of articulatory organs, including the vocal cords, tongue, lips, and cheeks, can cause changes in voice and speech production. In line with this, HNR and Shimmer showed significant differences between admission (wet) and discharge (dry). Voice recorded at discharge showed higher HNR and lower Shimmer compared to the admission state, indicating clearer and more stable phonation.

While low-level audio features alone showed limited classification performance, deep learning-based models using Mel-Spectrogram demonstrated meaningful discrimination between decompensated and recovered states in the patient-state-level analysis. Compared with segment-level analyses, performance estimates were more conservative, with the best-performing model (ConvNeXt-large) achieving a mean AUC of 0.83, although confidence intervals remained wide due to the small independent test cohort. Segment-level analysis was not intended to estimate clinical performance, but rather to evaluate whether discriminative acoustic patterns exist within short voice segments. In contrast, patient-state-level analysis reflects the clinically relevant unit of inference, where multiple recordings from the same patient are aggregated. The discrepancy between segment-level and patient-state-level performance highlights the distinction between signal detectability and clinical applicability. The addition of established HF markers, including NT-proBNP, sST2, and body weight, did not consistently improve model discrimination. This suggests that voice-based features may capture information related to congestion that is not fully reflected by conventional biomarkers, although this observation requires validation in larger cohorts. Our findings do not suggest that voice analysis should replace established biomarkers. Rather, voice could serve as a complementary, relatively continuous, and non-invasive monitoring tool, particularly in real-world settings where frequent hospital visits are impractical. Such monitoring may be valuable for detecting gradual physiological changes preceding overt decompensation and prompting timely clinical evaluation when indicated; however, further studies in longitudinal and ambulatory settings are needed to evaluate the potential of voice analysis to detect early or subclinical changes prior to overt clinical deterioration. We also observed heterogeneity in model performance across different pronunciation tasks, suggesting that pronunciation-specific modeling strategies may be required to optimize classification performance. Notably, consistent patterns across both patient-state-level and segment-level analyses suggest that certain phonation tasks may offer more robust discrimination, supporting the importance of task selection in future clinical applications.

A key strength of this study lies in its within-patient paired design, which could reduce inter-individual variability in voice characteristics and allow each patient to serve as their own control. This design choice is particularly relevant in voice-based analyses. Although the overall cohort size was modest, the paired structure and strict separation of patients between training and test datasets were intended to mitigate overfitting and to provide an initial assessment of generalizability.

This study has several limitations. First, only patients who could speak Korean were included in this study. Although we attempted to minimize the effect of pronunciation differences by including both sentence tasks and fundamental vowel phonations, it may be difficult to apply our results to other patient groups who speak different languages. However, analyses of fundamental vowel sounds - shared across many languages – showed consistent classification performance, suggesting applicability potential beyond Korean speakers. Second, because the voice recordings of the specific phrases had to be collected from patients, patients in intensive care units or the emergency room were not able to participate in the study because of intubation, or substantial noise from surrounding machines such as ventilators, continuous renal replacement therapy machines, cardiac monitors, and defibrillators. However, in real-world settings, voice recordings may be obtained under more diverse acoustic conditions and using different recording devices, which could influence signal quality. Further studies in more diverse settings are therefore warranted. Third, exclusion of patients with respiratory infections, lung diseases, and renal dysfunction could limit the complete reflection of the real-world HF patients. Fourth, the study population was restricted to patients hospitalized for ADHF, and chronic stable HF patients were not evaluated. In addition, potential influences of demographic factors such as age and sex on voice characteristics were not specifically examined and warrant further investigation. Fifth, the independent test cohort was relatively small, resulting in wide confidence intervals for performance estimates. Although patient-level evaluation and strict separation between training and test datasets were applied, the small sample size limits the precision of performance estimates and requires validation in larger cohorts. Finally, since the model validation was performed using an internal dataset, further studies may be needed to enhance generalizability.

Conclusions

In this study, specific voice features, including HNR and Shimmer, differed between admission and discharge in patients hospitalized with ADHF. Deep learning models demonstrated the ability to distinguish decompensated from recovered states at the patient-state level. These findings support the feasibility of voice analysis as a complementary, noninvasive approach for monitoring HF status.

Supplemental material

Supplemental material - Voice features and deep learning models for identifying acute decompensated heart failure

Supplemental material for Voice features and deep learning models for identifying acute decompensated heart failure by Jieun Lee, Gwantae Kim, Insung Ham, Kyungdeuk Ko, Soohyung Park, Dong Oh Kang, Jah Yeon Choi, Eun Jin Park, Sunki Lee, Seung Young Roh, Dae-In Lee, Jin Oh Na, Cheol Ung Choi, Jin Won Kim, Seung-Woon Rha, Chang Gyu Park, Jae-Gu Cho, Eugene Yang, Eung Ju Kim, Hanseok Ko by Digital health.

Footnotes

Acknowledgments

The authors thank all patients with ADHF for participating in this study.

ORCID iD

Eung Ju Kim

Ethical considerations

This study was approved by the Institutional Review Board of Korea University Guro Hospital (N: 2020GR0356), and all patients signed a written informed consent. This study was conducted in accordance with the ethical principles of the Declaration of Helsinki.

Author contributions

JE. Lee: Writing – original draft, Writing – review and editing, Investigation, Methodology, Visualization, Conceptualization, Data curation, Resources, Validation. GT. Kim: Writing – original draft, Writing – review and editing, Investigation, Methodology, Visualization, Conceptualization, Resources, Formal analysis, Validation. EJ. Kim: Writing – review and editing, Investigation, Conceptualization, Data curation, Resources, Validation, Funding acquisition, Supervision, Project administration. HS. Kim: Investigation, Conceptualization, Resources, Validation, Supervision, Project administration. IS. Ham: Methodology, Investigation. KD. Ko: Methodology, Investigation, SH. Park: Data curation. DO. Kang: Data curation. JY. Choi: Data curation. EJ. Park: Data curation. SK. Lee: Data curation. SY. Roh: Data curation. DI. Lee: Data curation. JO. Na: Data curation. CU. Choi: Data curation. JW. Kim: Data curation. SW. Rha: Data curation. CG. Park: Data curation. JG. Cho: Supervision. E. Yang: Supervision, Writing – review and editing, All authors read and approved the final manuscript for publication.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was selected as an interdepartmental collaborative study between the medical and engineering schools at Korea University. This selected interdepartmental study was partially funded by a research grant from Korea University (K2011361).

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship.

Data Availability Statement

The dataset generated and/or analyzed during the current study are available from the corresponding author on a reasonable request.*

Trial registration

The trial was registered at the Korean Clinical Trials Registry (The Clinical Research Information Service established at the Korea Disease Control and Prevention Agency with support from the Ministry of Health and Welfare) – number KCT0007646. (https://cris.nih.go.kr/cris/search/detailSearch.do?seq=25650&search_page=L).

Supplemental material

Supplemental material for this article is available online.

References

Heidenreich

Bozkurt

Aguilar

, et al. 2022 AHA/ACC/HFSA Guideline for the Management of Heart Failure: A Report of the American College of Cardiology/American Heart Association Joint Committee on Clinical Practice Guidelines. Circulation 2022; 145: e895–e1032. https://doi.org/10.1161/cir.0000000000001063

Savarese

Becher

Lund

, et al. Global burden of heart failure: a comprehensive and updated review of epidemiology. Cardiovasc Res 2023; 118: 3272–3287. https://doi.org/10.1093/cvr/cvac013

Ceia

Fonseca

Mota

, et al. Prevalence of chronic heart failure in Southwestern Europe: the EPICA study. Eur J Heart Fail 2002; 4: 531–539. https://doi.org/10.1016/s1388-9842(02)00034-x

Anguita Sánchez

Crespo Leiro

de Teresa Galván

, et al. Prevalence of heart failure in the Spanish general population aged over 45 years. The PRICE Study. Rev Esp Cardiol 2008; 61: 1041–1049. https://doi.org/10.1016/s1885-5857(09)60007-4

Nieminen

Harjola

. Definition and epidemiology of acute heart failure syndromes. Am J Cardiol 2005; 96: 5g–10g. https://doi.org/10.1016/j.amjcard.2005.07.015

McMurray

Pfeffer

. Heart failure. Lancet 2005; 365: 1877–1889. https://doi.org/10.1016/s0140-6736(05)66621-4

Gheorghiade

Zannad

Sopko

, et al. Acute heart failure syndromes: current state and framework for future research. Circulation 2005; 112: 3958–3968. https://doi.org/10.1161/circulationaha.105.590091

Ahmed

Allman

Fonarow

, et al. Incident heart failure hospitalization and subsequent mortality in chronic heart failure: a propensity-matched study. J Card Fail 2008; 14: 211–218. https://doi.org/10.1016/j.cardfail.2007.12.001

Solomon

Dobson

Pocock

, et al. Influence of nonfatal hospitalization for heart failure on subsequent mortality in patients with chronic heart failure. Circulation 2007; 116: 1482–1487. https://doi.org/10.1161/circulationaha.107.696906

10.

McDonagh

Metra

Adamo

, et al. ESC Guidelines for the diagnosis and treatment of acute and chronic heart failure. Eur Heart J 2021; 42: 3599–3726. https://doi.org/10.1093/eurheartj/ehab368

11.

Kim

Cho

Kim

, et al. Focused Update of 2016 Korean Society of Heart Failure Guidelines for the Management of Chronic Heart Failure. Int J Heart Fail 2019; 1: 4–24. https://doi.org/10.36628/ijhf.2019.0002

12.

Meijers

Bayes-Genis

Mebazaa

, et al. Circulating heart failure biomarkers beyond natriuretic peptides: review from the Biomarker Study Group of the Heart Failure Association (HFA), European Society of Cardiology (ESC). Eur J Heart Fail 2021; 23: 1610–1632. https://doi.org/10.1002/ejhf.2346

13.

Clephas

PRD

Radhoe

Boersma

, et al. Efficacy of pulmonary artery pressure monitoring in patients with chronic heart failure: a meta-analysis of three randomized controlled trials. Eur Heart J 2023; 44: 3658–3668. https://doi.org/10.1093/eurheartj/ehad346

14.

Boehmer

Cremer

Abo-Auda

, et al. Impact of a Novel Wearable Sensor on Heart Failure Rehospitalization: An Open-Label Concurrent-Control Clinical Trial. JACC Heart Fail 2024; 12: 2011–2022. https://doi.org/10.1016/j.jchf.2024.07.022

15.

Murton

Hillman

Mehta

, et al. Acoustic speech analysis of patients with decompensated heart failure: A pilot study. J Acoust Soc Am 2017; 142: El401. https://doi.org/10.1121/1.5007092

16.

Verdolini

Min

Titze

, et al. Biological mechanisms underlying voice changes due to dehydration. J Speech Lang Hear Res 2002; 45: 268–281. https://doi.org/10.1044/1092-4388(2002/021

17.

Maor

Perry

Mevorach

, et al. Vocal Biomarker Is Associated With Hospitalization and Mortality Among Heart Failure Patients. J Am Heart Assoc 2020; 9: e013359. https://doi.org/10.1161/jaha.119.013359

18.

Amir

Abraham

Azzam

, et al. Remote Speech Analysis in the Evaluation of Hospitalized Patients With Acute Decompensated Heart Failure. JACC Heart Fail 2022; 10: 41–49. https://doi.org/10.1016/j.jchf.2021.08.008

19.

Zhang

Ren

, et al. Deep Residual Learning for Image Recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, Jun 27–30 2016, pp. 770–778.

20.

Liu

Mao

, et al. A ConvNet for the 2020s. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, Jun 18–24 2022, pp. 11966–11976.

21.

Huang

Liu

van der Maaten

, et al. Densely Connected Convolutional Networks. In: 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, Jul 21–26 2017, pp. 2261–2269.

22.

Xie

Girshick

Dollár

, et al. Aggregated Residual Transformations for Deep Neural Networks. In: 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, Jul 21–26 2017, pp. 5987–5995.

23.

Tan

. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In: 36th International Conference on Machine Learning (ICML), Long Beach, CA, Jun 09–15 2019, pp. 6105–6114.

24.

Dosovitskiy

Beyer

Kolesnikov

, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In: ICLR 2021 - 9th International Conference on Learning Representations Virtual, Online, May 3–7 2021, International Conference on Learning Representations, ICLR.

25.

Wang

Xie

, et al. Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions.In: 18th IEEE/CVF International Conference on Computer Vision (ICCV) Virtual. Online, Oct 11–17 2021, pp. 548–558.

26.

Loshchilov

Hutter

. Decoupled weight decay regularization. In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, May 6–9 2019. International Conference on Learning Representations, ICLR.

27.

Sugrue

Mahowald

Asirvatham

. Hey Goglexiri, Do I Have Coronary Artery Disease? Mayo Clin Proc 2018; 93: 818–820. https://doi.org/10.1016/j.mayocp.2018.05.021

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

1.93 MB

0.00 MB