Abstract
Background
Most common forms of dementia, including Alzheimer's disease, are associated with alterations in spoken language.
Objective
This study explores the potential of a speech-based machine learning (ML) approach in estimating cognitive impairment, using inputs of speech audio recordings.
Methods
We develop an automatic ML pipeline that ingests multimodal inputs of audio and transcribed text, mapping speech and language to domain-specific biomarkers optimized for high explainability and predictive ability. The resulting features are fed through a multi-stage pipeline to determine efficient classification configurations.
Results
We evaluated the system on large real-world datasets, achieving above 90% and 70% weighted average F1 scores for two-class (AD versus normal controls) and three-class (AD versus mild cognitive impairment versus normal controls) classification tasks, respectively. Model performance remains stable across different population characteristics.
Conclusions
The study introduces a robust, non-invasive method for gauging the cognitive status of AD and MCI patients from speech samples, with the potential of generalizing effectively to multiple types of diseases/disorders which may burden language.
Keywords
Introduction
Common forms of dementia, such as Alzheimer's disease (AD) and Lewy body dementia (LBD), are associated with compromised cognitive functions, including impairments in memory, executive function, visuospatial recognition, and language. 1 Although memory deficit is the most characteristic symptom, language problems are apparent even at the early stages. 2 Therefore, assessments that are designed to capture deficits in a person's language can be sensitive indicators of early stages of cognitive decline 3 and useful aids to medical experts in promptly reaching an accurate diagnosis.
Language problems have been noted mostly in the lexical and semantic knowledge of people with AD (which is the most common type of dementia), particularly in naming specific part-of-speech elements, vocabulary recall, semantic fluency, morphosyntactic and grammar correctness.4–8
Current methods of early detection of cognitive decline via language examination have suggested that analysis of spontaneous or connected speech may lead to the discovery of linguistic biomarkers of dementia that can distinguish the speech of individuals with AD, mild cognitive impairment (MCI), or other types of dementia from healthy adults.9,10 Further studies 11 showed that patients with AD make more lexical errors (e.g., word-discovery difficulties, repetitions, phonemic paraphasias, and semantic substitutions) than controls, as well as using less complex sentences (coordinated and reduced).
In light of this body of evidence that illustrates the effects of cognitive impairments in speech and language, methods that analyze patient speech, using medically-oriented audio analysis for cognitive assessment have shown remarkable promise. However, language evaluation with traditional manual methods can be made easier to use, more straightforward, and above all more accessible to patients.
As a result, advances in machine learning (ML) methods have been met with increasing popularity in health and medicine research, 12 using data-driven models for robust and highly accurate decision making. These efforts however come with different challenges related to dataset size/quality requirements, poor explainability, as well as lack of utilization of multiple sources of information channels from source data. 13
Following this direction, the study presented in this paper addresses the aforementioned issues via the combined use of ML, explainable AI (xAI), audio analysis and natural language processing (NLP), to produce domain-focused solutions for language-based cognitive evaluation of patients with MCI and AD. We propose two different approaches; a two-class approach distinguishing between populations with AD and normal controls, and a three-class approach, which distinguishes between populations with MCI, AD, and normal controls. Combining explainability and efficiency, the proposed approaches are applicable as much to research as to commercial use cases.
Methods
Data collection
The present investigation on AD involves a data-driven empirical analysis, using a dataset that comprises real-world patient audio elicitation task recordings. The dataset involves hundreds of instances annotated with “AD”, “MCI”, and “Normal Control” labels and spans two languages (English and Greek). The instance pool involves more than 13,790 Greek and 2037 English samples, with a diverse population in terms of gender, education level and cognitive score that provides a rich set of cognitive assessment samples in terms of content and variance.
The data used was obtained from participating primary care facilities, memory/senior care clinics, and university hospitals. The organizations that contributed to our data collection procedures were: the IASIS non-profit organization (https://www.iasismed.eu/), the Greek Association of Alzheimer's Disease and Related Disorders (https://www.alzheimer-hellas.gr/index.php/en/home/aboutus), the Azura Living healthcare provider (https://www.azuraliving.com/), and the Watercrest senior care facilities (https://watercrestseniorliving.com/).
As the scope of this research was intentionally as broad as possible, the only exclusion criterion applied was that participants were 50 years or older. And, given the fact that the vast majority of the participants were in the desirable age group, it is safe to claim that our data can be considered real-world.
Patients were asked to complete different elicitation tasks that are commonly used as verbal cognitive assessment tests. 14 These include picture description tasks such as “Cookie Theft”, 15 everyday activity recounting (e.g., describing how to make coffee or tea), narrative tasks (e.g., describing an ideal holiday), and many other broadly accessible verbal tasks that can highlight useful patterns in use of language and speech. All tasks were completed with the help of hospital/clinic personnel, nurses, caretakers or family members, accompanied by appropriate instructions and participant understanding of task requirements, as well as informed consent. Patient responses were recorded as audio files in the test-taking device (e.g., computer or tablet) and anonymized, prior to storage and analysis.
Ground truth labels were provided by medical experts (through behavioral, clinical, radiological examinations, etc.) conducted in the source venues, following established medical practices and reference criteria. 16 Further details and statistics relevant to the employed data and domain (e.g., demographics, cognitive tests, train/test sizes, etc.) are available in Table 1.
Data statistics for the dataset used in the experimental evaluation.
The “Attribute” column portrays the names of descriptive data properties; columns labeled “Total”, “English”, and “Greek” represent their values across all observations and exclusively among the English and Greek subsets, respectively, for the AD (“AD”), MCI (“MCI”) and Normal Control (“NC”) cases. For attributes of age, Mini-Mental State Examination (MMSE) score, recording duration (in seconds) and transcript token length (extracted via Automatic Speech Recognition (ASR) from audio) we display their average and standard deviation values. Gender F, M and O/M refers to male, female and other/missing, while education B, U, P and M refers to basic (up to primary or high school), undergraduate, postgraduate and missing/unspecified, respectively. For both gender and education, we display what percentage of the participants belong to each subgroup.
Feature extraction and preprocessing
Our cognitive analysis approach utilizes different data modalities obtained from a patient's verbal elicitation task, namely audio, text, speech and language, into a multimodal processing and learning pipeline.
First, we process audio recordings of patient verbal elicitations to extract multiple audio features that capture different characteristics in the time and frequency domain representation of the source signal. These features include a wide array of spectral statistics, duration, pause distributions, silence and hesitation measurements, Mel-Frequency Cepstral Coefficients (MFCC) features 17 and more. Then we apply state-of-the-art Automatic Speech Recognition (ASR) methods, utilizing Google's ASR service (https://cloud.google.com/speech-to-text), in order to extract textual transcripts from the source audio. This is first used to extract a set of textual features that are domain-independent, which are subsequently mapped to token-level statistics, by tokenizing with existing AI tools such as spaCy 18 and applying TF-IDF weighting. 19 The end result is a set of importance-scaled term weights for tokens of interest in the transcript. Additional processing of the textual transcript is facilitated by spaCy to generate domain-sensitive features. This includes efficient tokenization, lemmatization, dependency parsing and further linguistic processing. This detailed set of linguistic information is utilized to define a large set of speech and language-oriented biomarker features, expanding on similar previous work, 3 and to develop new measures and descriptors.
The resulting feature pool captures major behavioral, verbal, linguistic views of the cognitive status of an individual, including language syntactic complexity and morphosyntactic structure, concept identification, lexical richness, variation and repetition, among others. These views group together features into coherent interpretable feature groups, validated and designed with the help of medical experts, and constituting effective biomarkers for cognitive decline. Each biomarker group reflects a major aspect of language-based evaluation. These biomarker groups are: i) syntactic complexity: the length of the sentences, the range of syntactic structures used and their degree of entanglement, ii) syntactic errors: violations of the syntactic and grammatical rules, such as omission of verbs, subjects or articles and incomplete sentences, iii) lexical variation: the amount of unique different words used, iv) lexical repetition: frequency of use of the same words, v) similarity: usage of similar phrases on a sentence level, and of similar sentences on a document level, vi) amount of content words: the amount of words with lexical meaning that represent ideas, objects, actions and attributes, vii) verbs: the choice and the use of the appropriate verb, as this is cognitively demanding, viii) function words: words that express grammatical relationship with other words within a sentence, ix) grammatical constituents: Elements and words that must be used in a specific order to form a grammatically correct sentence, x) determiners-pronouns: elements and words that occur together with a noun and express the reference of a noun in a sentence, xi) prepositions and conjunctions, xii) compounds of content included/concepts recall: How many of the actions and situations that exist in the picture or story are described in the patient's narration, xii) confrontation naming: recalling the semantically appropriate word to describe or name a concept, an object, or an action, and xiii) spatially organized syntactic structure: the extended use of the structure “here… and there..”, expresses lack of cohesion and fluency in a narrative.
Speech related biomarkers have also been utilized, by analyzing the vocal signal recorded. These biomarker groups are: i) extralinguistic vocal biomarkers: aspects of the speaker's voice quality (ex. pitch range, loudness), ii) speech pace / SpectrumR: The rate of the speech, as a low rate of speech is a sign of cognitive deficit, iii) duration of speech: the duration of speech implies the existence of cognitive deficits, and iv) pauses-silence duration: the quantity and the duration of pauses and fillers (e.g., ‘um’) used.
A comprehensive table of feature categories, the tools used to extract them (e.g., spaCy), and their quality measurements, can be found in Supplemental Material 1.
Feature selection
In order to alleviate noise, eliminate redundancies, and reduce the computational load of the subsequent steps in the pipeline, we apply two steps of feature selection. First, we apply correlation-based filtering to discard features that are highly correlated with others, and then we use statistical feature selection to limit the feature pool. We experiment with two different selection approaches, namely forward selection and recursive elimination, 20 which implement feature pool reduction via bottom-up and top-down approaches, respectively. In both cases, feature importance is gauged by the learning method, and statistical stability is provided by k-fold cross validation. The performance of the learned model in each fold is being evaluated by the area under the receiver operating characteristic curve 21 (RoC-AuC score). Once the selection procedure is complete, we obtain a set of salient biomarkers that are best suited for efficient cognitive classification.
Modeling
Given the selected features, we subsequently use ensemble-based ML methods to classify cognitive diseases of interest. We experiment with two different popular ensemble learning approaches. The first is Random Forests, 22 which uses voting-based aggregation over weak tree-based models, and the second is XGBoost, 23 which combines the optimization power of gradient-based learning with the explainability potential of tree-based models. For both approaches, we adopt decision trees 24 as weak learners in the ensemble, which are non-parametric models that partition the feature space into hierarchical structures. Given a data point, a decision tree pushes it from its root to one of its leaves, interrogating a single feature value at each step. The leaves of the tree ultimately classify the sample as belonging to one of two classes in a two-class problem (in our case, AD or Normal Controls) or as belonging to one of three classes in a three-class problem (in our case, AD or MCI or Normal Controls). During the learning procedure, the optimal splits are determined and during the prediction task the new, unknown data sample traverses the tree and is directed to a leaf node, which will denote its corresponding predicted label.
The ensemble combines the predictions of different weak learners (decision trees) to improve the aggregate prediction, e.g., via meta learning voting mechanisms (Random Forests) or residual-based model weight updates with regularized gradient descent (XGBoost). In both cases, the aggregate prediction is adopted as the final result of the classification component of the pipeline.
We train these learners for cognitive status classification, applying rigorous hyperparameter tuning via grid search under a k-fold cross-validation setting, to determine the number of trees in the ensemble, the feature split criterion and scope, size of leaf node support, etc. The final model is the best-performing configuration according to established classification metrics. A detailed table of the hyperparameters used, their definition and the best chosen values for each fold of each model can be found in Supplemental Material 2.
In terms of implementation, we develop a python-based machine learning stack, using established open-source audio analysis, text processing and machine learning libraries. These include librosa 25 for audio feature extraction, NLTK 26 and spaCy 18 for NLP tasks such as tokenization, lemmatization, dependency parsing and further linguistic processing.
Additionally, we use the sklearn, mlxtend, xgboost and pandas libraries for data processing, transformation and learning operations.23,27–29 The experiments were run on a virtual machine using an Intel Xeon 64-core 2.80 GHz CPU with 128GB of RAM.
Evaluation
Setup
The performance of the selected model and the robustness of the model selection process were assessed using another level of cross-validation. The outer cross-validation split was employed to evaluate model performance, while the inner, nested cross-validation split was used to build the learning models according to the methodology described in the methods section. Therefore, hyperparameter tuning and feature selection were performed each time on data that were independent from the data on which the model was tested. We empirically set the number of inner and outer cross-validation folds to k = 5, and the correlation feature filtering threshold to 0.8. Subsequently, the main experimental exploration evaluated different model configurations, including the two different ensemble methods (Random Forest versus XGBoost) and ensemble hyperparameters (e.g., number and architecture of weak learners, regularization strength, etc.), feature selection approaches (RFE versus SFS) and the use of combinations of input feature modalities (audio, NLP and domain-specific speech and language features), resulting in 128 distinct configurations. Each of these configurations produced a different model, and the best configuration was selected based on the average F1-score among the inner validation folds. The final result that we report is the average performance of the best configuration across 5 separate, unseen test sets (outer validation folds), evaluated using multiple classification metrics. 30
In order to provide a detailed understanding of model performance, all metrics are reported also at the level of individual classes, again using standard aggregation protocols that consider per-class instance support (e.g., macro and weighted averages).
In addition to measuring model performance on the complete test set, we also measure it on subsets corresponding to gender, education level and age. Furthermore, we also seek to identify the elicitation task that demonstrates the best performance, as relying on a single task is more practical for generating diagnostic estimates in real-world applications.
Results
Tables 2 and 3 show our experimental results for the AD versus Normal Control investigation and the AD versus MCI versus Normal Control investigation, respectively, using the selected configuration for each problem and language. We report performance scores in terms of Precision, Recall and F1-score, providing class-wise and average scores (macro and weighted), along with class support and performance baselines using a stratified probabilistic sampling classifier, where the training data label distribution is used to sample predictions for test instances. The best-performing model configuration uses recursive feature elimination to select features and XGBoost-based classification, utilizing all input modalities (audio, NLP and domain specific features) for both languages. In terms of elicitation task selection, the “Cookie Theft” task demonstrated the best performance.
Ad versus Normal Control classification results based on the proposed pipeline.
Precision, recall and F1-score values are shown in percentages. Weighted average depicts metric results weighted by class membership size. The baseline row corresponds to naive classification via stratified probabilistic sampling.
Ad versus MCI versus Normal Control classification results based on the proposed pipeline.
Precision, recall and F1-score values are shown in percentages. Weighted average depicts metric results weighted by class membership size. The baseline row corresponds to naive classification via stratified probabilistic sampling.
Further model evaluation examined classification performance in different groups of interest. Model results in terms of weighted F1-score, aggregated for gender, education and age bracket are presented in Figures 1–6 for the binary classification task. In general, our results demonstrate a good performance for most subgroups. Some variations that are observed suggest the need for further studies focusing on specific groups. Such focused studies require the collection of appropriate data.
Discussion
The main result of our work is the good performance of the classification models, trained to identify AD from simple and easy-to-acquire spoken language. In particular, our approach achieves macro average recall scores of 88.15% for the two-class English task and 90.91% for Greek. In the harder 3-class case, the corresponding scores are 77.68% for English and 74.89% for Greek. Additionally, the precision of the method seems to err towards the underrepresented class, across all use cases. Hence, we estimate that performance could improve further with the acquisition of more instances. Moreover, an inspection of the biomarker pool that was selected in the model building and optimization process provides avenues for explainability, evaluation and informed consideration of the classification outputs: Both English and Greek models share their most important categories, these being Vocal characteristics, Compounds of words included, Syntactic complexity, Lexical variation and Demographics. For the Greek population specifically, the “education” demographic illustrates the importance of education disparities in AD patients in Greece. 31 Importance distribution among features and biomarker groups is transparent in the generated results, imbuing the proposed method with qualities important to combating cognitive impairments in the real-world. However, general homogeneity in the prominent features between the two models indicates that we can use the same features for the training of a future, language-agnostic model.
Weighted F1 score by gender (English). Bar values indicate the number of instances in the group, while the plotted line corresponds to the group's weighted-average F1 score.
Weighted F1 score by education (English). Bar values indicate the number of instances in the group, while the plotted line corresponds to the group's weighted-average F1 score. Education levels with less than 10 observations are omitted.
Weighted F1 score by age (English). Bar values indicate the number of instances in the group, while the plotted line corresponds to the group's weighted-average F1 score. Age brackets with less than 10 observations are omitted.
Weighted F1 score by gender (Greek). Bar values indicate the number of instances in the group, while the plotted line corresponds to the group's weighted-average F1 score.
Weighted F1 score by education (Greek). Bar values indicate the number of instances in the group, while the plotted line corresponds to the group's weighted-average F1 score. Education levels with less than 10 observations are omitted.
Weighted F1 score by age (Greek). Bar values indicate the number of instances in the group, while the plotted line corresponds to the group's weighted-average F1 score. Age brackets with less than 10 observations are omitted.
Concerning education, we observe a significant difference among populations in Greece and the US (where corresponding language data is obtained from): in Greece, a larger portion of the population, i.e., 32.5% in a group with a mean 80 years of age, has only completed primary education.29,32,33 Prevalence of dementia-related diseases is known to be higher in this population. Looking closer at the Greek population, we notice a fall in performance as we move from primary to higher education levels. This could be related to the known fact that the diagnosis of the disease is harder at a higher education level.34–36 However, this pattern is not observed in the English-speaking population, where performance scores remain stable across all education levels.
Further, when analyzing performance at different age brackets, we observe minor variations, but no systematic trend. Small variations may be related to disproportionate representation of different age levels in the dataset. Finally, regarding gender, the performance gap is similar in the two languages (3%-4%). Overall, the results indicate a consistent and robust performance of the models across both languages.
Conclusion
In this work, we examined a novel approach for assisting the diagnosis of cognitive impairments like AD and MCI, using sophisticated speech and language biomarkers and machine learning methods. The experimental results showcase the effectiveness of the proposed solution in discriminating between AD, MCI, and Normal Controls in our main evaluation, as well as its robustness potential with respect to multiple investigative axes of interest. High classification performance and robust results are achieved in multiple standard metrics and across important dimensions, like gender, education, and age brackets. Additionally, the empirical evaluation showcases that high performance is achievable across different languages (i.e., English and Greek); notably, these critical scalability/robustness goals can be achieved using the same methodology (i.e., feature extraction, learning and optimization) and by employing explainable features and learning algorithms.
It is important to stress out that the above are achieved via simple, fast, non-invasive verbal assessments, which require a sample that is less than 2 min, and can produce results in an average of 3 min. The proposed method shows strong feasibility, as it can be implemented in any quiet setting (e.g., home, clinic), using any internet-enabled device (mobile phone, laptop, tablet). It can be administered with physician supervision, or be self-administered.
In light of the body of previous efforts and their limitations, the current study extends and improves upon related work on the task of the automated diagnosis of AD. Specifically, the contributions of the study are as follows:
Multi-language dataset and disease setting: Most related studies are limited to investigations in a narrow setting. Here, we performed an extensive analysis of audio data from English and Greek cohorts; we included normal control groups and patients diagnosed at various stages, for the task of early detection of AD. High performance and explainability: The proposed pipeline combines multimodal data (audio and text) and domain specific approaches, resulting in a pool of insightful biomarkers for disease classification. This approach combines the performance potential of an automated ML solution with interpretable results, validated and developed with the help of medical experts. Large-scale evaluation: While related studies are limited to analysis of very small collections of data (ranging from a dozen samples usually, to the low hundreds on rare occasions), this study presented a large-scale experimental evaluation of the proposed solution on multiple languages and disease labels, using thousands of data points annotated by medical experts. Additionally, we assessed both the performance and the robustness of the predictive models on the entire dataset, as well as on sub-populations with different characteristics of clinical interest. Real-world data and solution applicability: In contrast to contemporary cognitive evaluation systems that over-engineer limited research datasets and focus on achieving good results in a narrow academic scope, the proposed solution was evaluated in diverse, real-world data. Our experiments highlighted the applicability, ease of use, scalability and utility of the proposed solution as a decision support system in modern health care workflows.
The proposed predictive tools can be valuable decision-support systems, providing highly accurate, interpretable results for real-life use in primary care, memory and senior care clinics, etc. They are capable of achieving substantial coverage of a patient's cognitive status, across different languages, and enabling machine learning-powered workflows that capture a wide range of changes in speech and language. These include shifts, limitations and errors which may arise from cognitive decline due to AD.
Future work includes the combination of the present biomarker pool with deep neural networks on voice and language data, combining performance and scalability of distributed features with clinical interpretability and explainable semantics of domain-specific engineered features. Additionally, we are investigating an expansion of the multimodal suite of the proposed solution with medical images (e.g., CT/MRI scans). Further, we are planning to evaluate the methodology presented in this work in the detection of additional diseases, once sufficient annotated data have been collected. In particular, we are working on further comparative studies with different forms of dementia (e.g., Parkinson's disease) and application of the discrimination pipeline in a differential setting (e.g., different types of dementia).
Supplemental Material
sj-docx-1-alz-10.1177_13872877251319401 - Supplemental material for Linguistic cues for automatic assessment of Alzheimer’s disease across languages
Supplemental material, sj-docx-1-alz-10.1177_13872877251319401 for Linguistic cues for automatic assessment of Alzheimer’s disease across languages by Vassiliki Rentoumi, Evangelos Vassiliou, Nikiforos Pittaras, Admir Demiraj, Manolis Papageorgiou, Dimitra Sali, Athina Papatriantafyllou, Panagiotis Griziotis, Artemis Chardouveli, Konstantinos Pattakos and George Paliouras in Journal of Alzheimer's Disease
Footnotes
Acknowledgments
The authors extend their gratitude to IASIS AKME, the Greek Association of Alzheimer's Disease and Related Disorders, the Azura Living healthcare provider and the Watercrest senior care facilities for their support in providing invaluable data for our research. Their commitment to advancing scientific knowledge and their willingness to collaborate have been instrumental in the success of this study.
Author contributions
Vassiliki Rentoumi (Conceptualization; Supervision); Evangelos Vassiliou (Formal analysis; Investigation; Methodology; Visualization); Nikiforos Pittaras (Writing – original draft); Admir Demiraj (Investigation; Software; Validation; Writing – original draft); Manolis Papageorgiou (Investigation; Software; Validation); Dimitra Sali (Supervision); Athina Papatriantafyllou (Data curation; Project administration); Panagiotis Griziotis (Data curation; Project administration; Writing – review & editing); Artemis Chardouveli (Data curation; Project administration); Konstantinos Pattakos (Writing – review & editing); George Paliouras (Supervision; Writing – review & editing).
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Data availability
The data supporting the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy and ethical restrictions.
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
