Predicting mental disorder from noisy questionnaires: an anomaly detection approach based on keyword extraction and machine learning techniques

Abstract

The early warning of mental disorders is of great importance for the psychological well-being of college students. The accuracy of conventional scaling methods on questionnaires is generally low in predicting mental disorders, as the questionnaires contain much noise, and the processing on the questionnaires is rudimentary. To address this problem, we propose a novel anomaly detection framework on questionnaires, which represents each questionnaire as a document, and applies keyword extraction and machine learning techniques to detect abnormal questionnaires. We also propose a new keyword statistic for the calculation of option significance and three interpretable machine learning models for the calculation of question significance. Experiments demonstrate the effectiveness of our proposed methods.

Keywords

Mental health text analysis interpretability TF-IDF Likert scale

1 Introduction

The high stress of college students not only affects their academic performance [25] but also causes serious mental disorders, leading to self-harm and even suicide [7]. Studies show that the prevalence of mental disorders such as depression and self-harm among college students is on the rise [28]. Therefore, detecting potential mental disorders of college students at an early stage is of great importance for maximizing their psychological well-being.

The psychometric questionnaire is currently one of the main instruments for gathering information for the analysis of mental disorders. For example, the Likert scale [30], the most widely used approach to scaling responses in psychometric questionnaires, calculates the score of a questionnaire as the sum of scores of picked options, each of which is assigned with an integer value predefined by the questionnaire designers. This scaling method has caused considerable controversy, as the value assigned to each option is arbitrary, and it is assumed that each question is equally important and the distance between each option is equivalent, which has no objective basis of measure theory or numerical experiments [34].

Due to the shortages of conventional scaling methods, some researchers apply machine learning techniques to identify mental disorders. One advantage of machine learning technology is that it can learn appropriate weights for different features. For example, machine learning techniques such as support-vector network [9] and convolutional neural networks [42] are used for early warning of mental disorders based on clinical data [14], physiological data [35], video and audio data [36], behavioral data [37], social media data [6]. These data, however, are not easy to obtain in practical settings.

Compared to the above data source, questionnaires are a more economical and convenient data source. Applying machine learning techniques on questionnaires for early warning of mental disorder, however, faces the following challenges:

Noise data in the questionnaires. There are various types of noise during the design of the questionnaire and the collection of data. A questionnaire may contain questions that cannot provide valuable information to the detection of mental disorder. In addition, respondents may misunderstand the questions or try to conceal their real feelings or conditions. These noises significantly increase the difficulty of early warning.

Imbalanced distribution of samples. The number of students with mental disorders is much less than that of normal students. The performance of general machine learning techniques is seriously degraded when they are applied to the imbalanced dataset.

Interpretable requirements from users. Unlike general machine learning tasks, a model of early warning of mental disorders requires high interpretability. That is, the model must present the reason for the decision it has made.

To address the above problems, we propose an anomaly detection framework based on text analysis techniques, which can be used for early warning of mental disorders in colleges, or more generally, to analyze questionnaires containing closed-ended questions. We refine the framework and propose a process for early warning of mental disorders. By representing the combinations of picked options as a document, we can impose keyword extraction techniques to calculate significance scores of options and filter unrelated questions. We have also proposed several machine learning models for calculating the significance scores of questions. Experiments show that our methods outperform baseline machine learning techniques, and provide much higher accuracy in early warning of mental disorders compared with the conventional scaling method currently used in colleges.

The main contributions of this paper are as follows:

1. An anomaly detection framework based on text analysis techniques is proposed. This framework is applicable to general questionnaire data composed of closed-ended questions. Based on this framework, we present an early warning process for mental disorders, which can eliminate most noise in the questionnaire.

2. A novel statistic for calculating options significance and filtering questions is proposed. Unlike statistics commonly used in keyword extraction applications, this statistic takes into account the highly imbalanced characteristics of the dataset in anomaly detection problems, thus significantly improving the accuracy for identifying abnormal samples.

3. Three interpretable anomaly detection models are proposed. Among them, Naive OF-IDF is highly interpretable, while Attention OF-IDF and Graph OF-IDF achieve high accuracy of the model while preserving acceptable interpretability.

The remainder of this paper is organized as follows. Section 2 introduces some works related to this study. In Section 3, we describe the dataset of the application problem. We present our proposed methods in section 4, including the framework of anomaly detection, a novel statistic, and three prediction models, and demonstrate the effectiveness of these methods in section 5. Finally, we conclude our work in Section 6.

2 Related work

2.1 Mental disorders and college students

College students have to face many challenges, such as adapting to a new environment, establishing new relationships, and coping with academic pressure [29]. In the process of coping with these challenges, college students may suffer from various psychological distress, mental health problems, and even self-harm and suicidality. For example, it is found that there is a strong relationship between suicidal ideation and depression for college students [7]. Natural disasters or pandemics also have a noticeable impact on the mental health of college students. A recent study shows that 24.9%of college students become anxious, after the outbreak of novel coronavirus disease 2019 (COVID-19) [39]. In fact, college students have more serious mental health problems comparing with their contemporaries [40]. And it is found that the prevalence of mental illness among college students is on the rise [19]. If mental disorders of college students can be identified at an early stage, proper intervention or assistance can be provided to them, thus reducing the risk of depression and suicide, and mitigating the deleterious effect on their physical and mental wellbeing. Many researchers found that the risk factors for college students’ mental disorders include personality characteristics [26], academic performance [28], personal background [18], social ecology [41], loneliness [13] and so on. At the same time, researchers found that these factors react to each other. For example, it was found that loneliness is an important factor leading to depression in college students [4]. Our study also demonstrates that considering the interplay between various factors is helpful to improve the detection accuracy of mental disorders.

2.2 Detecting mental disorders by using questionnaires

At present, psychometric questionnaires are employed as one of the primary tools to collect data for the early detection of mental disorders for college students [33]. Likert scale, as a presentative of scaling approach for questionnaires is commonly used for scoring a psychometric questionnaire [30], where each question is a statement related to some measurement topic. The respondent chooses the degree of agreement or disagreement with the statement in the options, which usually includes five levels, corresponding to a set of increasing or decreasing scores. For instance, the scale of Baker Depression Inventory (BDI) [5] is designed to measure depression, the R-UCLA Loneliness Scale [12] to measure loneliness, and the Eysenck Personality Scale (EPQ) to analyze personality characteristics, which are all psychometric questionnaires used for the detection of specific mental disorders.

Although the questionnaire data is easier to obtain and analyze than other data sources such as clinical data and video data, there are considerable disagreement with the conventional scaling method [34], where the impact of each question on the questionnaire result is considered independent and equivalent, and each option in a question is assigned with an integer score by the designer of the questionnaire. For example, for a five-level Likert scale, the scores of the options generally range from 1 to 5, which has no objective basis of measure theory or numerical experiments [16]. Besides, such scoring method assumes that the distance between each successive option is equivalent, but it may not be interpreted as such by the respondents, resulting in inaccurate results [22]. Our experiments also demonstrate that the conventional scoring method has led to a higher rate of misjudgment.

2.3 Detecting mental disorders by using machine learning techniques

Some studies leverage machine learning techniques to forecast certain mental disorders in specific populations. For example, a neural network model is employed to predict the possibility of psychological conditions for patients with concussion, based on their past condition and longitudinal clinical trajectory [14]. Deep learning networks are used for automatic analysis of depression by extracting human behavior primitives from videos [36]. Other studies construct machine learning models to predict mental disorders based on data sources including sensor data [37], EMG signals and ECG signal [35], and social media [6]. The above methods require specialized data sources such as past medical conditions, videos, or sensor data that can reflect the mental states of people. However, these data are generally difficult to obtain in practical applications. They may be applicable to patients in hospitals but not suitable for common college students.

2.4 Keyword extraction techniques

Keyword extraction is the process of identifying keywords, i.e., important words or phrases representing the main topic of a given document [27]. In this paper, we propose a novel approach to calculate the significance scores of options and to filter questions by leveraging the keyword extraction technique. Keyword extraction is usually based on various text features, such as syntactic features, statistical features, and features calculated from external resources. Among them, statistical features are widely used in large corpora [23]. Common statistical features include TF-IDF [17], domain relevance (DR) [32], co-occurrence features (such as TextRank [31]), and similarity features (such as Jaccard index [10]). TF-IDF is a simple but effective statistic, which has been widely used in information retrieval, position detection [3], topic detection [1], and so on. Because the dataset for anomaly detection is highly imbalanced between normal and abnormal samples [38], TF-IDF is not the appropriate statistic for the identification of abnormal samples. To address this problem, we propose a revised statistic for anomaly detection on the basis of TF-IDF.

3 Datasets

This study aims to construct an interpretable model to identify the abnormal questionnaires that indicate the respondents might suffer from mental disorders in college. The dataset comes from the Mental Health Education Center of a university. It consists of 28,583 questionnaires, which were completed by respondents on the first month they came to the university after being anonymized. Each respondent answered only one questionnaire.

The questionnaire mainly includes three categories of questions: solid-state questions, Beck Depression Inventory (hereinafter referred to as BDI) [5], Eysenck Personality Questionnaire (hereinafter referred to as EPQ) respondents. Details about the questionnaire questions are presented in Table 1.

Table 1
Psychological scale composition

Question category Number Question scope

Solid-state 40 It contains questions about facts or questions with relatively stable answers over a long period, such as parents’ marital status, experience, social support, and learning motivation

BDI 21 It contains questions about the recent feelings and conditions of the respondents, including sadness, disappointment, guilt, their sleep quality, etc.

EPQ 85 It consists of four personality self-report scales: introversion/extroversion tendency scale (E), emotional scale (N, also known as neuroticism), psychopath scale (P, also known as psychoticism), and effectiveness scale (L).

Question category	Number	Question scope
Solid-state	40	It contains questions about facts or questions with relatively stable answers over a long period, such as parents’ marital status, experience, social support, and learning motivation
BDI	21	It contains questions about the recent feelings and conditions of the respondents, including sadness, disappointment, guilt, their sleep quality, etc.
EPQ	85	It consists of four personality self-report scales: introversion/extroversion tendency scale (E), emotional scale (N, also known as neuroticism), psychopath scale (P, also known as psychoticism), and effectiveness scale (L).

There are a total of 146 questions in the questionnaire. For each question in the questionnaire, a respondent picks one and only one option as the answer. Hence, each answered questionnaire can be represented by a 146-dimension feature vector, where each feature represents the picked option to a question.

Statistics data show that 90%or more of students study for no more than four years in this university, so we focus on predicting whether a respondent will suffer from some mental disorders in the first four years of their college lives. There are 265 respondents who were diagnosed with mental disorders within four years after being enrolled. The questionnaires completed by these respondents are marked as positive samples. A total of 17,112 respondents have no records of mental disorders in the first four years after being enrolled. The questionnaires completed by these respondents are marked as negative samples. Altogether, there are 17,377 positive or negative samples in our study.

4 Method

We first propose a novel framework for anomaly detection, which employs text analysis techniques to score questionnaires containing close-ended questions. Then we present a process, within this anomaly detection framework, to identify abnormal questionnaires with regards to mental disorders. Some details of this process will be presented at the end of this section.

4.1 An anomaly detection framework based on text analysis techniques

In this section, we present an anomaly detection framework based on text analysis, which accepts answers to a set of closed-ended questions and outputs anomaly detection results. Although the framework was originally designed for the identification of abnormal questionnaires on psychometric tests, it can also be applied to questionnaires for other purposes, as long as the questionnaire contains many close-ended questions.

Figure 1 shows our proposed anomaly detection framework. Before presenting details of the framework, we first introduce some basic notation. We assume that a questionnaire contains multiple close-ended questions, each question contains multiple options, and there are totally m questions and n options in the questionnaire. Let Q = {q₁, q₂, ⋯ , q_m} represent the set of questions, where q_i denotes the i-th question, and let O = {o₁, o₂, ⋯ , o_n} represent the set of options, where o_j represents the j-th option. The relationship between options and question is denoted by a function h, h : O → Q, which maps an option to its corresponding question. Note that h is a surjection but not an injection, which implies that an option must belong to some question and each question contains two or more options. We describe five steps of the proposed framework as follows.

Fig. 1

Anomaly detection framework based on text analysis techniques.

Mapping to Document. The first step of the framework is to convert a completed questionnaire into a document-like representation, thereby providing a basis for applying text analysis techniques to the problem of anomaly detection. Let w_k ∈ O denotes the option picked for the k-th question, then answers to the questionnaire can be represented as a document D = (w₁, w₂, ⋯ , w_m), where each option w_k corresponds to a word in document D.

Calculating option significance. The significance of each option to the detection result has much difference. Some options have more powerful effect than others. We denote the significance score of an option by function g : O → R⁺, where u_i = g (o_i) represents the significance of the i-th option. Conventionally, the significance score of each option is assigned by experts. Since a questionnaire is now represented as a document, we can employ some text analysis techniques to calculate the option significance.

Filtering Questions. Some questions of questionnaire may not be relevant for anomaly detection. A few questions can even be noise, thus having a negative effect on anomaly detection. Moreover, a large number of questions will seriously reduce computation efficiency. Therefore, we reduce the number of questions in this step by removing irrelevant or noisy questions.

Calculating question significance. Conventionally, each question in a questionnaire is treated equally, which implies that the significance of each question to the detection result is the same. We assume that different questions have distinct significance to anomaly detection. The calculation of the significance score for each question is denoted by the function f : Q → R⁺, where v_i = f (h (o_i)) represents the significance score of the question containing option o_i.

Outputting detection result. The detection results of a questionnaire is based on a total score of the questionnaire. The total score is calculated as the weighted sum of significance scores of all picked options where the weights are assigned as the significance score of the corresponding questions. Calculation of questionnaire scores s is formulated as follows: $s = \sum_{i = 1}^{m} u_{i} v_{i} = \sum_{i = 1}^{m} g (o_{i}) f (h (o_{i}))$ (1) where o_i denotes the i-th option. u_i and v_i represent the significance score of o_i and the significance score of the question containing o_i, g (·) and f (·) calculate the significance score of the option and the question. h (·) maps the option to its corresponding question. Based on the value of s, we can decide whether the questionnaire is normal or abnormal with some simple rules.

4.2 A process of detecting abnormal questionnaires for mental disorders

We proposed a process of identifying abnormal questionnaires with regards to mental disorders by bringing some machine learning techniques into the framework present in the last section (see Figure 2). The proposed process conforms to the anomaly detection framework depicted in Figure 1, but introducing three new components:

Fig. 2

Process of detecting abnormal questionnaires for mental disorders.

1. Machine learning algorithms. Some machine learning algorithms are employed to calculate the final score, including a keyword extraction algorithm and a classification algorithm. The former is introduced to construct a model for computing the option significance while the latter is for computing the question significance.

2. Supervised data set. The supervised data set is an component for building a supervised model, which is expected to achieve better performance than unsupervised models. The mental disorder records are introduced to supervise the construction of two machine learning models in this application.

3. Model interpretation. The model interpretation presents the rationale behind the model about the anomaly detection results.

Most steps shown in Figure 2 are the same as those in Figure 1. The other steps are described as follows:

Calculating option significance. Each picked option to a question is treated as a word in the document after we represent the questionnaire as a document. This enables us to calculate option significance with a resort to the keyword extraction technique. TF-IDF is one of the most popular statistic for keyword extraction. We propose a revised TF-IDF by considering the characteristics of the anomaly detection problems. The detailed process will be presented in Section 4.3.

Calculating question significance. The importance of each question can be acquired during the construction of a classification model. Such a model is constructed by applying a classification algorithm to the existing questionnaire data and mental disorder records. We propose three classification models for calculating question significance. This work will be described in detail in Section 4.4.

4.3 Calculation of option significance

Conventionally, option significance is designated by the designer of the questionnaire. We can also employ text analysis techniques, such as keyword extraction techniques, for the calculation of option significance after representing the questionnaire as a document. In this section, we propose a statistic for anomaly detection applications based on TF-IDF.

TF-IDF(Term Frequency–Inverse Document Frequency) is a statistical measure that is intended to evaluate the relevance of a word to a document in a corpus. The calculation of TF-IDF is based on the following observation: If a word occurs frequently in a particular document, but seldom in other documents, it may be regarded as a representative term of that document. Hence, we can define TF-IDF as follows: ${TF}_{i, j} = \frac{n_{i, j}}{\sum_{k} n_{k, j}}$ (2) ${IDF}_{i} = lg \frac{N}{N (r_{i})}$ (3) $TF - {IDF}_{ij} = {TF}_{i, j} * {IDF}_{i}$ (4) where n_i,j represents the number of occurrences of the word r_i in the document d_j, N represents the total number of documents in the corpus, and N (r_i) represents the number of documents containing r_i. If a word frequently appears in a particular document, its TF is high, and if it rarely appears in other documents, its IDF is high. The TF-IDF is the product of TF and IDF, so words with high value of TF-IDF can be taken as a keyword of the document.

In our approach, each picked option is modeled as a word, therefore we can leverage TF-IDF to calculate the importance of options. Unlike common document, however, each option occurs no more than once in a questionnaires. Hence we redefine the TF-IDF as follows: ${TF}_{i} = p_{i}^{+}$ (5) ${IDF}_{i} = lg \frac{N}{N (r_{i})}$ (6) $TF - {IDF}_{i} = {TF}_{i} * {IDF}_{i}$ (7) where $p_{i}^{+}$ denotes the proportion of abnormal questionnaires containing option r_i to all abnormal questionnaires, N denotes the number of all questionnaires, and N (r_i) denotes the number of questionnaires containing option r_i. It can be seen from Eq. (7) that the TF-IDF of an option is high if it appears frequently in the abnormal questionnaires but it total frequency is low in all questionnaires.

However, Eq. (5-7) does not take into account an essential fact about anomaly detection problems, i.e., normal samples and abnormal samples are highly unbalanced, and the number of abnormal samples is usually much lower than that of normal samples. For example, suppose there are 2000 normal questionnaires, but only 20 of them contain option r; the number of abnormal questionnaires is 200, and again, 20 of them contain option r. Intuitively, r is an important option for the detection of the abnormal questionnaire, because the likelihood that r occurs in an abnormal questionnaire is 10 times than that it occurs in a normal questionnaire. The TF-IDF value of this option, however, is usually not very high due to the small value of TF. Therefore, many important options might be missed in similar cases.

To address this problem, we propose a new statistic called option frequency-inverse document frequency (OF-IDF), considering the characteristics of anomaly detection. It is calculated as follows: $OF - {IDF}_{i} = \frac{p_{i}^{+}}{p_{i}^{-}} * {TF}_{i} * {IDF}_{i}$ (8) where $p_{i}^{-}$ denotes the proportion of normal questionnaires containing option r_i to all normal questionnaires. Unlike Eq. (7), OF-IDF introduces the ratio between $p_{i}^{+}$ and $p_{i}^{-}$ . As we demonstrate in experiment, this is an important revision for improving the performance of anomaly detection.

4.4 Calculation of question significance

After obtaining the significance of each option, we calculate abnormal scores for all questionnaires as follows: $S = f (X \times A)$ (9) where × denotes matrix multiplication. X ∈ R^(n×m) denotes the matrix of option significance (n and m denote the number of questionnaires and questions), and X_ij represents the significance of the picked option for the j-th question of i-th questionnaire, which is calculated by using the method introduced in section 4.2. A ∈ R^(m×m) denotes the matrix of the significance of the questions, where A_ii represents the significance of i-th question, and when i ≠ j, A_ij represents the impact of i-th question on j-th question; f : R^(n×m) → R^(n×1) represents the aggregation operation, which combines scores of all questions into one score for each questionnaire; S ∈ R^(n×1) denotes questionnaire scores, where S_i represents the abnormal score of the i-th questionnaire.

We proposed three approaches to acquire matrix A. The second and third approaches leverage machine learning techniques to compute the elements of A. These methods are described in the following subsections.

4.4.1 Naive OF-IDF

A straightforward approach is to calculate the abnormal score of the questionnaire as the sum of OF-IDF of each picked option. We name the model as Naive OF-IDF. Its calculation can be formulated as follows: $S = sum (X \times I)$ (10) where sum (·) denotes the operation of summing by each row, and I is the identity matrix. In other words, the question significance matrix A in Eq. (10) is assigned to the identity matrix I. Naive OF-IDF, which assumes that the weights of the questions are all equals, and there is no interdependence between questions.

4.4.2 Attention OF-IDF

Naive OF-IDF does not utilize label data in the training set. Nevertheless, it is possible to learn better question significance by taking advantage of the label information. Hence, we propose Attention OF-IDF, which computes the significance matrix based on the attention mechanism. The detailed process is shown in Figure 3(a).

Fig. 3

Flowchart of Attention OF-IDF(a) and Graph OF-IDF(b).

Specifically, we use the self-attention mechanism [2] to calculate the question significance matrix A. The calculation formula is as follows: $A = SM (\frac{{QK}^{T}}{\sqrt{d_{v}}})$ (11) where Q, K ∈ R^(m×d_v) represents two learnable matrices. SM (·) is the softmax function. A_ij represents the attention coefficient of the i-th problem to the j-th problem;

The questionnaire score is calculated as follows: $S = σ (mean (X \times A))$ (12) where mean (·) represents the average operation by row, and σ represents the sigmoid function.

We choose cross-entropy as the loss function to learn the attention coefficient matrix A by employing the backpropagation algorithm [20]. Note that this coefficient matrix is indeed the question significance matrix A that we try to obtain.

4.4.3 Graph OF-IDF

Our third approach models the relationships between questions with a graph. Each node of the graph represents a question, and each edge represents the interrelationship between two questions. By controlling the sparsity of the graph, a compromise between model accuracy and model interpretability is achieved. Inspired by a graph neural network model named GraphWaveNet [43], we construct an end-to-end supervised training model and learned an adaptive adjacency matrix from the training set. We call this model Graph OF-IDF, the detailed process of which is shown in Figure 3(b).

Graph OF-IDF calculates the adjacency matrix A as follows: $A = SM (ReLU (E_{1} E_{2}^{T}))$ (13) where E₁, E₂ ∈ R^(m×d) represent two randomly initialized learnable parameter matrices, which represent the embedding of the source question node and the target question node respectively, and m is the number of questions.SM (·) represents the softmax function.

To compute the questionnaire anomaly score S, we feed the adaptive adjacency matrix A and the questionnaire matrix X into a graph convolutional network. This process can be formulated as follows: $S = GCN (A, X) = SM (ReLU ((\tilde{A} X^{T})^{T} W))$ (14) where $\tilde{A} = A + I, I \in R^{(m \times m)}$ represents the identity matrix, which is used to strengthen the influence of a question on itself. W ∈ R^(m×1) represents a learnable matrix.

Again, we learn matrix A with a backpropagation algorithm. Besides the cross-entropy loss, we also introduce the L1 regularization loss of matrix A, to ensure its sparsity.

5 Experiment

5.1 Evaluation indicator

We use two evaluation indicators to verify the effectiveness of our proposed methods and compare them with some baseline approaches. Both indicators are suitable for scenarios where positive and negative samples are highly imbalanced.

AUC. The AUC value refers to the area under the receiver operating characteristic (ROC). ROC curve is a plot that visualizes the performance of a binary classifier based on TPR (true positive rate) and FPR (false positive rate). FPR and TPR are calculated as follows: $FPR = \frac{FP}{FP + TN}$ (15) $TPR = \frac{TP}{TP + FN}$ (16) where FP represents the number of negative or normal samples with incorrect predictions; TN represents the number of negative samples with correct predictions; FN represents the number of positive or abnormal samples with incorrect predictions; TP represents the number of positive samples with correct predictions.

TPR - P′. AUC reflects the average accuracy of the model under different thresholds. In a practical application, however, we have to determine one threshold for anomaly detection. Generally, samples identified as abnormal by the model require further processing by human resources, which is limited for a specific organization. Therefore, the number of samples identified as abnormal, P′ = FP + TP, should not be too large to exceed the supply of human resources, or too small to find only a few abnormal samples. Hence, the accuracy rate with a predefined P′ is also important for a specific organization. We use TPR-P’ to represent the TPR of the model given the value of P′. Note that when P′ and TPR is determined, FPR is also fixed, so TPR-P’ can represent the accuracy of the model.

5.2 Baseline methods

Scaling method. We choose the Likert-scale approach as the scaling method, which has been applied for many years by the university where the respondents study.

Classification methods. We choose three classification models with high interpretability as the baseline methods, including Decision tree(DT), Logistics Regression(LR) and Naive Bayesian(GNB).

Anomaly detection methods. We choose four anomaly detection models as the baseline methods, including Robust Covariance(RC), Local Outlier Factor(LOF), Isolation Forest(IForest) and One-Class SVM(OC-SVM).

5.3 Experimental setup

We randomly divide the original data set into the training set and the test set with the ratio of 7:3. Statistics about these data sets are shown in Table 2.

Table 2
Experimental data statistics

Data set Number of positive samples Number of negative samples

Train set 185 11978

Test set 80 5134

All 265 17112

Data set	Number of positive samples	Number of negative samples
Train set	185	11978
Test set	80	5134
All	265	17112

The three classification methods and the four anomaly detection methods in the baseline methods are all implemented using the machine learning library Scikit-learn [15]; the data standardization in the baseline method uses the StandardScaler method in Scikit-learn. For our proposed two models, we set the batch size as 100 and epochs _ num = 20 in stochastic gradient descent. The Adam optimizer is used to train the model with an initial learning rate of 0.01.

Other hyperparameters are set as follows: α = 1.6387 for Naive OF-IDF,d = 6 for Graph OF-IDF, and d_k = 6 for Attention OF-IDF. The number of predicted positive samples P′ is set to 500, as suggested by the expert in the university which provides the dataset, considering the capacity of its human resource.

5.4 Experimental results

It can be seen from Table 3 that most of the machine learning model methods significantly exceed the conventional scoring method for scales. Logistics regression (LR) achieved the best performance among baseline methods. But our proposed three models achieve better performance than it in both AUC and TPR-500. Among them, Graph OF-IDF is 5.25%higher than LR on the TPR-500. Compared with the classification models, the performance of the anomaly detection model is generally worse. This is probably because the supervised label data provide classification models with more information which helps them outperform regular anomaly detection models.

Table 3
Compare performance with the baseline methods

Model TPR-500 AUC

Scaling Scale-follow 21.25% 0.56

Classification DT 16.25% 0.53

LR 41.25% 0.66

GNB 33.75% 0.62

Anomaly detection RC 30.00% 0.60

LOF 13.75% 0.52

IForest 33.75% 0.62

OC-SVM 33.75% 0.62

Our methods Naive OF-IDF 45.00% 0.68

Attention OF-IDF 42.50% 0.67

Graph OF-IDF 47.50% 0.69

	Model	TPR-500	AUC
Scaling	Scale-follow	21.25%	0.56
Classification	DT	16.25%	0.53
	LR	41.25%	0.66
	GNB	33.75%	0.62
Anomaly detection	RC	30.00%	0.60
	LOF	13.75%	0.52
	IForest	33.75%	0.62
	OC-SVM	33.75%	0.62
Our methods	Naive OF-IDF	45.00%	0.68
	Attention OF-IDF	42.50%	0.67
	Graph OF-IDF	47.50%	0.69

We test the effectiveness of OF-IDF by comparing it with the TF-IDF defined in Eq. (7) for three models. As can be seen from Table 4, the introduction of OF-IDF effectively improves the performance of three models comparing with those based on TF-IDF. There is at least 6%improvement at TPR-500 for our three methods. And there are also improvement on AUC with varying degrees.

Table 4

Comparison of OF-IDF and TF-IDF

	TPR-500		AUC
	TF-IDF	OF-IDF	TF-IDF	OF-IDF
Direct	38.75%	45.00%	0.65	0.68
Attention	35.00%	42.50%	0.63	0.67
Graph	41.25%	47.50%	0.66	0.69

We also verify the effectiveness of the question filtering step in our proposed framework. Table 5 lists the results of ablation experiments where we use QF+ to indicate the inclusion of the question filtering step in the process, and QF- to indicate the exclusion of it. It can be found that both the Baseline method and the method we proposed have significantly improved the effect after question filtering.

Table 5

Comparison of with or without question filtering steps

	TPR-500		AUC
	QF-	QF+	QF-	QF+
Naive OF-IDF	38.75%	45.00%	0.65	0.68
Attention OF-IDF	26.25%	42.50%	0.58	0.67
Graph OF-IDF	32.50%	47.50%	0.62	0.69
DT	16.25%	20.00%	0.53	0.55
LR	41.25%	42.50%	0.66	0.67
GNB	33.75%	41.25%	0.62	0.66
RC	30.00%	37.50%	0.60	0.64
LOF	13.75%	22.50%	0.52	0.57
IForest	33.75%	42.50%	0.62	0.67
OC-SVM	33.75%	36.25%	0.62	0.64

Finally, we tested the stability of our models when P′ is set to different values. In this application, P′ is proportional to the capacity of human resources in a university to carry out follow-up operations for the questionnaires identified as abnormal. Therefore, the value of P′ may change across different universities. As depicted in Figure 4, our proposed methods consistently achieve the highest accuracy for different values of P′, which convinces the stability of that our methods.

Fig. 4

Comparison of each method under different P′.

The above experimental results have demonstrated the superiority of our proposed models to other baseline methods. Particularly, our methods can provide universities with more reliable support in the detection of abnormal questionnaires compared with the conventional scoring method for scales.

5.5 Case study

We choose four representative questionnaires as the cases to illustrate the interpretability of OF-IDF. The option significance scores and questionnaire scores of these questionnaires are shown in Table 6.

Table 6
Analysis of prediction results of case questionnaire

QN ₁ QN ₂ QN ₃ QN₄1

1 1.48 0.05 0.05 0.05

2 2.00 0.01 0.01 0.01

3 0.04 0.04 0.04 0.04

4 1.53 0.04 1.53 1.53

5 0.01 0.01 0.01 0.01

6 0.01 0.01 0.01 0.01

7 0.00 0.00 0.00 0.00

8 0.00 0.00 0.00 0.00

9 1.37 0.01 0.01 0.01

10 1.65 0.05 0.05 0.05

11 0.55 0.23 0.23 0.23

12 1.82 0.05 0.05 0.05

13 1.17 1.17 1.17 1.17

14 1.48 0.09 1.48 0.09

S 13.12 1.79 4.67 3.29

Predicted result abnormal normal normal normal

True label abnormal normal normal abnormal

After question filtering, a questionnaire contains only fourteen questions, and the significance scores of the selected options for these questions are listed in the first fourteen rows of the table, where each column corresponds to one of four questionnaires. Options with greater significance scores are in bold type. The third row from the bottom lists the abnormal score of the questionnaire calculated by Naive OF-IDF, which equals the sum of the scores of the first fourteen rows. The last two rows show the predicted results and true labels of the questionnaire.

As shown in the last two rows of Table 6, the first three questionnaires are correctly identified by the model. QN₁ contains plenty of options with high significance scores, therefore are identified as an abnormal questionnaire. QN₂ and QN₃ are identified as normal questionnaires, as they contain few options with higher scores. For the same reason, the abnormal questionnaire QN₄ is incorrectly identified as a normal one. Actually, the distribution of QN₄ is very similar to that of QN₃, which explains why the model is misled. Although the judgment of the model might be incorrect sometimes, it can always present reasonable evidence for its judgment. Therefore, we can say that the Naive OF-IDF has high interpretability.

Figure 5 plots the probability density curves [8] of the Naive OF-IDF scores for both normal questionnaires and abnormal questionnaires. The horizontal coordinate indicates the Naive OF-IDF score, and the vertical coordinate indicates the probability density, and the black dashed line indicates the corresponding abnormal score threshold when P′ = 500. It can be seen from Figure 5 that the higher the score is, the more likely the questionnaire is an abnormal one. However, the positive and negative samples overlap in a large range. In particular, the scores of some abnormal samples are the same as or even smaller than normal samples. It is probably that the respondents were in a normal state of mind when filling out the questionnaire, or some respondents concealed their real situations, causing the questionnaires to fail to include significant options for anomaly detection.

Fig. 5

Naive OF-IDF probability density plot.

To explain the effect of the question weight matrix on questionnaire score, we visualized the weight matrix learned by Graph OF-IDF, as shown in Figure 6.

Fig. 6

Heat map of question adjacency matrix learned by Graph OF-IDF.

The horizontal and vertical axes in Figure 6 denote the sequence numbers of fourteen questions. The gray level is proportional to the mutual impact between the two questions. Elements on the diagonal have the largest values, indicating that the maximum impact on a question is from itself. In addition, there are also noticeable influences between some pairs of questions. By filtering out question pairs that have little influence on each other, a sparse directed graph can be constructed, as shown in Figure 7.

Fig. 7

Sparse graph constructed from the question adjacency matrix shown in Figure 7.

In Figure 7, the tail endpoint of directed edges represents the affected nodes. For example, question 1 has a significant impact on question 14. In this way, when calculating the score of question 14, we also need to consider the options of question 1. Graph OF-IDF has achieved a higher accuracy rate than Naive OF-IDF, which may imply that these questions are not completely independent, and a more accurate model can be constructed by considering their mutual impact. Graph OF-IDF achieves a higher accuracy at the expense of interpretability. Nevertheless, models with lower interpretability cannot guarantee higher accuracy. For example, the accuracy of Attention OF-IDF is not as good as Naive OF-IDF. Among the three proposed models, if interpretability is most concerned, the Naive OF-IDF is preferred; if the model performance is most valued, we can choose Graph OF-IDF, which can also provide good interpretability.

6 Conclusion

Early warning of mental disorders for college students helps to improve their psychological well-being in college. We propose a novel anomaly detection framework on questionnaires and refine it to a process of detecting abnormal psychometric questionnaires based on keyword extraction and machine learning techniques. We also propose a new keyword statistic and three models to calculate the option significance and question significance. Experiments demonstrate that our proposed methods outperform conventional scaling methods or common machine learning techniques. The proposed framework and process can also be applied to general questionnaires containing closed-end questions.

Acknowledgments

This work is supported by National Natural Science Foundation Project of CQ CSTC (No.cstc2020jcyj-msxmX).

References

Daud

, Abbas

, Amjad

, Alshdadi

A.A.

and AlowibdiFinding

J.S.

, rising stars through hot topics detection, FutureGeneration Computer Systems 115 (2021), 798–813.

Vaswani

, Shazeer

, Parmar

, Uszkoreit

, Jones

, Gomez

A.N.

, Kaiser

and Polosukhin

Attention Is All You Need, ArXiv:1706.03762 [Cs], (2017).

Al-Ghadir

A.I.

, Azmi

A.M.

and Hussain

, A novel approach to stancedetection in social media tweets by fusing ranked lists andsentiments, Information Fusion 67 (2021), 29–40.

Rich

A.R.

and Scovel

, Causes of Depression in College Students: ACross-Lagged Panel Correlational Analysis, Psychol Rep 60 (1987), 27–30.

Beck

A.T.

, An Inventory for Measuring Depression, Arch GenPsychiatry 4 (1961), 561.

Hao

, Li

and Zhu

Predicting Mental Health Status on Social Media, (2013), 101–110.

Weber

, Metha

and Nelsen

Relationships among Multiple Suicide Ideation Risk Factors in College Students, Journal of College Student Psychotherapy (1997), 49–64.

Silverman

B.W.

Density estimation for statistics and data analysis, Chapman & Hall/CRC, Boca Raton, (1998).

Cortes

and Vapnik

, Support-vector networks, Mach Learn 20 (1995), 273–297.

10.

Manning

and Schütze

Foundations of statistical Natural Language Processing, MIT Press, (1990).

11.

Gutierrez

, Forbes

and Johnson

S.K.

, Physical and PsychologicalHealth Predict Adherence to an Online Mindfulness Program forCollege Students, Counseling and Values 65 (2020), 206–221.

12.

Russell

, Peplau

L.A.

and Cutrona

C.E.

, The Revised UCLALoneliness Scale: Concurrent and Discriminant Validity Evidence, Journal of Personality and Social Psychology 3 (1980), 472–480.

13.

Chang

E.C.

, Chang

O.D.

, Lucas

A.G.

, Li

, Beavan

C.B.

, Eisner

R.S.

, McManamon

E.C.

, Rodriguez

N.S.

, Katamanin

O.M.

, Bourke

E.C.

, de la Fuente

, Cardeñoso

, Wu

, Yu

E.A.

, Jeglic

E.L.

and Hirsch,

J.K.

, Depression, Loneliness, and Suicide Risk among LatinoCollege Students: A Test of a Psychosocial Interaction Model, Social Work 64 (2019), 51–60.

14.

Dabek

and Caban

J.J.

ANeural Network Based Model for Predicting Psychological Conditions, in: Y. Guo, K. Friston, F. Aldo, S. Hill, and H. Peng (Eds.), Brain Informatics and Health, Springer International Publishing (2015), 252–261.

15.

Pedregosa

, Varoquaux

, Gramfort

, Michel

, Thirion

, Grisel

, Blondel

, Müller

, Nothman

and Louppe

, Scikit-learn: Machine Learning in Python, Journal of MachineLearning Research 12 (2011), 2825–2830.

16.

Norman

, Likert scales, levels of measurement and the “laws” ofstatistics, Adv in Health Sci Educ 15 (2010), 625–632.

17.

Salton

, Yang

C.S.

and Yu

C.T.

, A theory of term importance inautomatic text analysis, J Am Soc Inf Sci 26 (1975), 33–44.

18.

, Li

, Liu

, Zhao

, Prevatt

and Yang

, Variablespredicting the mental health status of Chinese college students, Asian Journal of Psychiatry 1 (2008), 37–41.

19.

Xiao

, Carney

D.M.

, Youn

S.J.

, Janis

R.A.

and Castonguay

L.G.

, J.A.Hayes and B.D. Locke, Are we in crisis? National mental health andtreatment trends in college counseling centers, PsychologicalServices 14 (2017), 407–415.

20.

Hecht-Nielsen , Theory of the backpropagation neural network, International Joint Conference on Neural Networks (1989), 593–605.

21.

Qiu

, Shen

, Zhao

, Wang

, Xie

and Xu

, A nationwidesurvey of psychological distress among Chinese people in theCOVID-19 epidemic: implications and policy recommendations, GenPsych 33 (2020), e100213.

22.

Blaikie

Analyzing Quantitative Data London, SAGE Publications Ltd, (2003).

23.

Firoozeh

, Nazarenko

, Alizon

and Daille

, Keywordextraction: Issues and methods, Nat Lang Eng 26 (2020), 259–291.

24.

Othman

, Ahmad

, El Morr

and Ritvo

, Perceived impact ofcontextual determinants on depression, anxiety and stress: a surveywith university students, Int J Ment Health Syst 13(2019), 17.

25.

Frazier

, Gabriel

, Merians

and Lust

, Understanding stressas an impediment to academic performance, Journal of AmericanCollege Health 67 (2019), 562–570.

26.

Ganem

P.A.

, de Heer

and Morera

O.F.

, Does body dissatisfactionpredict mental health outcomes in a sample of predominantly Hispaniccollege students? Personality and Individual Differences 46 (2009), 557–561.

27.

Turney

P.D.

, Learning Algorithms for Keyphrase Extraction, Information Retrieval 2 (2000), 303–336.

28.

Selvaraj

P.R.

and Bhat

C.S.

, Predicting the mental health of collegestudents with psychological capital, Journal of Mental Health 27 (2018), 279–287.

29.

Beiter

, Nash

, McCrady

, Rhoades

, Linscomb

, Clarahanand

and Sammut

, The prevalence and correlates of depression, anxiety,and stress in a sample of college students, Journal ofAffective Disorders 173 (2015), 90–96.

30.

Likert

, A technique for the measurement of attitudes, Archives of Psychology 140 (1932), 1–55.

31.

Mihalcea

and Tarau

Text Rank: Bringing Order into Texts, Proc Conference on Empirical Methods in Natural Language (2004).

32.

Navigli

and Velardi

Semantic interpretation of terminological strings, In Proceedings of the Conference on Terminology and Knowledge Engineering (TKE) (2002), 95–100.

33.

Bonner

R.L.

and Rich

A.R.

, Toward a Predictive Model of SuicidalIdeation and Behavior: Some Preliminary Data in College Students, Suicide and Life-Threatening Behavior 17 (1987).

34.

Jamieson

, Likert scales: how to (ab)use them, Med Educ 38 (2004), 1217–1218.

35.

Pourmohammadi

Stress detection using ECG and EMG signals: A comprehensive study, Computer Methods and Programs in Biomedicine (2020), 13.

36.

Song

, Shen

and Valstar

, Based Automatic Depression Analysis Using Hand-Crafted Statistics and Deep Learned Spectral Features, IEEE International Conference on Automatic Face & Gesture Recognition (2018), 158–165.

37.

Thakur

S.S.

and Roy

R.B.

Predicting mental health using smart-phone usage and sensor data, J Ambient Intell Human Comput (2020).

38.

Chandola

, Banerjee

and Kumar

, Anomaly detection: A survey, ACM Comput Surv 41 (2009), 1–58.

39.

Cao

, Fang

, Hou

, Han

, Xu

, Dong

and Zheng

, Thepsychological impact of the COVID-19 epidemic on college students inChina, Psychiatry Research 287 (2020), 112934.

40.

Larcombe

, Finch

, Sore

, Murray

C.M.

, Kentish

, Mulder

R.A.

, Lee-Stecum

, Baik

, Tokatlidis

and Williams

D.A.

, Prevalence and socio-demographic correlates of psychologicaldistress among students at an Australian university, Studies inHigher Education 41 (2016), 1074–1091.

41.

Usher

and Curran

, Predicting Australia’s university students’mental health status, Health Promotion International 34(2019), 312–322.

42.

LeCun

, Boser

, Denker

J.S.

, Henderson

, Howard

R.E.

, Hubbard

and Jackel

L.D.

, Backpropagation Applied to Handwritten ZipCode Recognition, Neural Computation 1 (1989), 541–551.

43.

, Pan

, Long

, Jiang

and Zhang

Graph WaveNet for Deep Spatial-Temporal Graph Modeling, International Joint Conference on Artificial Intelligence (2019).

	QN ₁	QN ₂	QN ₃	QN₄1
1	1.48	0.05	0.05	0.05
2	2.00	0.01	0.01	0.01
3	0.04	0.04	0.04	0.04
4	1.53	0.04	1.53	1.53
5	0.01	0.01	0.01	0.01
6	0.01	0.01	0.01	0.01
7	0.00	0.00	0.00	0.00
8	0.00	0.00	0.00	0.00
9	1.37	0.01	0.01	0.01
10	1.65	0.05	0.05	0.05
11	0.55	0.23	0.23	0.23
12	1.82	0.05	0.05	0.05
13	1.17	1.17	1.17	1.17
14	1.48	0.09	1.48	0.09
S	13.12	1.79	4.67	3.29
Predicted result	abnormal	normal	normal	normal
True label	abnormal	normal	normal	abnormal

Predicting mental disorder from noisy questionnaires: an anomaly detection approach based on keyword extraction and machine learning techniques

Abstract

Keywords

1 Introduction

2 Related work

2.1 Mental disorders and college students

2.2 Detecting mental disorders by using questionnaires

2.3 Detecting mental disorders by using machine learning techniques

2.4 Keyword extraction techniques

3 Datasets

4.1 An anomaly detection framework based on text analysis techniques

5.1 Evaluation indicator

5.3 Experimental setup

Table 2 Experimental data statistics Data set Number of positive samples Number of negative samples Train set 185 11978 Test set 80 5134 All 265 17112

Acknowledgments

References

Table 2
Experimental data statistics

Data set Number of positive samples Number of negative samples

Train set 185 11978

Test set 80 5134

All 265 17112