Development of a Suicidal Ideation Detection Tool for Primary Healthcare Settings: Using Open Access Online Psychosocial Data

Abstract

Background: Suicidal patients often visit healthcare professionals in their last month before suicide, but medical practitioners are unlikely to raise the issue of suicide with patients because of time constraints and uncertainty regarding an appropriate approach. Introduction: A brief tool called the e-PASS Suicidal Ideation Detector (eSID) was developed for medical practitioners to help detect the presence of suicidal ideation (SI) in their clients. If SI is detected, the system alerts medical practitioners to address this issue with a client. The eSID tool was developed due to the absence of an easy-to-use, evidence-based SI detection tool for general practice. Material and Methods: The tool was developed using binary logistic regression analyses of data provided by clients accessing an online psychological assessment function. Ten primary healthcare professionals provided advice regarding the use of the tool. Results: The analysis identified eleven factors in addition to the Kessler-6 for inclusion in the model used to predict the probability of recent SI. The model performed well across gender and age groups 18–64 (AUR 0.834, 95% CI 0.828–0.841, N = 16,703). Healthcare professionals were interviewed; they recommended that the tool be incorporated into existing medical software systems and that additional resources be supplied, tailored to the level of risk identified. Conclusion: The eSID is expected to trigger risk assessments by healthcare professionals when this is necessary. Initial reactions of healthcare professionals to the tool were favorable, but further testing and in situ development are required.

Introduction

Suicide is the second leading cause of death among young people aged 15 to 29 years and the fifth leading cause of death among 30 to 49 year olds.¹ One of the strongest predictors of suicide attempts is the experience of suicidal ideation (SI), which includes having thoughts, beliefs, mental images, voices, or other cognitions about killing oneself.² Analysis of World Health Organization mental health (MH) survey data from 17 countries and involving more than 48,000 participants revealed an estimated lifetime prevalence of SI of 9.2%, with 60% of first-ever suicide attempts occurring within the first year of SI onset.³ Hence, early identification of SI risk has strong potential to aid suicide prevention.

Detection of at-risk people can be challenging due to their reluctance to report SI and seek help.³ This is particularly relevant for men, despite men having a higher suicide death rate than women.⁴ Of 3,242 consecutive 15–24-year-old general practice male patients, 22% experienced current SI, yet only 12% presented to their general practitioner citing psychological complaints.⁵ More importantly, Busch et al.⁶ reported denial of SI for 78% of clinical inquiries regarding SI, which were conducted close to suicide. Despite the substantial proportion of at-risk people who do not seek MH support for SI, many of these people do visit a primary healthcare physician within the months that precede suicidal behavior. Estimates for the percentage of individuals who die by suicide having visited their primary healthcare physician within a month of their death, range between 45% and 75%.^7,8 This research suggests that primary healthcare physicians are well positioned to detect and manage suicide risk.

However, there remain barriers to suicide risk assessment in primary healthcare settings in addition to a reluctance among suicidal patients to verbalize thoughts of suicide.^9,10 Primary healthcare physicians have reported insufficient time to conduct suicide risk assessments within a single consultation^11,12 and many physicians cite low confidence in their risk assessment skills.^11,13,14,15 As a result of these barriers, the rate of inquiry for SI in primary care settings is low, even when patients present with depression or requests for antidepressants (36%–42% according to Feldman et al.⁸

Computerized assessment has been suggested as a way to overcome patient-related barriers to SI disclosure, providing more accurate information to healthcare professionals than face-to-face enquiries.¹⁶ The current research uses such data, collected in a computerized self-reported environment to develop a detection tool for SI. This is very different from a suicide screening tool, of which there are many, usually asking very direct questions about suicide planning and previous attempts.^14,15,17,18 The tool developed as part of the present study predicts the probability of SI using sociodemographic risk factors for SI, alcohol and substance use, and psychological distress as measured using the Kessler Psychological Distress Scale-6 (K6).

There are several sociodemographic risk factors for SI such as age, gender, education, employment history, and marital history.¹⁹ Female gender, younger age, and low educational attainment have consistently been identified as risk factors of SI among adults, with Nock et al.³ proposing that these three risk factors may be “universal.” Prospective research has identified that being unemployed and/or disabled is the strongest predictor of SI among adults and being separated, divorced, or widowed has also been identified as a strong risk factor for SI.²⁰ These sociodemographic risk factors may be a reflection of the risk imposed through social disadvantage and isolation.³ Military service has also been noted as an important risk factor for SI.²¹

This article describes the development of an SI detection tool on the basis of computerized self-report assessment data, designed for patients aged 18–64 who present with psychological distress at a medical center, because the presence of mood and anxiety disorders significantly increases the risk of SI.²⁰ The use of a large data set, collected using an open access online MH Web site, meant that the costs of development for this tool were relatively low.

Materials

The Kessler Psychological Distress Scale-10 (K10) is a simple checklist that aims to measure the level of depression and anxiety in the past 4 weeks, with higher scores evidencing more distress.²² This study used the Kessler-6 (K6), which is a shortened form of the K10 and has been found to have good reliability in this study (Cronbach's alpha = 0.855) and studies conducted in a number of different countries, showing good concordance with independent clinical ratings of serious mental illness.^23,24 Given the wide use of the K10 and the K6 in the healthcare system, there is strong potential to integrate the use of these brief scales with known sociodemographic risk factors to produce an SI detection tool. This article describes the development of such a tool using data from the “electronic psychological assessment screening system” (e-PASS). The e-PASS is the assessment module in Anxiety Online²⁵—now Mental Health Online (MHO). MHO is an open access online psychoeducational, assessment, and treatment service for anxiety disorders (www.mentalhealthonline.org.au).

The e-PASS includes a variety of demographic and personal questions, the K6 and a recent SI question, as well as an online program, for the diagnosis of 21 mental disorders according to the American Psychiatric Association criteria (DSM-IV-TR).²⁶ Clients can access MHO from anywhere in the world, provided they have an Internet connection, to obtain a psychological assessment and/or an online anxiety treatment program. However, to date, the majority (76%) of clients have been Australian residents. Those who undertake the e-PASS are first required to register and consent to the MHO terms and conditions. The procedures for collection and reporting of the MHO data were approved by the Swinburne University Human Research Ethics Committee. The e-PASS has been described and validated in other studies. In particular, a sample of 158 participants was used to evaluate the diagnostic capability of the e-PASS with a structured clinical interview serving as the gold standard,²⁷ and an analysis of e-PASS diagnostic comorbidities confirmed previous research results.²⁸

The e-PASS was completed 16,703 times between October 2009 and June 2013 by clients aged from 18 to 64 years. It was completed by 15,605 separate individuals with 94.5% of these people completing the e-PASS only once, 4.4% completing it twice, 0.7% completing it thrice, and 0.4% completing it more than three times with a maximum of seven times. The e-PASS is an open access platform, and so, it is important that some attempt be made to compare the demographics of its clients with those attending medical practitioner MH appointments, since this is the targeted population for which the e-PASS Suicidal Ideation Detector (eSID) was designed. However, a comparison of both these demographic patterns with those for deaths by suicide is also of interest, since suicide prevention is the ultimate focus of the eSID.

Table 1 shows how the age and gender statistics for the e-PASS sample differs from the 2013–2014 Bettering the Evaluation and Care of Health (BEACH) medical practitioner MH encounter data and the 2009–2013 Australian Bureau of Statistics (ABS) deaths by suicide data.²⁹ As reported by Britt et al.,³⁰ every year the BEACH survey collects detailed information from a random sample of 1,000 Australian medical practitioners, to provide an ongoing database of reliable medical practitioner/patient encounter data.

Table 1.

Age and Gender Breakdowns

	e-PASS		ABS DEATHS BY SUICIDE		BEACH MEDICAL PRACTITIONER MH ENCOUNTERS
AGE (YEARS)	MEN, N (%)	WOMEN, N (%)	MEN, N (%)	WOMEN, N (%)	TOTAL/1,000 POPULATION, N (%)
<25^a	897 (18.4)	3,062 (25.9)	1,151 (14.6)	442 (17.5)	436.7 (16.7)
25–34	1,615 (33.2)	4,210 (35.6)	1,664 (21.1)	476 (18.8)	645.6 (20.2)
35–44	1,165 (23.9)	2,440 (20.6)	1,985 (25.2)	615 (24.3)	896.3 (24.2)
45–54	808 (16.6)	1,458 (12.3)	1,847 (23.5)	586 (23.2)	955.3 (21.8)
55–64	386 (7.9)	662 (5.6)	1,224 (15.6)	409 (16.2)	989.5 (17.2)
Total	4,871 (100)	11,832 (100)	7,871 (100)	2,528 (100)	735 (100)^b

18–24 for e-PASS, 15–24 for BEACH medical practitioners, and ABS completed suicide data.

607.8/1,000 population 15+ for men and 858.0/1,000 population 15+ for women.

ABS, Australian Bureau of Statistics; BEACH, Bettering the Evaluation and Care of Health.

Table 1 suggests that the e-PASS data under-represent the number of MH patients accessing medical practitioner services in the older age groups (45+ years) and over-represent the number of MH patients in the younger age groups (<35 years). This may be due to the better computer access and/or literacy of younger MH patients. In addition, they over-represent the number of women accessing these services. This makes it essential that any model developed using the e-PASS data is checked for consistency across all relevant gender/age demographic groups.

The development of the eSID had two phases as described in the Methods section below. In the first phase, a model for predicting the probability of SI was developed using the e-PASS data and in the second phase the feedback of healthcare professionals was sought.

Methods

Phase 1: The Development of the eSID TOOL

Initially, the validity of the e-PASS recent SI question “Have you had recent thoughts of suicide?” was assessed in terms of the association between the responses to this question and the anxiety and depression severity estimates obtained from the e-PASS data. As shown in Table 2, independent sample t-tests suggest significantly higher mean anxiety and depression levels for those who respond positively to this recent SI question than for those who respond negatively, suggesting that a positive response to this SI question is indeed indicative of a higher level of psychological distress.

Table 2.

Validation of “Recent Thoughts of Suicide” with Severity Measures for Anxiety and Depression

	MEAN SEVERITY SCORES (SD)
RECENT SUICIDAL IDEATION	PANIC DISORDER	SOCIAL ANXIETY	GENERALIZED ANXIETY	POST-TRAUMATIC STRESS	DEPRESSION
No (N = 10,500)	1.11 (1.92)	1.46 (1.88)	1.87 (2.02)	0.61 (1.27)	1.63 (1.98)
Yes (N = 6,203)	1.93 (2.53)	2.62 (2.46)	3.15 (2.47)	1.45 (2.18)	4.14 (2.06)
t-Statistic	19.66^a	29.70^a	32.80^a	24.40^a	67.39^a

p < 0.001.

Based on prior research,^3,19,20 e-PASS items that were identified as relevant risk factors for SI included gender, age, employment, marital status, and social support. In addition, the e-PASS collects MH information relevant to SI risk, such as recency of access to MH treatment; interest in making changes to improve one's MH; confidence in one's ability to manage one's MH; quality of life; main MH concern; seeking MH assessment in regard to the type and severity of MH difficulties being experienced; using alcohol or substances to cope with stress; and the K6.

The e-PASS data (N = 16,703) were randomly split (40%:30%:30%) for training, validation, and test purposes, and using the previously identified risk factors for SI found in the e-PASS data, a binary logistic regression model for recent SI was developed. The model fit was evaluated for these three sets of data using areas under the receiver operating curve (ROC). These areas (AUR) can be interpreted as the probability that a client reporting SI will be ranked higher on SI risk by the model than a client who does not report SI.³¹ The validity of the final model was confirmed for men and women for five age groups between the ages of 18 and 64 years, with AUR values exceeding 0.8 in all cases. The odds ratios produced for the final binary logistic regression model indicate the relative importance of the various SI risk factors (Table 4).

In the second phase, the eSID tool was presented to healthcare professionals for their feedback in two separate focus group sessions. The first focus group meeting was held in September 2014 with two doctors, two nurses, and a medical center manager, all employed by a university-based health service. The second was held in November 2014 with three clinical psychologists and two MH nurses, employed by a community treatment and prevention program, which assists medical centers with the management of patients with MH problems. Four members of the research team attended both these meetings. The purpose of these meetings was to determine if the proposed eSID tool would be useful to medical centers and, if considered useful, to determine how such a tool should be best used.

Results

Development of the eSID Tool

The percentage of responses for recent SI was 37.1% (38.0% for men and 36.8% for women). These rates are higher than what is expected for paper-pencil questionnaires or interviews. It is thought that this is because the anonymous nature of Web-based testing invites a higher endorsement of SI.³² For those who endorsed recent SI, the mean (MN) K6 level was significantly higher (for men MN = 19.96, SD = 4.02, for women MN = 20.99, SD = 3.91) than for those who did not (for men MN = 14.99, SD = 4.40, for women MN = 15.58, SD = 4.44).

A binary logistic regression model for responses to the recent SI question was developed using the K6 and the other risk factors identified in the e-PASS data. Model fit was evaluated using the area under the ROC. Values of 0.820, 0.825, and 0.831 for the training, validation, and test data indicated that the model had good predictive accuracy when the model coefficients obtained from the training data were applied with the validation and test data. In addition, the fit of the final general model was compared with models fitted separately for each of the 10 gender/age groups indicated in Table 1. As shown in Table 3, the general model provided a good fit for all 10 of these demographic groups with the separate models providing AUR measures well within the 95% confidence intervals for the general model. This suggests that the same general model can be used for all these demographic groups.

Table 3.

Model Comparison Using Area Under Receiver Operating Curve

					AUR
	SAMPLE SIZE (N)		GENERAL MODEL		95% CONFIDENCE INTERVAL (GM)		SEPARATE MODELS
AGE (YEARS)	MEN	WOMEN	MEN	WOMEN	MEN	WOMEN	MEN	WOMEN
18–24	897	3,062	0.808	0.836	0.780–0.837	0.822–0.850	0.821	0.840
25–34	1,615	4,210	0.803	0.840	0.780–0.825	0.827–0.852	0.810	0.842
35–44	1,165	2,440	0.843	0.832	0.821–0.866	0.815–0.849	0.850	0.839
45–54	808	1,458	0.833	0.839	0.804–0.861	0.817–0.860	0.856	0.851
55–64	386	662	0.815	0.820	0.770–0.859	0.786–0.854	0.837	0.850

AUR, area under receiver operating curve; GM, general model.

Table 4 provides descriptive statistics and odds ratios for the variables used to predict the probability of SI using the general model. There were significant differences (p < 0.001) between those who endorsed recent SI and those who did not for all the risk factors included in Table 4. This table shows adjusted odds ratios for SI using the first category as the reference category for each categorical variable, when the multivariate binary logistic regression model was used to predict SI. Predictions for the probability of SI (p) were obtained using dummy (zero-one) coded data for all the categorical variables in Table 4, weighting the client data using β weights calculated as the natural logarithm of the adjusted odds ratios, because ln(exp(β)) = β.

Table 4.

Descriptive Statistics and Adjusted Odds Ratios for the e-PASS Suicide Ideation Detector Tool

		RESPONSES (N)			ADJUSTED ODDS RATIOS FOR SI
RISK FACTOR (X)	CATEGORY	TOTAL	SUICIDAL IDEATION (%)	OR (EXP(β))	95% CI LOWER LIMIT	95% CI UPPER LIMIT	SIGNIFICANCE
Gender	Men	4,871	1,853 (38.0)	1
	Women	11,832	4,350 (36.8)	0.788	0.720	0.863	0.000
Age	18–24	3,959	1,855 (46.9)	1			0.001^a
	25–34	5,825	2,020 (34.7)	0.793	0.709	0.886	0.000
	35–44	3,605	1,230 (34.1)	0.821	0.720	0.937	0.003
	45–54	2,266	774 (34.2)	0.834	0.718	0.970	0.018
	55–64	1,046	323 (30.9)	0.953	0.782	1.160	0.630
Employment	Full-time	6,851	2,110 (30.8)	1			0.003^a
	Part-time	4,160	1,528 (36.7)	1.107	0.998	1.229	0.056
	Home duties	1,246	400 (32.1)	0.841	0.714	0.992	0.040
	Disability	572	363 (63.1)	1.319	1.050	1.656	0.017
	Unemployed	1,919	1,013 (52.8)	1.037	0.908	1.185	0.588
	Other (e.g., student)	1,955	789 (40.4)	1.126	0.986	1.285	0.080
Access MH help	Never	5,246	1,390 (26.5)	1			0.000^a
	Not in last year	3,408	1,113 (32.7)	1.304	1.161	1.465	0.000
	In last year, not now	2,216	948 (42.8)	1.564	1.373	1.781	0.000
	Now	5,833	2,752 (47.2)	1.670	1.497	1.863	0.000
Marital status	Married	5,444	1,518 (27.9)	1			0.002^a
	Single	4,978	2,282 (45.8)	1.229	1.095	1.380	0.000
	Cohabiting	3,149	1,107 (35.1)	1.125	0.997	1.270	0.056
	Other	3,132	1,296 (41.4)	1.213	1.075	1.368	0.002
MH change	Not interested/no need	253	46 (18.2)	1			0.006^a
	Neither here nor there	1,279	511 (40.0)	0.834	0.520	1.338	0.451
	Prepared to make change	8,886	3,058 (34.4)	0.696	0.441	1.100	0.121
	Already making change	4,425	1,614 (36.5)	0.741	0.467	1.177	0.205
	Relapsed, need help	1,860	974 (52.4)	0.849	0.531	1.359	0.496
MH confidence	Very poor	1,297	894 (68.9)	1			0.019^a
	Poor	4,462	2,274 (51.0)	1.021	0.871	1.197	0.796
	Neither good Nor poor	6,048	2,105 (34.8)	0.924	0.785	1.087	0.341
	Good	4,110	824 (20.0)	0.826	0.689	0.991	0.039
	Very good	786	106 (13.5)	0.874	0.648	1.179	0.378
Quality of life	Very poor	543	423 (77.9)	1			0.000^a
	Poor	2,648	1,769 (66.8)	1.040	0.807	1.340	0.762
	Neither good nor poor	4,743	2,159 (45.5)	0.773	0.600	0.997	0.047
	Good	7,147	1,656 (23.2)	0.579	0.446	0.752	0.000
	Very good	1,622	196 (12.1)	0.517	0.379	0.705	0.000
Main MH concern	Depression	2,145	1,435 (66.9)	1			0.000^a
	Substance abuse	164	77 (47.0)	0.656	0.420	1.024	0.064
	Anxiety	11,523	3,729 (32.4)	0.728	0.458	1.158	0.180
	Stress	1,278	347 (27.2)	1.655	1.050	2.609	0.030
	Eating/weight	743	281 (37.8)	1.213	0.671	2.191	0.523
	None	301	38 (12.7)	0.873	0.541	1.407	0.576
Seek MH assessment	No	8,854	2,857 (32.3)	1
	Yes	7,849	3,346 (42.6)	1.211	1.120	1.310	0.000
Adequate social support	No	9,148	4,312 (47.1)	1
	Yes	7,555	1,891 (25.0)	0.826	0.759	0.900	0.000
Use alcohol to cope	No	12,726	4,496 (35.5)	1
	Yes	3,977	1,707 (42.9)	1.285	1.172	1.410	0.000
Use substances to cope	No	15,465	5,488 (35.5)	1
	Yes	1,238	715 (57.8)	1.331	1.151	1.540	0.000
K6				1.651	1.532	1.778	0.000
K6 × K6				0.992	0.991	0.994	0.000
Total		16,703	6,203 (37.1)

Significance for all categories together; \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document}$$p = {\rm exp} \left( - 6.30 + \sum \beta X \right) \bigg / \left[- 5.30 + \sum \beta X \right]$$ \end{document} = estimated probability of suicidal ideation.

CI, confidence interval; OR, odds ratio; MH, mental health.

Table 4 shows that after controlling for all the other risk factors in the model, the odds for SI were on average 21% lower for women than men. This should not be an unexpected result if one considers the higher suicide death rates for men shown in Table 1. Also, the odds for SI are significantly lower for people aged 25–54 years than for people aged 18–24 years on average, but not significantly lower for those aged 55–64 years. On average, the odds of recent SI are 32% higher for those on a disability pension than for those in full-time employment, and significantly higher for those who have previously sought MH treatment, 67% higher for those currently accessing face-to-face MH treatments. On average, single people are 23% more likely to have experienced recent SI than married people, with similar rates for those widowed, separated, or divorced. Interestingly, the odds for SI are highest for those who are not interested in making changes in regard to their MH, and SI is less likely among those with more confidence in their ability to manage their MH. Not surprisingly, this is also true for those with better quality of life.

Figure 1 shows how the estimated probability of SI varies in relation to the K6 value. The highest risk relates to patients having the highest odds ratios for SI (e.g., men, 18–24 years, disability employment status, currently accessing MH treatment, single, not interested/no need to change MH, poor confidence in ability to manage MH, poor quality of life, main MH concern is stress, seeking MH assessment, social support not adequate, using alcohol and substances to cope with stress). The lowest risk relates to patients with the lowest odds ratios for SI (e.g., women, 25–34 years, home duties employment status, never accessed MH help, married, prepared to make MH change, good confidence in ability to manage MH, very good quality of life, main MH concern is substance abuse, not seeking MH assessment, social support adequate, not using alcohol or substances to cope with stress). This graph shows that the K6 on its own is not a reliable indicator of SI.

Fig. 1.

Relationship between the estimated probability of suicidal ideation and the K6 for patients with the highest and lowest risk on all other risk factors.

Feedback from Healthcare Professionals

Using the regression weights obtained from the binary logistic regression, an Excel worksheet shown in Figure 2 was produced to demonstrate how the eSID could be used in practice, using a mobile phone or iPad/tablet device. Feedback on the form of the assessment output was especially important because the raw tool produced only an estimated probability for SI based on the variables found in Table 4.

Fig. 2.

Excel VBA user form to demonstrate the tool.

The overall response of the healthcare professionals to the above SI detection tool was very positive. The need for such a brief SI detection tool was acknowledged, particularly for high-risk university populations and in other practices where there is a high frequency for patients presenting with depression or anxiety. It was acknowledged that medical practitioners were well placed to monitor the use of the tool because distressed patients commonly asked for consultations with medical practitioners who could provide appropriate referrals and medication. It was also considered useful as a triage tool for distressed patients. Ideally, it was thought that the tool should be integrated in medical practice software, with medical practitioners receiving automatic SI prompts when accessing patient records, a major advantage over paper-based tools for assessing suicide risk. Overall, the assessment tool was seen as an effective, nonthreatening way in which to “start a conversation with patients about their mental health.”

However, there were also some concerns. Short medical practitioner consultancy times of only 15 min meant that it might not be possible for the tool to be accessed during a consultation. “Medical practitioners have limited time to hear the patient's story, re-package it, and come up with recommendations (e.g., referrals or treatment/management plans).” Also, it was felt that medical practitioners had very limited referral options (e.g., Crisis Assessment and Treatment Team [CATT] for emergencies but few sources of counseling help available at a reasonable/low cost). It was suggested that the tool should provide “clear standardized recommendations corresponding to the assessed level of risk.” While referrals or ambulance calls were needed for high-risk cases, it was recommended that for low-risk cases “the tool should prompt medical practitioners to book a follow-up appointment in the following two days, and print off a safety plan and/or list of appropriate support contacts, so that the patient knows who to contact if risk escalates before the follow-up appointment.” It was felt that medical practitioners would be open to this suggestion because “medical practitioners don't want to be just a triage/referral service.”

Although the brevity of the tool was thought to be important, some additional questions were suggested for inclusion in the tool. Additional categories for “coping with stress” were suggested (e.g., “talking with friends and family”) and a question regarding financial situation was considered particularly important. The effective trialing of the tool was also discussed, and it was suggested that the MH nurses, seconded from the local agency assisting medical centers with MH referrals and other requirements, would be helpful for this purpose.

Discussion

This study has shown that online open access Web sites have the potential to enhance the quality and reduce the development costs of new screening tools in MH and other areas. This study has produced such a tool, which shows good predictive power as a detection tool for SI, for men and women aged 18–64 years in a sample of clients accessing an online psychotherapy platform. This tool exhibits the importance of considering other risk factors, not just the K6, when assessing the risk of SI. The initial reactions of healthcare professionals to this tool were favorable. It was seen as a rapid but effective method of engaging patients in a conversation about their MH, and if necessary, asking some of the more direct questions about suicide found in the commonly used suicide screeners.^14,15,17,18

The anonymous online nature of the data used to develop this tool and the nonthreatening nature of the questions included in the tool (no mention of suicide) mean that the eSID may be more reliable than a clinical inquiry about SI, in which questions that are more explicit are used. However, this is intended only as an SI detection tool, to be followed up with standard screening and a clinical assessment of risk when there is some indication of suicidal concern. The determination of what estimated probabilities for SI are indicative of “clear suicidal concern” is critical, requiring careful calibration. Current research suggests that primary care medical practices are an appropriate setting for the use of the tool, in that it is very common for those contemplating suicide to visit their local medical practitioner in the month before suicide. Initial work suggests that the tool would be well received by primary health professionals if it can be integrated into busy medical practices by means of existing medical software systems. However, discussions with healthcare professionals have indicated that the tool could be much more useful if it could also provide appropriate referral information and safety plans. However, this SI detection tool could also be used in other settings provided that the people conducting the SI detection were given appropriate training. Detecting SI and suicidal behaviors needs to become a community activity, suggesting that organizations such as community centers, employment centers, outpatient wards in hospitals, and parole offices could also find such an assessment tool useful for monitoring the MH of their more vulnerable clients.

Clearly, further work is needed to make the tool useful for physicians. Also, the tool has been developed using a specific online community of people who had exhibited concern regarding their MH, by completing a psychiatric assessment on the MHO platform. The performance of this tool in other populations has not yet been tested. Feedback, especially in regard to related referral resources, is to be sought from medical practitioners by way of an active learning module. It is hoped that this will be offered through the Royal Australian College of General Practitioners training program. In addition to providing readings and structured instruction, this module will involve the use of the eSID with 20 patients for each of the participating medical practitioners. As recommended by Christensen et al.,³³ this means that a risk assessment tool will be combined with online personalized programs of therapy. It is hoped that this in situ testing of the eSID tool by medical practitioners enrolled in the module will ensure that this tool can be further developed and adopted by healthcare professionals more widely.

Footnotes

Acknowledgments

The authors thank Swinburne University of Technology for the seed funding used to finance this research and the Australian Department of Health and Aging for the funding used to establish the e-PASS and associated online platform. In addition, they thank the health professionals who provided so much valuable advice.

Disclosure Statement

No competing financial interests exist.

References

World Health Organization. Preventing suicide: A global imperative. Geneva, Switzerland. 2014. Available at http://apps.who.int/iris/bitstream/10665/131056/1/9789241564779_eng.pdf?ua=1 (last accessed September 17, 2016 ).

Wenzel

, Brown

, Beck

. Classification and assessment of suicide ideation and suicidal acts. In: Wenzel

, Brown

, Beck

, eds. Cognitive therapy for suicidal patients: Scientific and clinical applications. Washington, DC: American Psychological Association, 2009.

Nock

, Borges

, Bromet

, Alonso

, et al. Cross-national prevalence and risk factors for suicidal ideation, plans and attempts. Br J Psychiatry, 2008; 192:98–105.

Moller-Leimkuhler

. Barriers to help seeking by men: A review of sociocultural and clinical literature with particular reference to depression. J Affect Disord, 2002; 71:1–9.

McKelvey

, Pfaff

, Acres

. The relationship between chief complaints, psychological distress, and suicidal ideation in 15–24 year-old patients presenting to general practitioners. Med J Aust, 2001; 175:550–552.

Busch

, Fawcett

, Jacobs

. Clinical correlates of inpatient suicide. J Psychiatry, 2003; 64:14–19.

Louma

, Martin

, Pearson

. Contact with mental health and primary care providers before suicide: A review of the evidence. Am J Psychiatry, 2002; 159:909–916.

Feldman

, Duberstein

, Vannoy

, Epstein

, et al. Let's not talk about it: Suicide inquiry in primary care. Ann Fam Med, 2007; 5:412–418.

Saunders

, Brand

, Lascelles

, Hawton

. The sad truth about the SADPERSONS Scale: An evaluation of its clinical utility in self-harm patients. Emerg Med J, 2013; 31:796–798.

10.

Harris

, Syu

, Lello

, Chew

YLE

, et al. The ABC's of suicide risk assessments: Applying a tripartite approach to individual encounters. PLoS One, 2015; 10:e0127442.

11.

Bajaj

, Borreani

, Ghosh

, Methuen

, et al. Screening for suicidal thoughts in primary care: The views of patients and general practitioners. Mental Health Fam Med, 2008; 5:229–235.

12.

Vail

, Adams

, Gilbert

, Nettleingham

, et al. Investigating mental health risk assessment in primary care and the potential role of a structured decision support tool, GRIST. Ment Health Fami Medi, 2012; 9:57–67.

13.

Mott

. An integrated approach to estimating suicide risk. Suicide Life Threat Behav, 1991; 21:74–89.

14.

Chan

, Batterham

, Christensen

, Galletly

. Suicide literacy, suicide stigma, and help-seeking intentions in Australian medical students. Australas Psychiatry, 2014; 22:132–139.

15.

Graham

, Rudd

, Bryan

. Primary care providers' views regarding assessing and treating suicidal patients. Suicide Life Threat Behav, 2011; 41:614–623.

16.

Lawrence

, Willig

, Crane

, Ye

, et al. Routine, self-administered, touch-screen, computer-based suicidal ideation assessment linked to automated response team notification in an HIV primary care setting. Clin Infect Dis, 2010; 50:1165–1173.

17.

Stanley

, Brown

. Safety planning intervention: A brief intervention to mitigate suicide risk. Cogn Behav Pract, 2012; 19:256–264.

18.

Jobes

, Jacoby

, Cimbolic

, Hustead

LAT

. Assessment and treatment of suicidal clients in a university counseling centre. J Couns Psychol, 2013; 44:368–377.

19.

Scocco

, de Girolamo

, Vilagut

, Alonso

. Prevalence of suicide ideation, plans, and attempts and related risk factors in Italy: Results of the European study on the epidemiology of mental disorders—World mental health study. Compr Psychiatry, 2008; 49:13–21.

20.

Borges

, Angst

, Nock

, Meron Ruscio

, et al. Risk factors for the incidence and persistence of suicide-related outcomes: A 10-year follow-up study using the National Comorbidity Surveys. J Affect Disord, 2008; 105:25–33.

21.

Hoffmire

, Kemp

, Bossarte

. Changes in suicide mortality for veterans and nonveterans by gender and history of VHA service use. Psychiatr Serv, 2015; 66:959–965.

22.

Kessler

, Barker

, Colpe

, Epstein

, et al. Screening for serious mental illness in the general population. Arch Gen Psychiatry, 2003; 60:184–189.

23.

Australian Bureau of Statistics (ABS). Information paper: Use of the Kessler psychological distress scale in ABS health surveys, Australia. Catalogue No. 4817.0.55.001. Australian Bureau of Statistics: Canberra. 2001.

24.

Cornelius

BLR

, Groothoff

, van der Klink

JJL

, Brouwer

. The performance of the K10, K6 and GHQ-12 to screen for present state DSM-IV disorders among disability claimants. BMC Public Health, 2013; 13:128.

25.

Klein

, Meyer

, Austin

, Kyrios

. Anxiety online—A virtual clinic: Preliminary outcomes following completion of five fully automated treatment programs for anxiety disorders and symptoms. J Med Internet Res, 2011; 13:e89.

26.

American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders (4th edition, text revision). Washington, DC: American Psychiatric Association, 2000.

27.

Nguyen

, Klein

, Meyer

, Abbott

, et al. The diagnostic validity and reliability of an internet-based clinical assessment program for mental disorders. J Med Internet Res, 2015; 17:e218.

28.

Al-Asadi

, Klein

, Meyer

. Multiple comorbidities of 21 psychological disorders and relationships with psychosocial variables: A study of the online assessment and diagnostic system within a web-based population. J Med Internet Res, 2015; 17:e55.

29.

Australian Bureau of Statistics (ABS). Suicide statistics. Causes of death, Australia. Catalogue No. 3303.0. Australian Bureau of Statistics: Canberra. 2014.

30.

Britt

, Miller

, Henderson

, Bayram

, et al. General practice activity in Australia 2013–14. General practice series no. 36. Sydney: Sydney University Press, 2014.

31.

Fawcett

. An introduction to ROC analysis. Pattern Recognit Lett, 2006; 27:861–874.

32.

Hemelrijk

, van Ballegooijen

, Donker

, van Straten

, et al. Internet-based screening for suicidal ideation in common mental disorders. Crisis, 2012; 33:215–221.

33.

Christensen

, Cuijpers

, Reynolds

. Changing the direction of suicide prevention research. JAMA Psychiatry, 2016; 73:435.