Patient portal message characteristics and reported thoughts of self-harm and suicide: A retrospective cohort study

Abstract

Introduction

As use of electronic portal communication with healthcare teams increases, processes that effectively recognize messages that contain critical information are needed. This study aims to evaluate whether certain language and other characteristics of patient portal messages are associated with expressions of self-harm and suicidal ideation.

Methods

Using patient portal messages sent between 1 January 2013 and 30 June 2017, we searched for words and letter combinations ‘suicid’ (to identify words suicide and suicidal), ‘depress’ (for depression, depressed, depressing), ‘harm himself’ (or ‘herself ‘or ‘myself’), ‘hurt himself’ (‘herself’ or ‘myself’), ‘kill’, ‘shoot’, ‘cutting’, ‘knife’, ‘gun’, ‘overdose’, ‘over dose’ and ‘jump’.

Results

Of 831,009 messages, 11,174 messages contained one or more search terms. We manually reviewed 7,736 messages for content expressing self-harm or suicidality. Of the reviewed messages, 3.2% indicated thoughts of self-harm or suicide and 2.2% of messages suggested active suicidality. Of those expressing any thoughts of self-harm or suicide, 13.4% mentioned a specific plan, 20% were passively suicidal. Messages indicating thoughts of self-harm and suicide were more common in patients who were unmarried, non-white and younger than 18 years. Factors significantly associated with thoughts of self-harm were messages addressed to psychiatry or containing the letter combinations ‘suicide’, ‘die’, ‘depress’ and ‘harm/hurt my/her/himself’.

Discussion

Certain letter combinations and patient portal message characteristics may be associated with expressions of self-harm and suicide. These factors should be considered as we develop systems of effectively screening patient portal messages for critical clinical information.

Keywords

Online messaging patient portal health care proxy self-harm suicide personal health information health information technology secure messaging depression anxiety psychiatry telehealth

Introduction

Electronic patient portal messages are becoming an increasingly common form of communication between patients and their healthcare team. This increase is partly due to the Centers for Medicare & Medicaid Services’ (CMS) Electronic Health Record (EHR) incentive program, which mandates the ‘use of secure electronic messaging to communicate with patients on relevant health information’ as a core objective of Stage 2.^1–3 The introduction of this requirement has coincided with increased use of patient-provider electronic messaging.⁴ The electronic patient portal is used as a tool in managing chronic conditions such as diabetes,⁵ hypertension⁶ and depression.⁷ Managing depression and other psychiatric conditions via telemedicine modalities, such as video visits and internet-based therapies, has been an increasing focus of mental health care to improve patient outcomes and access to care.^8,9 Patients are generally satisfied with the care they receive via this virtual interface.¹⁰

Patients utilize the portal for a number of sensitive topics. Although prompts are typically used to remind patients of appropriate use of portal messaging, patients may still choose to disclose urgent and even emergent issues electronically. North et al. found that up to 3.5% of patient portal messages contain content detailing high-risk symptoms.¹¹ Examples include instances of expressing suicidality and a desire for self-harm. Indeed, new technology and the ease of electronic communication have created important new venues for expressing thoughts of suicide. The Crisis Text Line reports exchanging over 100 million messages since August 2013.¹² In this context, suicide mortality is increasing within almost every demographic in the United States.¹³ In light of such evidence, in 2016 The Joint Commission issued a Sentinel Event Alert on suicide, suggesting actions to improve detection of individuals with suicidal ideation.¹⁴

Developing tools to identify suicidal patients is an emerging area of interest. We know that patients tend to reach out to their health care providers around the time of suicide, with an average of 45% of suicide victims contacting their primary provider within 1 month of suicide. Rates are even higher in older individuals.¹⁵ Recent studies evaluated the use of computerized text analytics on unstructured medical records and found a correlation between specific words and the estimated risk of suicide.¹⁶ Research performed by the Crisis Text line compiles the most commonly used words in messages related to suicidal ideation, with words like ‘life’, ‘better’ and ‘people’ topping the lists.¹²

Our study aimed to evaluate whether certain language used in patient portal messages is associated with expressed thoughts of self-harm and suicidal ideation. Within these subsets of portal messages, we evaluated the overall rates of self-harm and suicidal expressions.

Methods

Setting

This study took place at Mayo Clinic Rochester. Mayo Clinic is a multi-specialty practice with over 4700 physicians and scientists. In 2017, Mayo Clinic participated in the care of 1.3 million patients. The Rochester Minnesota campus, where the study took place, has over 2400 physicians and scientists.

Message abstraction

Using a retrospective cohort design, we reviewed messages sent by patients and their proxies via a secure patient portal to their health care team that was created between 1 January 2013 and 30 June 2017, for content suggesting self-harm and suicidal ideation. Based on group consensus of potentially high-yield terms, messages were electronically searched for the following letter combinations: ‘suicid’ (to identify the words suicide and suicidal), ‘depress’ (to identify the words depression, depressed, depressing), ‘harm himself’, ‘harm herself’, ‘harm myself’, ‘hurt himself’, ‘hurt herself’, ‘hurt myself’, ‘kill’, ‘shoot’, ‘cutting’, ‘knife’, ‘gun’, ‘overdose’, ‘over dose’ and ‘jump’. For letter search results, we identified less than 1500 messages containing these letter combinations. All messages identified as containing the letter combinations were then manually reviewed to determine if they contained content related to suicide or thoughts of self-harm. For letter search results where ‘depress’ and ‘shoot’ were contained in more than 1500 messages, a random sample of 1500 messages were generated for manual review. Additionally, a random selection of 1500 messages that did not contain any of the above letter combinations was also manually reviewed for content. Messages were coded as having content reporting active thoughts of suicide (with or without a specific plan stated), passive thoughts of suicide or wanting to be dead and reports of thoughts of self-harm without intention to die. Manual review of messages for content was performed by JLP and MJD. These two reviewers independently evaluated data and addressed discrepancies through consensus decision-making.

Demographics of the patient (gender, age, race, marital status), the specialty the message was sent to and whether the message was sent during clinic hours or outside of clinic hours was recorded. In addition, we captured whether the message was sent through the patient account or via a proxy account (i.e. by a person other than the patient using proxy account login credentials).

For purposes of data analysis, the messages containing: ‘harm himself’, ‘harm herself’, ‘harm myself’, ‘hurt himself’, ‘hurt herself’ and ‘hurt myself’ were collapsed into a single category and messages containing ‘over dose’ and ‘overdose’ were combined into another single category. For the purposes of statistical analyses messages containing active or passive thoughts of suicide or thoughts of self-harm without suicidality were combined into one category.

Unadjusted analyses to assess association of the above letter combinations with messages expressing thoughts of self-harm were performed using chi-square tests. Logistic regression analysis was used for multivariate analyses using the following covariates: Age greater or less than 18 years, message sent by patient or proxy account, marital status (married/unmarried), race (white/non-white) and letter combinations that were statistically significantly associated with messages expressing thoughts of self-harm in the unadjusted analysis (‘suicid’, ‘depress’, ‘die’ and ‘hurt/harm my/her/himself’).

JMP 13.2.1 was used for statistical analysis. This study was approved by the Mayo Clinic Institutional Review Board.

Results

During the study period there were 831,009 messages from 128,202 unique patients.

Overall, 11,174 (1.3%) messages contained one or more of the search terms. The frequency of the specified letter combinations in messages is shown in Table 1.

Table 1.

Frequency of searched letter combinations in 831,009 portal messages sent via the patient portal.

Letter combination	N (%)
‘suicid’	471 (0.057)
‘depress’	5163 (0.62)
‘kill’	1012 (0.12)
‘die’	245 (0.03)
‘Hurt/harm my/her/himself’	97 (0.01)
‘shoot’	1551 (0.19)
‘jump’	1467 (0.18)
‘cutting’	1180 (0.14)
‘knife’	335 (0.04)
‘gun’	107 (0.01)
‘overdose’	88 (0.01)

Since we limited some reviews to 1500 messages as noted in the methods, our manual review consisted of 7736 messages. Of these, 172 (2.2%) had messages reporting active thoughts of suicide, with 23 of these messages (23/172 [13.4%]) mentioning a specific plan. A total of 245 (3.2%) messages expressed thoughts of suicide and self-harm with 50 of these (50/245 [20%]) reporting passive thoughts of suicide/wanting to be dead and 23 (23/245 [9.4%]) reporting thoughts of self-harm while denying suicidality.

Unadjusted analysis results are shown in Tables 2 and 3. Messages found to contain thoughts of self-harm upon manual review were significantly more common in unmarried persons, non-white race and in individuals less than 18 years of age. Messages sent by proxy account were more likely to contain thoughts of self-harm than messages sent via the patient’s portal account. Messages sent to primary care and psychiatry were more likely to contain thoughts of self-harm than messages sent to non-psychiatry specialties.

Table 2.

Demographics and other variables in messages expressing thoughts of self-harm.

	Total N	Portal message with thoughts of self-harm N (%)	P value
Age, years			<0.0001
≥18	6992	155 (2.2)
<18	744	90 (12.1)
Gender			0.25
Female	4878	163 (3.3)
Male	2858	82 (2.9)
Marital status			<0.0001
Married	4635	69 (1.5)
Unmarried	3101	176 (5.7)
Race			0.0006
White	7267	216 (3.0)
Non-white	469	29 (6.2)
Message sent by			<0.0001
Patient account	7125	170 (2.4)
Proxy account	611	75 (12.3)
Time message sent			0.83
During clinic hours	5354	168 (3.1)
Outside of clinic hours	2382	77 (3.2)
Specialty type message directed to			<0.0001
Primary care	2917	96 (3.3)
Psychiatry	439	72 (16.4)
Non-psychiatry specialty	4380	77 (1.8)

Table 3.

Letter combinations in messages expressing thoughts of self-harm.

Letter combinations	Total N	Portal message with thoughts of self-harm N (%)	P value
‘suicid’			<0.0001
Contained letter combination	471	162 (34.4)
Did not contain letter combination	7265	83 (1.1)
‘depress’			<0.0001
Contained letter combination	1730	99 (5.7)
Did not contain letter combination	6006	146 (2.4)
‘die’			0.001
Contained letter combination	245	18 (7.3)
Did not contain letter combination	7491	227 (3.0)
‘harm/harm my/him/herself’			<0.0001
Contained letter combination	97	19 (19.6)
Did not contain letter combination	7639	226 (3.0)
‘kill’			0.43
Contained letter combination	1012	28 (11.4)
Did not contain letter combination	6724	217 (3.2)
‘overdose’			0.90
Contained letter combination	88	3 (3.4)
Did not contain letter combination	7648	242 (3.2)
‘shoot’			<0.0001
Contained letter combination	1546	4 (0.3)
Did not contain letter combination	6190	241 (3.9)
‘cutting’			<0.0001
Contained letter combination	1180	17 (1.4)
Did not contain letter combination	6556	228 (3.5)
‘gun’			0.40
Contained letter combination	107	2 (1.9)
Did not contain letter combination	7629	243 (3.2)
‘jump’			<0.0001
Contained letter combination	1467	6 (0.4)
Did not contain letter combination	6269	239 (3.8)
‘knife’			0.11
Contained letter combination	335	6 (1.8)
Did not contain letter combination	7401	239 (3.2)

Messages containing the letter combinations of ‘suicid’, ‘depress’, ‘die’ and ‘hurt/harm my/her/himself’ were more likely to contain thoughts of self-harm. Messages containing the letter combinations of ‘shoot’, ‘cutting’ and ‘jump’ were significantly less likely to contain expressions of self-harm.

On adjusted analysis (Table 4), the only factors significantly associated with messages expressing thoughts of self-harm were in those aged less than 18 years and in those with a marital status of unmarried, with messages sent to psychiatry and messages that contained the letter combinations ‘suicide’, ‘die’, ‘depress’ and ‘harm/hurt my/her/himself’.

Table 4.

Adjusted odds ratio of messages expressing thoughts of self-harm, logistic regression modelling using independent variables listed.

	Odds ratio	95% CI	P value
Age<18 years	2.68	1.45–4.96	0.002
Message sent by proxy	1.76	0.93–3.33	0.08
Marital status (unmarried)	1.79	1.24–2.58	0.002
Race (non-white)	1.53	0.92–2.53	0.1
Letter combinations
‘suicid’	48.3	34.2–68.1	<0.0001
‘depress’	1.74	1.24–2.42	0.001
‘die’	9.59	5.27–17.4	<0.0001
‘hurt/harm her/my/himself’	10.26	5.26–20.2	<0.0001
Specialty type message directed to
Primary care	Reference	Reference
Psychiatry	2.93	1.96–4.40	<0.0001
Non-psychiatry specialty	0.97	0.67–1.40	0.87

A manual review of 1500 messages that did not contain any of the letter combinations did not show any messages expressing suicide, thoughts of self-harm or passive thoughts of death.

Discussion

As communication via online patient portals becomes more common, the appropriate use of this technology and tactics to manage concerns about patient safety must be considered. Our study evaluated patient portal messages and their association with thoughts of self-harm and suicidality. Indeed, 3.2% of the messages we evaluated contained potentially critical information about a patient’s desire for self-harm or suicide. This information was included despite the fact that prior to sending a portal message, a prompt informs the sender that the expected response time from the health care team is normally one to two days (excluding weekends and holidays) and recommends patients call 911 or utilize local emergency services if the concern is a medical emergency.

We identified some patient characteristics that were associated with these high-risk mental health messages. From the condensed review group, there were a statistically significant higher proportion of messages concerning suicidal content from ages less than 18 years. This is despite overall messages from those under age 18 representing only 1.6% of all messages.

The Centers for Disease Control and Prevention 2017 data on the epidemiology of suicide mortality in the United States reports that the highest overall rate of suicide is among males aged 65 years or older.^13,17 Our finding of high risk messages being associated with ages less than 18 years could be a reflection of current data that shows that the rate of suicide is increasing across all age groups and that suicide is now the second leading cause of death between the ages of 10–24 years.¹³ Additionally, based on the 2017 Youth Risk Behaviors Survey, 7.4% of youths in grades 9–12 have made at least one suicide attempt in the past 12 months.¹⁸ Given this context, our findings are likely consistent with current data on national trends.

Additionally, unadjusted analysis showed that messages sent via a proxy account were significantly associated with thoughts of self-harm. The use of a proxy account is typically used by caregivers or family members sending messages on behalf of patients, such as a parent for their child. This correlates with our finding of high-risk messages being associated with patients younger than age 18 years. However, adjusted analysis did not find a significant association between messages sent via proxy accounts and high-risk message content. However, anecdotally we noticed that there were many messages sent in using the patient’s own login credentials that discussed the patient in the third person, suggesting that someone other than the patient was logging in using the patient’s own login information and sending the message. When we reclassified those messages as being sent by a proxy, the adjusted odds ratio for messages reporting risk of self-harm went from an odds ratio of 1.76 (95% confidence interval (CI) 0.3–1.07, P = 0.08) if sent by a proxy account to an odds ratio of 10.6 (95% CI 6.33–17.66, P < 0.0001) if sent by a proxy account or via the patient’s own account referring to the patient in the third person (suggesting the message was actually sent by a proxy). This suggests that others are more likely to report concerns about patient’s thoughts of self-harm using the patient portal than the patients themselves.

Unmarried status was correlated with an increased risk of expressing thoughts of self-harm. This finding has been demonstrated in previous literature, where unmarried individuals are at higher risk of suicide compared to married individuals. Though the rates vary, this risk is increased for never-married, divorced and widowed individuals.¹⁹

Messages were also more likely to be high risk if they were addressed to psychiatry, as opposed to other specialties. Certainly, most patients with an established relationship with psychiatry have pre-existing mental health conditions that may be risk factors for self-harm and suicide. However, national data indicates that 53% of suicides are among those without known mental health conditions,²⁰ with almost half of patients who commit suicide reaching out to their primary team within the prior month.¹⁵ This finding highlights the importance of examining all message content for potential self-harm.

In terms of message content and language used, our data also found the letter combinations most significantly associated with expressions of thoughts of self-harm were ‘suicid’, ‘die’, ‘depress’ and ‘harm/hurt my/her/himself’. Language referring to ‘shoot’ and ‘cutting’ were less likely to contain expressions of thoughts for self-harm. This may be due to these messages often containing statements about ‘cutting back’ on medications or certain foods and ‘shooting’ for a certain date or target or describing ‘shooting’ pain. When we found messages containing the letter combinations ‘suicid’, ‘depress’ or ‘harm/hurt my/her/himself’ that did not appear to be expressing thoughts of self-harm, there was generally language stating that the patient was ‘not suicidal’ or ‘not depressed’ or was ‘not at risk for harming my/her/himself’.

Emerging technology such as natural language processing (NLP) may serve as a potential tool for screening the language used within patient portal messages. Recent studies have reported that NLP may be used for creating an algorithm to predict suicidal actions,^21,22 as well as screening for suicidality within the electronic medical record.²³

As exemplified by our study, NLP could screen messages and identify those containing language strongly associated with thoughts of self-harm and suicide, such as messages using the words ‘suicid’, ‘die’, ‘depress’ and ‘harm/hurt my/her/himself’. Our study showed a positive association of these letter combinations with expressions of self-harm while a random review of messages without these letter combinations did not reveal any message content suggesting thoughts of self-harm. The use of third-person language without utilisation of a proxy-account could also be identified. Screening with NLP could potentially distinguish messages where there is expression of negative intent, such as those with language expressing that the patient was ‘not suicidal’ or ‘not depressed’ or was ‘not at risk for harming my/her/himself’. Future directions could include utilising NLP for evaluating messages from suicidal patients to detect language patterns that are sensitive and specific to these high-risk patients. Another potential direction for future research would be reviewing messages sent by patients with suicide attempts or suicide completion in the weeks prior to their suicide attempt/completion for any patterns or words that could be used by NLP to classify messages as high risk.

Additional processes that could be developed to identify portal messages for content related to thoughts of self-harm and suicide may include creating risk-calculators based on patient and message characteristics. These could include factors like patient age, marital status, ethnicity, the department to which the message was addressed as well as whether a proxy was used. Based on the score, messages could be flagged as high-risk and given higher triage priority. In combination with NLP, the development of multifaceted tools could help improve the process of identifying and caring for patients at risk for self-harm and suicide. Recent studies have found that predictive models using EMR data have been able to accurately stratify patients into high-risk categories.²⁴ However, this process would still be limited by a healthcare organisation’s ability to act on this data. In addition to developing a process to accurately and promptly identify high-risk patients, organisations need to have real-time interventions, such as on-call providers, a 24/7 triage service or the use of chatbots. Requiring these interventions could create barriers to addressing high-risk messages in ways that are different than low-risk messages. Additionally, if messages were identified that indicated a high-risk for suicidal thoughts, especially if received after clinic hours, an automatic response could be sent via the patient portal notifying the patient that the message may not be reviewed or responded to immediately by the health care team. This could then include possible alternate sources of care such as contacting a suicide hotline, calling the clinic immediately (if during office hours) or accessing emergency services. Other potential future processes that could be woven into caring for patients via the online portal could include developing suicide safety planning within the patient portal,²⁵ where high risk messages could prompt a reminder of this plan, particularly for patients with a history of mood disorders.

Limitations of our study include the retrospective design, predominantly white population and reliance on manual review limiting the potential power of our study.

Another limitation is that our study population only includes those who are enrolled in and utilize the patient portal. This inherently excludes patients who are not enrolled in and do not use the portal. Studies have found differences in patients utilising the portal, with less adoption of portal use in patients who are younger, healthier, less educated, or are ethnic minorities.^26–28 In contrast, patients with chronic conditions, disabilities, high-utilizers of healthcare services and caregivers tend to show the most interest and engagement with patient portals.^29–31 We did not specifically control for these characteristics in our analysis and our sample may have been skewed by these variables.

Finally, our search terms were chosen by group consensus which may not be reflective of the most sensitive and specific terms to identify high-risk messages. Future studies could retrospectively review messages sent by patients prior to carrying out a documented act of self-harm and use NLP to identify the most commonly used terms in these individuals.

In conclusion, the increasing use of electronic patient portals allows access to patient-to-provider messaging which may include messages with potentially urgent critical clinical information such as thoughts of suicidality and self-harm. Technological advances are needed to promptly identify this information to avoid potentially devastating delays in treatment. In our study we found patient and portal message characteristics, as well as the use of certain language, can help identify high-risk messages to expedite their review. This information can be used to build electronic tools to improve patient care and safety.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship and/or publication of this article.

References

Centers for Medicare & Medicaid Services. EHR incentive programs in 2015 through 2017 patient electronic access, https://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/Downloads/2016_PatientElectronicAccess.pdf (2016, accessed 7 August 2019).

National Learning Consortium. How to optimize patient portals for patient engagement and meet meaningful use requirements, https://www.healthit.gov/sites/default/files/nlc_how_to_optimizepatientportals_for_patientengagement.pdf (2013, accessed 7 August 2019).

Centers for Medicare & Medicaid Services. Stage 2 Eligible professional meaningful use core measures measure 17 of 17, https://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/downloads/Stage2_EPCore_17_UseSecureElectronicMessaging.pdf (2012, accessed 7 August 2019).

Dexter

Fields

Rdesinski

, et al. Patient-provider communication: Does electronic messaging reduce incoming telephone calls? J Am Board Fam Med 2016; 29(5): 613–619.

Devkota

Salas

Sayavong

, et al. Use of an online patient portal and glucose control in primary care patients with diabetes. Popul Health Manag 2016; 19(2): 125–131.

Zhou

Kanter

Wang

, et al. Improved quality at Kaiser permanente through e-mail between physicians and patients. Health Affair 2010; 29 (7): 1370–1375.

Pecina

North

Williams

, et al. Use of an on-line patient portal in a depression collaborative care management program. J Affect Disord 2017; 208: 1–5.

Deslich

Stec

Tomblin

, et al. Telepsychiatry in the 21st century: Transforming healthcare with technology. Perspect Health Inf Manag 2013; 10: 1f.

Garcia-Lizana

Munoz-Mayorga

What about telepsychiatry? A systematic review.

Prim Care Companion J Clin Psychiatry 2010; 12(2): PCC.09m00831.

10.

Goldzweig

Orshansky

Paige

, et al. Electronic patient portals: Evidence on health outcomes, satisfaction, efficiency, and attitudes: A systematic review. Ann Intern Med 2013; 159 (10): 677–687.

11.

North

Crane

Stroebel

, et al. Patient-generated secure messages and eVisits on a patient portal: Are patients at risk? J Am Med Inform Assoc 2013; 20(6): 1143–1149.

12.

Crisis Text Line. Crisis Trends, www.crisistrends.org (2018, accessed 8 May 2019).

13.

Hedegaard

Curtin

Warner

Suicide mortality in the United States, 1999–2017. NCHS Data Brief no. 330. Hyattsville, MD: National Center for Health Statistics, 2018.

14.

The Joint Commission. Sentinel event alert. Detecting and treating suicide ideation in all settings. Issue no. 56, 24 February 2016. Oak Brook (IL), Joint Commission.

15.

Luoma

Martin

Pearson

Contact with mental health and primary care providers before suicide: A review of the evidence.

Am J Psychiatry 2002; 159(6): 909–916.

16.

Poulin

Shiner

Thompson

, et al. Predicting the risk of suicide by analyzing the text of clinical notes. PLoS ONE 2014; 9(1): e85733.

17.

National Institute of Health Mental. Health information statistics, suicide, https://www.nimh.nih.gov/health/statistics/suicide.shtml (2019, accessed 10 April 2019).

18.

Kann

McManus

Harris

, et al. [Centers for Disease Control and Prevention]. Youth Risk Behavior Surveillance – United States, 2017. MMWR Surveill Summ 2018; 67(8): 1–114.

19.

Kposowa

Marital status and suicide in the National Longitudinal Mortality Study.

J Epidemiol Community Health 2000; 54:254–261.

20.

Centers for Disease Control and Prevention. National Vital Statistics System, Vital Signs 2018, https://www.cdc.gov/vitalsigns/suicide/infographic.html (2018, accessed 7 Aug 2019).

21.

Cook

Progovac

Chen

, et al. Novel use of natural language processing (NLP) to predict suicidal ideation and psychiatric symptoms in a text-based mental health intervention in Madrid. Comput Math Method M 2016; 2016: 8708434.

22.

Simon

Johnson

Lawrence

, et al. Predicting suicide attempts and suicide deaths following outpatient visits using electronic health records. Am J Psychiatry 2018; 175: 10.

23.

Downs

Velupillai

George

, et al. Detection of suicidality in adolescents with autism spectrum disorders: Developing a natural language processing approach for use in electronic health records. AMIA Annu Symp Proc 2018; 2017: 641–649.

24.

Cronin

Fabbri

Denny

, et al. A comparison of rule-based and machine learning approaches for classifying patient portal messages. Int J Med Inform 2017; 105: 110–120.

25.

Little

Neufeld

Cole

Integrating safety plans for suicidal patients into patient portals: Challenges and opportunities.

Psychiatr Serv 2018; 69: 618–619.

26.

Ketterer

West

Sanders

Correlates of patient portal enrollment and activation in primary care pediatrics.

Acad Pediatr 2013; 13(3): 264–271.

27.

Lober

Zierler

Herbaugh

, et al. Barriers to the use of a personal health record by an elderly population. AMIA Annu Symp Proc 2006; 2006: 514–551.

28.

Goel

Brown

Williams

, et al. Patient reported barriers to enrolling in a patient portal. Patient reported barriers to enrolling in a patient portal. J Am Med Inform Assoc 2011; 18 Suppl 1(Suppl 1): i8–i12.

29.

Sanders

Winters

Fortuna

, et al. Internet access and patient portal readiness among patients in a group of inner-city safety-net practices. JACM 2013; 36(3): 251–259.

30.

Krist

Woolf

Bello

, et al. Engaging primary care patients to use a patient-centered personal health record. Ann Fam Med 2014; 12(5): 418–442.

31.

Taha

Czaja

Sharit

, et al. Factors affecting usage of a personal health record (PHR) to manage health. Psychol Aging 2013; 28(4): 1124–1139.