Social Listening as a Rapid Approach to Collecting and Analyzing COVID-19 Symptoms and Disease Natural Histories Reported by Large Numbers of Individuals

Abstract

Given the severe and rapid impact of COVID-19, the pace of information sharing has been accelerated. However, traditional methods of disseminating and digesting medical information can be time-consuming and cumbersome. In a pilot study, the authors used social listening to quickly extract information from social media channels to explore what people with COVID-19 are talking about regarding symptoms and disease progression. The goal was to determine whether, by amplifying patient voices, new information could be identified that might have been missed through other sources. Two data sets from social media groups of people with or presumed to have COVID-19 were analyzed: a Facebook group poll, and conversation data from a Reddit group including detailed disease natural history-like posts. Content analysis and a customized analytics engine that incorporates machine learning and natural language processing were used to quickly identify symptoms mentioned. Key findings include more than 20 symptoms in the data sets that were not listed in online lists of symptoms from 4 respected medical information sources. The disease natural history-like posts revealed that people can experience symptoms for many weeks and that some symptoms change over time. This study demonstrates that social media can offer novel insights into patient experiences as a source of real-world data. This inductive research approach can quickly generate descriptive information that can be used to develop hypotheses and new research questions. Also, the method allows rapid assessments of large numbers of social media conversations that could be applied to monitor public health for emerging and rapidly spreading diseases such as COVID-19.

Introduction

Coronavirus disease 2019 (COVID-19) results from infection with the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), the cause of the current ongoing pandemic. The novel coronavirus initially manifested in Wuhan, China, in late 2019 and in merely 6 months resulted in more than 13 million cases and 580,000 deaths worldwide.¹

Given the severe nature and rapid impact of this disease, the pace of research and information sharing has been accelerated in a variety of ways, including increased speed of research publications by reducing turnaround time for peer review and posting pre peer-reviewed papers.² The result is that more than 32,000 publications related to COVID-19 have been generated in these past 6 months, according to PubMed. Although it is beneficial to disseminate this information quickly, culling information from this large publication volume can be difficult and time-consuming.³

There is a tremendous amount of information relating to symptoms at the onset of disease, and disease symptoms, treatments, and management for hospitalized patients. Responses of individual patients can vary widely, from asymptomatic or mild disease to multi-organ failure and death.⁴ Accumulating evidence suggests that COVID-19 can impact many bodily systems that encompass respiratory,⁵ immunologic,⁴ gastrointestinal,⁶ ocular,⁷ urologic,⁸ hepatic,⁹ dermatologic,¹⁰ neurologic,¹¹ and psychiatric¹² effects. To help people determine whether they might have COVID-19, leading health information providers list symptoms of COVID-19 online. For example, the Centers for Disease Control and Prevention (CDC) lists 16 symptoms¹³ and the World Health Organization (WHO) lists 14 symptoms.¹⁴ The Mayo Clinic¹⁵ and WebMD¹⁶ also provide lists online. Nine symptoms on all 4 lists, as of July 13, 2020, include fever, cough, difficulty breathing, fatigue, body aches, headache, sore throat, diarrhea, and loss of taste or smell.

There is less information about the full course of the disease, including pre and post hospitalization, with recovery spanning months.¹⁷ Also, it is plausible that the course of the disease for people who never see a doctor might be different, and there are few if any resources to learn of their experiences. Details about impacts and experiences with COVID-19 in daily life pre and post hospitalization might not be communicated to clinicians or captured in publications.

The research team used social listening methods,^18,19 which were developed to extract information from social media channels, to explore symptoms and disease progression of COVID-19. The team analyzed data from 2 social media communities: conversation data from a public Reddit group, including natural history-like posts detailing the time line of the disease, and a closed Facebook group poll. Several approaches were used to quickly identify disease symptoms being discussed in social media conversations. The goal of this pilot study was to determine whether, by analyzing voices of patients from social media posts, new information could be identified that might have been missed by other methods of collecting information about patient experiences.

Methods

Reddit COVID19-positive subreddit conversation text analysis

TREND Community, using the Reddit API, accessed social media conversation data from the Reddit group (subreddit) called “COVID19-Positive”²⁰ that self-describes as “a place for people who came back positive for COVID19 can share your stories, experiences, answer questions and vent!” At the time the conversations analyzed for this study were downloaded, the group had 67,000 members from all over the world, many in the United States. The data set includes conversations posted during March 14–June 23, 2020. The subreddit is publicly available.

Full conversation data set analysis

The full conversation subreddit data set was analyzed using the TREND Community analytics engine named “Krystie,” which uses supervised learning and natural language processing techniques to identify disease burden and disease management concepts. The primary tool for identification of disease symptoms is an ever-expanding taxonomy of terms used in patient conversations on social media in communities for rare and chronic diseases. The taxonomy has been developed over the past 2 years as TREND listens to these communities and learns how they describe and manage their symptoms. There is no demographic information for this group because Reddit does not collect or make public any information on demographics. The subreddit allows for tags or flairs to be added to each post, and in this subreddit those tags let others know if the author tested positive for the virus or is presumed positive by a doctor.

Subpopulation analysis of time line posts

This rich data set contains 81 posts that are similar to natural histories in which the author gave a detailed time line of the symptoms and the course of the disease. These were found using keyword searches for “timeline,” “day 1,” or “week 1.” The research team considers these posts to be patient-authored natural histories of their own disease. Analysts manually labeled text (words or phrases) as falling into one of 8 predetermined categories: age, gender, week 1 symptoms, week 2 symptoms, week 3+ symptoms (symptoms described during week 3 and beyond), start date, end date, and geographic location. Within the posts, some authors identified their age, gender, and geographic location, which are summarized in Table 1. The analysts used Sagemaker's Grounded Truth to label the data.

Table 1.

Demographics and Characteristics of the COVID19-Positive Subreddit Subgroup for Natural History Analysis¹

Characteristic	Specific descriptor	Total number of posts
Gender	Mentioned gender	47²
	Male	27
	Female	22
Age	Mentioned age	32
	Teens	1
	20s	11
	30s	15
	40s	3
	50s	3
	60+	2
Geography	Mentioned location	26
	New York City	4
	Northeastern US	2
	Southeastern US	3
	Midwest	4
	West	5
	Brazil	1
	Canada	1
	Poland	1
	Russia	1
	Southeast Asia	1
Disease duration	Mentioned duration	64
	1–6 days	3
	7–13 days	12
	14–20 days	14
	21–27 days	3
	28+ days	32
Symptom start	Mentioned start date	56
	Before February	1
	February	4
	March 1–15	21
	March 16–31	21
	April	7
	May	2
Positivity	Mentioned test status	77
	Tested positive	43
	Presumed positive by doctor	15

The subreddit group (n = 81 posters) was identified as posters whose posts contain longitudinal information about the course of their disease.

The number of genders specified is greater than the number of authors who mentioned gender because some posts mentioned more than 1 gender.

The list of terms for week 1 symptoms, week 2 symptoms, and week 3+ symptoms were reanalyzed by the analytics engine Krystie, which classified the symptom concepts. In some cases, concepts included several different symptom terms found in the text. For example, “fever” and “temperature 101.5” are classified as fever and counted together to get the total number of mentions for fever. Examples of concepts are shown in Supplementary Table S1.

Facebook poll of symptoms

The founder and administrators of a private Facebook group called COVID-19 Support Group,²¹ which welcomes people who have, have had, or are presumed to have or have had COVID-19, approved use of data from a poll that was conducted within the online group. The administrators started the poll on May 25, 2020 using a list of symptoms named by patients from a previous poll, where participants could describe symptoms not listed in the comments section. For the current poll, administrators added any symptom to the list that had ≥10 positive responses in the first poll and locked the list. Eligibility for the group included confirmed or presumptively positive people. Data were downloaded from the poll on July 13, 2020. There were 5691 members when the poll started and 15,208 members in the group when the poll concluded. Although Facebook provides some demographic information about group members, the demographic breakdown for those who filled out the poll was not available.

COVID-19 symptom lists from online resources

To identify commonly used online resources for COVID-19 symptoms, the research team conducted a Google search on July 13, 2020, and identified top results to compare with the pilot study data. The 4 leading resources were the CDC,¹³ WHO,¹⁴ WebMD,¹⁶ and Mayo Clinic¹⁵ websites, and their online web page symptom lists at the time of this analysis are listed in Supplementary Table S2.

Analytical considerations

Data were collected from conversations posted by individuals on social media and thus might not be representative of all people living with COVID-19. By virtue of being able to access social media, the method favors literate people with the means, time, interest, and ability to participate in conversations. Although 3 of the natural history-like posts were written by family members of people who were too sick to keep track themselves, most posts were by people who were able to keep track of their own illness and describe it.

The team is limited by the assumption that the people who responded to the Facebook poll or posted on Reddit honestly represent their experience of the illness, their symptoms, their diagnoses, and disease progression. It is possible that people could be misrepresenting themselves, the illness, and their situation for a variety of reasons, including a possible Hawthorne effect.²² However, the large size of the data sets for the Reddit conversation data and the Facebook poll should trend to data validity and will provide precision, yet this will not correct for potential bias (inaccuracy). In the 81 natural histories, we could be seeing some descriptions of an illness that is not COVID-19, but no descriptions stood out as substantially different from the rest.

Inter-analyst reliability for the labeling of the natural history data set was calculated by comparing a random set of the analyzed data from the Reddit subset. Agreement was found between coders 85% of the time, with a Cohen's kappa value of 0.62. All disagreements seen were from items not being labeled by one or the other of the analyzers. Given this same set of data, the research team is confident that the measures would be repeated if performed again. The team notes that, because of the shifting nature of information about COVID-19, both within the medical community and the patient community, the way people talk about the illness and the symptoms has shifted even as this paper was being written. Symptom lists provided by the CDC,¹³ for example, have changed over time as more became known about the illness. Thus, a similar comparison of patient lists of symptoms to online resources' lists of symptoms in 2 months might show more cohesion than the current lists.

The study was conducted with an exemption under 45 CFR § 46.104(d)(4) by the Western Institutional Review Board.

Results

Full data set analysis of COVID-19-positive subreddit

Conversations on the COVID-19-Positive subreddit were analyzed using the analytics engine Krystie, which matches terms in the conversation with concepts in the taxonomy to identify topics of conversation. This analysis focused on the symptoms mentioned in the conversations. Krystie identified counts of mentions of symptoms in the full conversation data set. The top 15 symptoms are listed in Table 2, and the full list of symptoms is displayed in Supplementary Table S3. These data reveal what the authors are talking about as descriptive results. It is important to note that, currently, the analytics engine only notes that a term was mentioned and does not provide context for the mention, such as sentiment. With more than 14,000 mentions of some terms, the research team cannot determine the context in which each was mentioned, but assumes that large numbers of people mentioning the same term suggests that it is worth further study. The team also is unable to assign mentions to specific authors and thus cannot discern whether a term is mentioned once by multiple authors or mentioned many times by a few authors. Most likely it is a combination of both.

Table 2.

Top 15 Symptoms Mentioned in the Subreddit Full Conversation Data Set and the Disease Natural History Subpopulation¹

Rank order	Full conversation data set		Disease natural history subpopulation
Rank order	Symptoms	Number of mentions	Symptoms	Number of mentions
1	Cough	14,870	Fever	68
2	Fever	14,673	Fatigue	51
3	Pain/ache	13,037	Cough	51
4	Fatigue	8695	Shortness of breath	48
5	Gastrointestinal issues	7035	Anxiety	48
6	Anxiety/panic	6729	Headache	45
7	Shortness of breath	6066	Lung issues	44
8	Headache	5903	Chest pain	44
9	Breathing	5655	Sore throat	41
10	Worry	3807	Difficulty breathing	41
11	Sore throat	3677	Body aches	40
12	Heart rate/tachycardia	3181	Gastrointestinal issues	30
13	Burning	1869	Congestion	29
14	Chest pain	1857	Anosmia	28
15	Loss of smell/taste	1353	Eye issues	28

These lists show the top 15 symptoms in the 2 subreddit data sets (full subreddit conversations and natural disease history subpopulation conversations) in order of number of mentions of the symptoms. The count represents the number of times the term was mentioned in the text. The full lists of symptoms are included in Supplementary Tables S3 and S4.

Natural disease history analysis of COVID-19-positive subreddit

Using the 81 subreddit natural history-like posts, the team found patterns that emerged from the posts and presents major findings about symptoms and symptom changes over time (Figures 1 and 2). The list of the top 15 most frequently mentioned symptoms is shown in Table 2, and the full list of symptoms is included in Supplementary Table S4.

FIG. 1.

Symptoms that increase over time in patient-authored natural disease histories.¹

FIG. 2.

Symptoms that decrease over time in patient-authored natural disease histories.¹

The 81 natural history-like posts typically would start with a paragraph about the person writing and their situation. Some mentioned their gender, age, smoking status, health conditions, exercise schedule, and other information. The limited demographic information gleaned from author descriptions is shown in Table 1. Some authors described how they think they caught the virus. For example, a post might say: “29 F, non-smoker, marathoner. My boyfriend is a nurse and I think I caught the virus from him, although he did not appear sick.” A few of the posts detailed the symptoms of multiple people in a family. Interestingly, a large proportion of authors reported having symptoms for at least 4 weeks, with the longest reported as 90 days. There was no pattern relating the length of the illness to age. Of the 27 people younger than age 40 who listed their age, 8 did not mention week 3+ symptoms and 19 did.

After introductory information, the typical post would then provide a time line with dates or day numbers (eg, “day 1”), the symptoms experienced, and actions taken on that day. For example, a history might start like this: “March 9: Woke up with a scratchy throat, thought it was allergies and went about my day. At 4 p.m. I was exhausted and hot, realized I had a fever.” In most posts, people mentioned their symptoms and the day on which they experienced those symptoms.

Many of the symptoms changed over the progression of the illness, with 13 symptoms increasing and 10 symptoms decreasing over time (Figures 1 and 2; Supplementary Table S5). Three symptoms remained relatively constant (shortness of breath, no appetite, and fever; Supplementary Table S5). The full count data for all symptoms are provided in Supplementary Table S5.

These analyses present anecdotal, qualitative data regarding the symptoms that the authors were talking about in the context of their disease experiences over time. It is important to note that the data reflect author conversations and the symptoms they choose and remember to report on social media, and limitations of the data should be considered when interpreting results (see the Discussion). For example, some people might have mentioned only new symptoms rather than repeating all symptoms they felt each day.

COVID-19 Facebook group poll results

TREND Community also analyzed another data source to learn about patient experiences with COVID-19. The administrative team of the Facebook COVID-19 Support Group first created a poll for which participants were asked to list their COVID-19 symptoms. One of the administrators has COVID-19 and felt that the symptoms publicized were not inclusive of symptoms that patients were experiencing, so she started the poll. The poll methodology and group characteristics are described in Methods. They used the symptoms list from the first poll to create a second poll with locked options. When poll data were pulled on July 13, 2020, there were 99 descriptors of patient experiences listed in the poll (the full list is included in Supplementary Table S6). The numbers of respondents per symptom ranged from 5 for “I have been on a ventilator” to 2803 for “fatigue, lethargy, weakness.” The poll included the descriptors “symptomatic at 30 days,” “symptomatic at 60 days,” and “symptomatic at 90 days” that are not related to specific symptoms.

Comparison of social listening results with online symptom information resources

The research team compared the top 40 symptoms in the Facebook poll results and corresponding subreddit conversation data to symptoms listed on leading resource information web pages (Tables 3 and 4). The team selected the top 40 symptoms so that “high temperature,” which does not appear in poll results until item 39, could be included. There is variability among the 6 lists. The specific symptoms that are listed on all of them include sore throat, cough/dry cough, and fever/high temperature. Grouping related terms, all lists named some type of fatigue, though the common specific symptom of “talking causes exhaustion” in the poll was not listed on any of the web pages. This phrase refers to authors feeling exhausted even from just talking, based on reading posts that name the symptom. All lists named something related to body aches and pains, and the Facebook poll was highly specific in describing body locations of pains. The Mayo Clinic description of “muscle aches” did not capture pains described in other areas of the body. The WHO and WebMD description of “dry cough” was narrower than the descriptor “cough” in other lists. Given descriptions of congestion and mucus in the symptom lists, it is possible that some coughs were not dry.

Table 3.

Comparison of Symptoms Named in COVID-19 Facebook Poll¹ with Subreddit Data Set and Lists from Other COVID-19 Information Resources²

Ranking in Facebook poll¹	Top 40 descriptors in Facebook poll¹	Symptoms in subreddit analysis	CDC symptom list³	WHO symptom list⁴	Mayo Clinic list⁵	WebMD list⁶
1	Fatigue, lethargy, weakness	Fatigue, tired, weakness, exhaustion	Fatigue	Tiredness	Tiredness	Fatigue
2	Relapse (feel good for a couple days then fall ill again)	Relapses	n/a	n/a	n/a	n/a
3	Chest pain/tightness/heaviness	Chest pain	Chest pain or pressure	Chest pain or pressure	Chest pain	Constant pain or pressure in your chest
4	Headaches/migraines	Headache	Headache	Headache	Headache	Headache
5	GI issues/diarrhea, constipation, flatulence	GI issues-diarrhea	Diarrhea	Diarrhea	Diarrhea	Diarrhea
6	Respiratory distress: short of breath, rapid, difficulty, shallow, labored breathing	Shortness of breath, breathing	Difficulty breathing or shortness of breath	Difficulty breathing or shortness of breath	Shortness of breath or difficulty breathing	Shortness of breath, trouble breathing
7	Emotional distress	Anxiety, worry, stress	n/a	n/a	n/a	n/a
8	Confusion/neurological issues/brain fog/forgetfulness	Forgetting/confusing	New confusion	Loss of speech or movement	n/a	Sudden confusion
9	Sore throat	Sore throat	Sore throat	Sore throat	Sore throat	Sore throat
10	High heart rate/pulse rate, palpitations	Heart rate/Tachycardia	n/a	n/a	n/a	n/a
11	Chills	Chills	Chills	n/a	Chills	Chills
12	Cough	Cough	Cough	Dry Cough	Cough	Dry cough
13	Talking causes exhaustion	n/a	n/a	n/a	n/a	n/a
14	Loss of smell/taste	Loss of smell/taste	New loss of taste or smell	Loss of taste or smell	Loss of taste or smell	Loss of smell or taste
15	Joint pain	Joint pain	Muscle or body aches	Aches and pains	n/a	Body aches
16	Eye pain (strained eyesight, watery eyes, blurry vision, burning, crustiness)	n/a	n/a	Conjunctivitis	n/a	n/a
17	Sinus pressure	Sinus infection	Congestion	n/a	n/a	Congestion
18	No appetite	n/a	n/a	n/a	n/a	Loss of appetite
19	Dizziness	Dizziness	n/a	n/a	n/a	n/a
20	Symptomatic 60 days	n/a	n/a	n/a	n/a	n/a
21	Insomnia	Sleeplessness	n/a	n/a	n/a	n/a
22	Tingling body sensation	n/a	n/a	n/a	n/a	n/a
23	Lower back pain	Pain	Muscle or body aches	Aches and pains	Muscle aches	Body aches
24	Nausea/vomiting	GI issues-nausea, vomiting	Nausea or vomiting	n/a	Nausea or vomiting	Nausea or vomiting
25	Congestion	Congestion	Congestion	n/a	n/a	Congestion
26	Change in taste/smell⁷	n/a	n/a	n/a	n/a	n/a
27	Muscle or body aches	Pain/aches	Muscle or body aches	Aches and pains	Muscle aches	Body aches
28	Feeling shaky/internal tremors	Shivering	n/a	n/a	n/a	Shaking
29	Heavy sweating	Sweating	n/a	n/a	n/a	n/a
30	Lungs, heart, liver pain	Chest pain/Heart attack	Persistent pain or pressure in the chest	Chest pain or pressure	Chest pain	Constant pain or pressure in your chest
31	Extreme temperature fluctuations	n/a	n/a	n/a	n/a	n/a
32	Stiff neck	Stiff	n/a	n/a	n/a	n/a
33	Abdominal pain	Abdominal pain	n/a	n/a	n/a	n/a
34	Burning sensation in lungs	Burning	n/a	n/a	n/a	n/a
35	Symptoms in waves	n/a	n/a	n/a	n/a	n/a
36	Symptomatic 30 days	n/a	n/a	n/a	n/a	n/a
37	Pin and needle feeling in feet, hands, toes	n/a	n/a	n/a	n/a	n/a
38	Low temperature (below 99 degrees)	n/a	n/a	n/a	n/a	n/a
39	High temperature	Fever	Fever	Fever	Fever	Fever
40	Dehydration	Dehydration	n/a	n/a	n/a	n/a
	Not listed in poll	Runny nose, stroke, balance, fainting	Runny nose	Runny nose, discoloration of fingers or toes	Runny nose	Runny nose, mucus or phlegm, bluish lips or face

Data were pulled from a poll conducted in the COVID19-Positive Support Group as described in Methods. The top 38 most frequently tagged symptoms are listed out of a total of 78 symptoms in the poll, and they are listed in order of decreasing number of respondents.

Other resources included the CDC and WHO as leading health organizations, and 2 other top Google search results after the CDC from a Google search for “COVID-19 symptoms,” excluding blogs. The online symptom lists were pulled on July 13, 2020 (see the appendix).

Symptoms listed on the CDC Symptoms of Coronavirus web page¹³ as of July 13, 2020.

Symptoms listed on the WHO Coronavirus Symptoms web page¹⁴ as of July 13, 2020.

Symptoms listed on the Mayo Clinic Coronavirus Disease 2019 (COVID-19) web page¹⁵ as of July 13, 2020.

Symptoms listed in the WebMD Signs of Coronavirus (COVID-19) web page¹⁶ as of July 13, 2020.

Based on context, “change in smell” was interpreted as being able to smell but smelling was off or unusual smells were noticed.

CDC, Centers for Disease Control and Prevention; GI, gastrointestinal; n/a, not applicable (because not listed in the indicated online resource); WHO, World Health Organization.

Table 4.

Patient Voice Terms Absent from Online Symptom Lists¹

Patient Voice Terms refers to symptoms described in the Facebook poll or the subreddit group conversation. The table includes terms for which at least 1 online resource was missing the term. “Yes” indicates the same symptoms are described. “Absent” refers to a lack of any similar or related term in the online list. “Related” refers to terms that are similar or affect related areas (eg, feet, toes) but are different symptoms.

The symptoms are from online lists provided by the CDC,¹³ WHO,¹⁴ Mayo Clinic,¹⁵ and WebMD¹⁶ websites as of July 13, 2020.

CDC, Centers for Disease Control and Prevention; n/a, not applicable (because not listed in the indicated online resource); WHO, World Health Organization

There were symptoms on the online web page lists that were not included in the top 40 results of the poll, as shown in the last row of Table 3. Runny nose, mucus or phlegm, and bluish lips or face were not in the poll. In total, there were 22 symptoms listed in the poll that were mentioned in at least 1 online list as a direct or related match. In contrast, there were 59 symptoms from the Reddit full conversation data and 80 symptoms mentioned in the Facebook poll that did not appear on any of the 4 online resource symptom lists. Table 4 shows a summary of terms for which at least 1 online site was lacking a patient voice term. For terms unique to the patient conversations, lists of the top 25 most frequently mentioned symptoms and mentions from both conversation data sets are shown in Supplementary Table S7. These comparisons show that people are talking about symptoms that are not listed as key symptoms on publicly posted information sites. Given the limitations of the data sets and the descriptive nature of the study approach, the research team cannot make any suggestions as to whether these additional symptoms should be included on those lists.

Discussion

This pilot study was designed to assess whether social listening might be a valuable tool for understanding new and emerging diseases such as COVID-19, and as a rapid method to query tens of thousands of people. The aim was to determine whether people with COVID-19 are talking about symptoms that are not listed by major health information resources.^13

–16 Two data sets from social media groups of people with or presumed to have COVID-19 were evaluated: a poll conducted in a Facebook group, and conversation data from a subreddit including time line conversation data from a subset of 81 individuals to develop natural history-like descriptions of patient experiences. Both data sets contained many symptoms that were not listed in the CDC, WHO, Mayo Clinic, or WebMD lists of symptoms. Furthermore, using natural history-like posts in the conversation data, symptoms were identified that increased or decreased in mention frequencies from week 1 through weeks 3 or more of patient experiences (Figures 1 and 2).

Other groups have used different methodologies to extract information about large numbers of people infected with COVID-19. Two studies stand out as interesting examples for comparison. The first study performed a systematic review of published studies that reported drug therapies for COVID-19 patients.³ The authors screened 2706 publications and identified 155 published studies that met their inclusion criteria. Two individuals read all 155 studies and manually extracted information about administered drugs into spreadsheets, and a third person reviewed the extracted data to resolve discrepancies. They identified 115 drugs that were administered to the first 9152 reported COVID-19 cases. The second study recruited 30 medical students, who trained for hours to extract data from electronic medical records from the first 1000 cases of confirmed COVID-19-positive patients in New York City.⁵ They manually culled details about demographics, highest level of care (eg, intensive care), comorbidities, drug treatments, and symptoms.

The research team highlights these studies to contrast the level of effort and time commitment needed for these methodologies compared to social listening, in which the team gathered information from tens of thousands of people with substantially less time and effort. Each approach has strengths and weaknesses, and each is designed to gather different types of information. The strength of social listening, when compared to these 2 studies, is that within 2 weeks, a substantial amount of qualitative information can be garnered from large populations. The qualitative information can be used to provide snapshots of current situations and to serve as a foundation to build hypotheses and further quantitative analyses. Social listening amplifies voices of people experiencing diseases such as COVID-19 by producing qualitative data that may inform future research. Some of the information might otherwise be missed (eg, symptom information from people who have not seen a doctor). Social listening is hampered by problems with representing the entire population. For example, participants are not randomly selected from the patient population and the terms mentioned are not taken in context so the symptoms might not be tied to the disease. The approach is qualitative in revealing what is out there, or the possible, to facilitate new hypotheses in preparing for more rigorous research that otherwise might not be conducted.

TREND Community has successfully applied the technology in other arenas, including rare disease populations and pharmaceutical applications. For example, TREND Community brought attention to sleep issues among children with Prader-Willi Syndrome, a rare disorder affecting approximately 1 in 15,000 to 30,000 births, by quantifying caregiver mentions of various concerns in social media conversations.²³ That information subsequently led to identifying new treatments for narcolepsy and cataplexy-like symptoms, leading to improvements in children's lives.^24
–26 Also, TREND Community used social listening to identify drug therapies, under-recognized disease symptoms, and unmet needs for numerous rare disease communities. Results have been shared with a variety of stakeholders, including pharmaceutical companies and appreciative patient populations.

In the case of COVID-19, the situation is at the other end of the spectrum from rare diseases. The exponential rise in infections has led to more than 3 million cases in the United States alone.²⁷ People want to learn about symptoms to determine if they have the disease and need to go to the hospital. Early detection is important to contain and slow transmission of the illness. Accumulated evidence shows that SARS-CoV-2 infection impacts nearly every, if not every, bodily system.⁴ Furthermore, the list of associated symptoms has been growing. For example, loss of taste or smell was recognized as an early symptom after the accumulation of reports of the symptom in infected individuals.²⁸

Using social listening, the research team inquired as to whether some symptoms are being overlooked. In this pilot study, the team compared the results of that analysis with lists of symptoms provided by 4 sources of information: the CDC,¹³ WHO,¹⁴ Mayo Clinic,¹⁵ and WebMD.¹⁶ That comparison revealed that some symptoms were absent from these lists, including several terms in the Facebook poll (Table 4). Symptoms that stood out as unusual were high heart rate, talking causes exhaustion, emotional distress, anxiety, and neurological symptoms such as dizziness and tingling. It is possible that the emotional distress and anxiety may be symptoms related to other neurological symptoms that may warrant special attention from physicians. In summary, this pilot study identified many symptoms that patients are discussing that were not in the 4 online symptom lists. Further hypothesis-driven research can use that information to explore whether those symptoms are important for early diagnosis or treatment of COVID-19.

The Facebook poll provided qualitative information about the range of symptoms that people had been experiencing. One strength of the poll is that the descriptors were created by people experiencing COVID-19, drawn from an initial poll that collected symptoms from patient input. For the poll results analyzed for the present study, the symptom list was locked so that all respondents could choose from all symptoms and thus the ranking order by number of responses can be informative. For example, although fever features prominently as a symptom for identifying people with COVID-19 and businesses rely on temperature checks, fever was ranked 39th in the list of responses in the poll. One shortcoming is that, with no demographic information, the research team cannot account for possible selection bias in the population who responded (eg, it may be skewed toward people who are sicker or more willing to interact on social media than the general population). Another shortcoming of the poll is that the process of curating new symptoms was labor intensive, especially for the administrators who themselves are battling the disease. In spite of shortcomings, the list provides qualitative value in that it provides an accumulated list of symptoms experienced by participants, yielding a list of 99 symptoms, 59 of which were not listed by the 4 other resources.

Likewise, the subreddit analyses provide useful insights into symptoms and experiences of people with COVID-19. The patient-authored disease natural histories show that some symptoms increase and some decrease over the course of the disease. The symptoms that are more common early on might be more important to include in lists of symptoms, as people often will search online to determine if they have the disease. The more common symptoms in the present analyses were chest pain, anxiety, fatigue, cough, sore throat, headache, and body ache, which completely overlap the online medical reference site lists. It would be interesting to probe more deeply into the time line conversation to tease out whether the same individuals who are experiencing decreases in one symptom, such as cough, are also experiencing increases in other symptoms, such as difficulty breathing. Unfortunately, there are limited demographic data about the authors of these posts. It is possible that the trends may represent 2 different subsets of patients – namely, ones with mild-to-moderate disease (eg, the decreasing common symptoms of cough and loss of smell) as opposed to the potentially more ominous symptoms of “difficulty breathing,” “lung issues,” and “chest pain” that trend upward over the first 3 weeks of illness. Alternatively, the symptoms that are declining may in fact be going away as the other symptoms emerge in either real or perceived importance across the entire population and represent the broader natural history of COVID-19 in general.

There are important limitations to consider when interpreting the results. In addition to constraints related to limited demographic information, the team assumes that the authors of the posts are honestly representing their disease experience. It is possible that people could be misrepresenting themselves, the illness, and their situation for a variety of reasons, including a possible Hawthorne effect.²² Although the large size of the data sets for the Reddit conversations and the Facebook poll provide precise information, they do not overcome bias (inaccuracy) considerations. The information does not support rigorous statistical comparisons. Also, mentions are counted without context, introducing uncertainty in the interpretation. Given the limitations, the research team emphasizes that this study is an exploratory descriptive study, not a confirmatory quantitative study. These data may provide new information that can generate hypotheses for further quantitative and controlled research designs.

Remarkably, 32 of the 81 subgroup authors reported having symptoms for 4 weeks or longer. These results suggest COVID-19 may linger with patients for weeks or months. Many mentioned the symptoms coming in waves, such that they might feel good for a few days and horrible for a few days. This aspect of the disease is critical for public health and policy. For example, 2 weeks off to recover from the illness may not be enough. These issues related to disease persistence need to be better understood as part of the public policy response to this illness.

The use of social media to gather health information is growing. Some studies used social media surveillance, which they refer to as “infoveillance,” during the early phase of COVID-19 in China to report on symptoms to better detect infections as the virus spread worldwide.^29
–31 Social listening contributes real-world data (RWD) that falls outside of conventional or traditional modes of research data collection such as conventional randomized controlled clinical trials. In the context of therapeutics, real-world evidence, derived from the analysis of RWD, is clinical evidence regarding the use and potential benefits or risks of a medical product.^32,33 Stakeholders, including the US Food and Drug Administration, currently recognize data derived from the following sources as examples of RWD: electronic health records, medical claims and billing data, data from product and disease registries, and data from mobile health devices. Real-world evidence is informing and accelerating the pace of research, regulatory review, and medical product commercialization. The ability to detect signals in very large data sets using social listening could be especially meaningful for rare and emerging diseases, or to detect future pandemics.

In future work, the research team will explore how these analyses can be extended to develop a system for near-real-time analysis of data on an ongoing basis to monitor the disease for possible detection of changes in the nature of the disease course. The long-term goal is to link social media to medical records for the purpose of capturing COVID-19 patient experiences throughout their entire journey, including pre- and post-hospitalization, ultimately leading to a better understanding of disease and improved care.

Conclusions

This pilot study analyzed 2 data sets from social media groups of people with or presumed to have COVID-19: a poll conducted in a Facebook group, and conversation data from a Reddit group including patient-written natural history-like posts of COVID-19 experiences. The study identified 13 symptoms that were increasing and 12 symptoms that were decreasing or stable in prevalence in time line conversation data. Furthermore, Reddit and Facebook groups reported that the disease was lasting longer than 3 weeks, and up to 90 days or more. Persistent disease was reported across a wide range of ages, including people younger than age 40. The combined data sets contained more than 60 symptoms that were not listed in the 4 online symptom lists: CDC, WHO, Mayo Clinic, or WebMD. This pilot study demonstrates that social media data can offer novel insights into patient experiences as a source of RWD. This inductive research approach can quickly generate descriptive information that could help form hypotheses for deductive research approaches. The method also allows for timely assessments of large numbers of social media conversations that could be applied to monitor public health for rapidly spreading diseases such as COVID-19.

Footnotes

Author Disclosure Statement

Ms. Picone and Mr. DeFelice are co-owners of TREND Community, and Drs. Inoue and Wassman hold stock options in the company. TREND Community extracts and analyzes information from social media for the benefit of patients and furthering clinical research that leads to improvements in their quality of life. TREND is a Public Benefit Corporation, an entity that allows it to be self-supporting without needing to fundraise. The results of the analyses have no bearing on the revenue incurred, avoiding any pressure to introduce bias into the analysis. The analyses provide information to TREND Community customers who choose what to do with the information. The remaining authors declared that they have no conflicts of interest.

Funding Information

No funding was received for this article.

Supplementary Material

Supplementary Table S1

Supplementary Table S2

Supplementary Table S3

Supplementary Table S4

Supplementary Table S5

Supplementary Table S6

Supplementary Table S7

References

World Health Organization. WHO coronavirus disease (COVID-19) dashboard. 2020. https://covid19.who.int/ Accessed July 17, 2020 .

Horbach SPJM. Pandemic publishing: medical journals drastically speed up their publication process for Covid-19. https://www.biorxiv.org/content/10.1101/2020.04.18.045963v1 Accessed August 6, 2020 .

Fajgenbaum

, Khor

, Gorzewski

, et al. Treatments administered to the first 9152 reported cases of COVID-19: a systematic review. Infect Dis Ther, 2020; 1–15.

Matricardi

, Dal Negro

, Nisini

The first, holistic immunological model of COVID-19: Implications for prevention, diagnosis, and public health measures. Pediatr Allergy Immunol 2020. [Epub ahead of print]; DOI:10.1111/pai.13271.

Argenziano

, Bruce

, Slater

, et al. Characterization and clinical course of 1000 patients with coronavirus disease 2019 in New York: retrospective case series. BMJ, 2020; 369:m1996.

Velev

, Popov

, Velikov

, et al. COVID-19 and gastrointestinal injury: a brief systematic review and data from Bulgaria. Infez Med, 2020; 28(suppl 1):37–41.

Bostanci Ceran

, Ozates

. Ocular manifestations of coronavirus disease 2019. Graefes Arch Clin Exp Ophthalmol, 2020; 1–5.

Chan

, Chiu

, Yee

, Yuan

, Ng

, Teoh

. A systematic review on COVID-19: urological manifestations, viral RNA detection and special considerations in urological conditions. World J Urol, 2020; 1–12.

Kunutsor

, Laukkanen

. Hepatic manifestations and complications of COVID-19: a systematic review and meta-analysis. J Infect, 2020; 81:e72–e74.

10.

Rubio-Muniz

, Puerta-Pena

, Falkenhain-Lopez

, et al. The broad spectrum of dermatological manifestations in COVID-19: clinical and histopathological features learned from a series of 34 cases. J Eur Acad Dermatol Venereol 2020. [Epub ahead of print]; DOI:10.1111/jdv.16734.

11.

Pinzon

, Wijaya

, Buana

, Al Jody

, Nunsio

. Neurologic characteristics in coronavirus disease 2019 (COVID-19): a systematic review and meta-analysis. Front Neurol, 2020; 11:565.

12.

Rogers

, Chesney

, Oliver

, et al. Psychiatric and neuropsychiatric presentations associated with severe coronavirus infections: a systematic review and meta-analysis with comparison to the COVID-19 pandemic. Lancet Psychiatry, 2020; 7:611–627.

13.

US Centers for Disease Control and Prevention. Symptoms of Coronavirus. 2020. https://www.cdc.gov/coronavirus/2019-ncov/symptoms-testing/symptoms.html Accessed July 13, 2020 .

14.

World Health Organization. Coronavirus: Symptoms. 2020. https://www.who.int/health-topics/coronavirus#tab=tab_3 Accessed July 13, 2020 .

15.

Mayo Clinic. Coronavirus disease 2019 (COVID-19). 2020. https://www.mayoclinic.org/diseases-conditions/coronavirus/symptoms-causes/syc-20479963 Accessed July 12, 2020 .

16.

WebMD. Symptoms of Coronavirus. 2020. https://www.webmd.com/lung/covid-19-symptoms#1 Accessed July 13, 2020 .

17.

Wang

, Xu

, Jiang

, et al. The clinical features and outcomes of discharged coronavirus disease 2019 patients: a prospective cohort study. QJM 2020. [Epub ahead of print]; DOI:10.1093/qjmed/hcaa178.

18.

Convertino

, Ferraro

, Blandizzi

, Tuccori

. The usefulness of listening social media for pharmacovigilance purposes: a systematic review. Expert Opin Drug Saf, 2018; 17:1081–1093.

19.

McDonald

, Malcolm

, Ramagopalan

, Syrad

. Real-world data and the patient perspective: the PROmise of social media?. BMC Med, 2019; 17:11.

20.

Reddit./r/COVID-19Positive. 2020. https://www.reddit.com/r/COVID19positive/ Accessed July 12, 2020 .

21.

Facebook. COVID-19 Support Group. 2020. https://www.facebook.com/groups/517530815866036 Accessed July 12, 2020 .

22.

Franke

, Kaul

. The Hawthorne experiments: first statistical interpretation. Am Sociological Rev, 1978; 43:623–643.

23.

Patel

, Glaze

, Picone

, DeFelice

, Pullen

. The utilization of social media to identify sleep problems associated with PraderWilli syndrome. Sleep, 2019; 42(suppl 1):A309.

24.

Pullen

, Picone

, Tan

, Johnston

, Stark

. Pitolisant treatment improves multiple clinical symptoms of Prader-Willi Syndrome (PWS) in children (P3.6-024). Neurology, 2019; 92(15 suppl):6-024.

25.

Pullen

, Picone

, Tan

, Johnston

, Stark

. Pitolisant is a safe and effective treatment for children with Prader-willi syndrome (pws). Sleep, 2019; 42(suppl 1):A309–A310.

26.

Pullen

, Picone

, Tan

, Johnston

, Stark

. Cognitive improvements in children with Prader-Willi Syndrome following pitolisant treatment-patient reports. J Pediatr Pharmacol Ther, 2019; 24:166–171.

27.

The COVID Tracking Project. COVID Data Tracker. 2020. https://covidtracking.com/ Accessed July 16, 2020 .

28.

Vaira

, Deiana

, Fois

, et al. Objective evaluation of anosmia and ageusia in COVID-19 patients: single-center experience on 72 cases. Head Neck, 2020; 42:1252–1258.

29.

, Xu

, Cuomo

, Purushothaman

, Mackey

. Data mining and content analysis of the Chinese social media platform Weibo during the early COVID-19 outbreak: retrospective observational infoveillance study. JMIR Public Health Surveill, 2020; 6:e18700.

30.

Mackey

, Purushothaman

, Li

, et al. Machine learning to detect self-reporting of symptoms, testing access, and recovery associated with COVID-19 on Twitter: retrospective big data infoveillance study. JMIR Public Health Surveill, 2020; 6:e19509.

31.

Shen

, Chen

, Luo

, Zhang

, Feng

, Liao

. Using reports of symptoms and diagnoses on social media to predict COVID-19 case counts in mainland China: observational infoveillance study. J Med Internet Res, 2020; 22:e19421.

32.

US Food and Drug Administration. Framework for FDA's Real-World Evidence Program. 2018. https://www.fda.gov/media/120060/download Accessed July 16, 2020 .

33.

US Food and Drug Administration. Real-World Evidence. 2020. https://www.fda.gov/science-research/science-and-research-special-topics/real-world-evidence#:∼:text=Real%2Dworld%20data%20are%20the,Claims%20and%20billing%20activities Accessed July 11, 2020 .

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB

0.04 MB

0.03 MB