Should You Search the Internet for Information About Your Acute Symptom?

Abstract

Objective: To determine if symptom-related Web sites give sufficient information for users to seek urgent care when warranted. Materials and Methods: We reviewed 120 Web sites (15 sites for each of eight acute symptoms). Symptom-related sites were identified with Google, Yahoo!^®, and Bing™ searches and focused on potentially hazardous symptoms such as chest pain, shortness of breath, abdominal pain, and syncope. We reviewed each symptom-related site for the presence of critical symptom indicators (key symptom characteristics and associated factors) that triage the user to urgent care. Results: Of the 120 sites reviewed, 41 (33%) contained no critical symptom indicators. No site contained a complete set of critical symptom indicators. Overall, out of the 1,020 total critical symptoms searched for in the sites, we only found 329 (32%). When present, critical symptom indicators were found on the top half of the first page of the site in only 34%. Specific recommendations for further care were absent in 42% of the cases where critical symptom indicators were identified. Conclusions: Symptom-related sites ranked highly by major search engines lack much of the information needed to make a decision about whether a symptom needs urgent attention. When present, this information is usually not located where users can rapidly access it and often lacks prescriptive guidance for users to seek care. Until more sites contain at least minimal triage advice, relying on an Internet search to help determine the urgency of a symptom could be risky.

Introduction

Patients increasingly use the Internet as a resource for medical information. Recent surveys from the Pew Research Center for the study of the Internet show that 59% of all adults (80% of Internet users) use the Internet for health information.¹ The Health Information National Trends Survey found that 61% of people used the Internet first, before any other sources, when looking for health information.²

Although many people are using the Internet for health-related information, it is not entirely clear how well they can screen massive amounts of available information, capture and understand relevant details, judge quality, avoid distractions, and apply that information. We know that patients are not only interested in specific disease information; they look up information on symptoms as well. Our own analysis of a large health information Internet site showed that during 1 year, over 2 million clicks were registered on a symptom checker page, which was separate from other medical information pages.³ This traffic included large numbers of clicks for potentially serious symptoms, including chest pain and abdominal pain, which accounted for 98,000 and 253,000 views, respectively. Although we do not know exactly how many people who searched for chest pain information were actually experiencing unstable angina, the sheer number of people looking up symptom information suggests that at least a few may be experiencing a problem requiring urgent attention.

We wanted to know if symptom-related sites contained triage information sufficient to direct users to urgent evaluation if additional symptom indicators were present. To accomplish this, we modified established quality measures from the telephone triage literature to evaluate symptom-related Web sites.

Materials and Methods

We reviewed 120 Web sites between January 2011 and April 2011 and determined whether the sites contained specific critical symptom indicators, which conferred urgency to the symptom. We excluded sites that were strictly limited to symptom information about children.

This study was approved by the Mayo Clinic Institutional Review Board.

Identification of Web Sites

We searched with Google, Yahoo!^®, and Bing™ for the same eight symptoms. We took the top 10 search hits from each of the search engines (30 total), and, starting with Google, we sequentially reviewed unique sites. For Yahoo! and Bing results, we excluded the sites we had already reviewed and continued to review the nonduplicated sites from these search engines. We stopped the search when we reached 15 unique sites. Our rationale for using the top 15 sites comes from Eysenbach and Kohler,⁴ who found that health information seekers on the Internet do not look much further down a search list after the 10th site. Our rationale for using Google first for our searches came from national search data showing Google accounts for 68% of searches, followed by Yahoo! with 14% and Bing with 13%.⁵

Web Site Review Procedure

Two authors (F.N. and W.J.W.) reviewed all the sites. We explored the contents of the sites up to one link from the search-generated Web address. Our intent was to capture all the critical clinical indicators present on the sites that users could reasonably expect to view. We piloted our review procedure using 30 sites and calculated the Cohen kappa interrater statistic.⁶ Because the critical symptom indicators were well defined and did not leave much room for interpretation (see Appendix), there was excellent kappa agreement (>0.80). For other measures with lower interrater agreement, we revised the criteria to be more objective and improve the kappa agreement. For the final data used in the analysis, we used a research electronic data capture database to capture the search information and used the compare feature to combine the information from both sources into a single combined site review.⁷ As noted, our intent with the two observers was to ensure completeness of information capture. The kappa was used as a check to ensure that we consistently captured the same information. All measures that we report had a kappa of 0.6 or greater.

Web Site Content Appraisal

To evaluate site contents, we modified a method established in telephone triage literature. In telephone triage, calls are monitored to ascertain whether certain critical symptom indicators are addressed during the call. For example, Yanovski et al.⁸ used this technique to evaluate calls to pediatricians in the United States. To assess the quality of telephone triage regarding infant diarrhea, they established nine “critical areas” that constituted a minimum assessment of this symptom. Triage quality was graded by how complete these nine critical symptom indicators were assessed. Derkx et al.⁹ used a similar method to evaluate the quality of triage in The Netherlands.

Like telephone triage, sites sometimes give advice about specific symptoms. For example, a site for chest pain can recommend that users currently having chest pain associated with sweating and shortness of breath call an ambulance. To assess the triage information in symptom-related sites, we created checklists of critical symptom indicators, which conferred increased risk, and used these to judge whether a user could get adequate triage information to assess the urgency of a symptom. For example, in sites for chest pain, we determined whether there was information in the site about chest pain associated with diaphoresis, shortness of breath, crushing pain, history of blood clots, lightheadedness, associated nausea or vomiting, radiation of pain to shoulder or jaw, long period of immobility prior to pain, or loss of consciousness. These critical features of chest pain provided a benchmark for comparison of different sites. We examined chest pain sites based on the presence or absence of those critical symptom indicators.

Identification of Critical Symptom Indicators

We obtained the critical symptom indicators for each symptom from telephone triage textbooks. We used four published triage textbooks, each addressing adult symptoms, to construct checklists of critical symptom indicators.^10

–13 Each of these books is organized by symptoms. For example, each book has a section on chest pain, abdominal pain, shortness of breath, headache, etc. Within each section are lists of critical symptom indicators that would warrant urgent evaluation. We compared lists of all critical symptom indicators from each book. To qualify as a critical symptom indicator, it had to be listed in at least three of the four triage textbooks as requiring urgent evaluation (emergency department [ED] recommendation or to be seen within 8 h). By doing this for several symptoms, we were able to obtain lists of critical symptom indicators that provided literature-supported benchmarks to compare sites.

Selection of Acute Symptoms to Review

We reviewed eight different symptoms, selected a priori based on three criteria. First, the symptom had to be included in the four triage textbooks. Second, the symptom had to have several critical symptom indicators indicating the need for urgent evaluation. Third, the symptom had to be associated with significant potential morbidity. We limited the study to adult symptoms.

To determine which symptoms were most likely to result in morbidity, we used the most current National Hospital Ambulatory Medical Care Survey (2008) and examined hospitalization rates associated with these symptoms. The method for this is described in a previous study.¹⁴

Measures

Our primary measure was the identification of critical symptom indicators on the first page or within one descriptively labeled link from the first page (e.g., “when to seek care,” “symptoms,” “when to see your provider/doctor”). If a critical symptom indicator was present, we assessed for the recommended level of care, time frame for seeking care, and where in the site that advice was located. Other measures included whether the date of last review was present, accessibility of triage recommendation, presence of a differential diagnosis, presence of recommendations for emergency care on the first page, differentiation of symptoms by their relative urgency, presence of disclaimer information, relative level of distracting ads, word count, grade level of text, medical professional authorship, oversight, and discoverable references. Several of these measures were modified from Sagaram et al.¹⁵ in their study of quality measures of online health information.

Following the review of the site, we obtained the word count and the Flesch–Kincaid grade level of the text on the first page of the site.

National Data on Searches and ED Visits and Hospitalizations for Symptoms

To determine the number of national searches for acute symptoms, we used the Google AdWords™ Keyword Tool.¹⁶ The search totals were for the 12-month period from April 2010 through March 2011. For estimates of ED visits for each symptom and hospital admission rates from the ED for the symptoms, we used the National Hospital Ambulatory Medical Care Survey data from 2008.^17,18

Statistical Methods

We used JMP version 9.0.1 (SAS institute, Cary, NC) for statistical analysis of the site review. We captured the content text of the first page of every site. By pasting the content into Microsoft Word 2003 (Microsoft Corp., Redmond, WA), we determined the first page word counts and Flesch–Kincaid grade level of the content.

Results

Eight acute symptoms met our criteria. For each symptom, we identified six to nine distinct critical symptom indicators (see Appendix).

There were 50 unique domain names included in the 120 sites reviewed. The distribution of the number of sites per domain name was skewed. There were two domain names (MedlinePlus and MedicineNet) that together represented 21 sites reviewed. Because there were only eight symptoms, these domain names had more than one distinct site reviewed for each symptom. Three domain names (Wikipedia, FamilyDoctor, and MayoClinic) had a site for each symptom. There were 33 domain names, each having only one site on the review list. The top five domain names accounted for 38% (45 of 120) of search engine picks.

Of the 50 unique domain names, only 8 had sites containing greater than 50% of the critical symptom indicators. There were 16 unique domains whose sites contained no critical symptom indicators. No unique domain contained an average of two-thirds or more of the critical symptom indicators.

Compared with sites with a lower search engine rank, sites with a higher search engine rank showed a significant trend toward containing more critical symptom indicators, identifying medical oversight of content, and listing a differential diagnosis (Table 1).

Table 1.

Percentage Counts of Web Sites by Search Rank Group

	SEARCH RANK (%)
WEB SITE MEASURE	1–5	6–10	11–15	CHI SQUARED TEST FOR HOMOGENEITY P VALUE (PEARSON)	COCHRAN ARMITAGE TEST FOR TREND P VALUE
Number of Web sites	40	40	40	NA	NA
No critical symptom indicators present	9 (23)	13 (33)	19 (48)	0.06	0.018
Critical symptom indicators present
25% or more	30 (75)	24 (60)	17 (43)	0.013	0.003
50% or more	17 (43)	13 (32)	8 (20)	0.10	0.031
75% or more	7 (17.5)	4 (10)	3 (8)	0.35	0.16
All	0 (0)	0 (0)	0 (0)	0.99	0.99
First page identifies specific symptom indicators for which emergency care is recommended	15 (38)	17 (43)	11 (28)	0.36	0.35
Indicator symptoms clearly ranked by urgency on first page	21 (53)	23 (58)	18 (45)	0.53	0.50
Differential diagnosis present	40 (100)	36 (90)	34 (85)	0.047	0.015
Disclaimers present	36 (90)	29 (73)	33 (83)	0.13	0.39
Discoverable references present	26 (65)	18 (45)	18 (45)	0.12	0.074
Medical oversight identified by name and medical degree	22 (55)	14 (35)	12 (30)	0.054	0.023
Date of last content review present	30 (75)	22 (55)	25 (63)	0.11	0.24
Potentially distracting advertising^a	15 (38)	12 (30)	19 (48)	0.27	0.36
Inconsistent triage advice^b	6 (15)	9 (23)	8 (20)	0.69	0.57

Defined as a Web site with multiple pop-ups or dynamic ads.

Defined as a Web site containing different recommendations for the same critical symptom indicator.

NA, not applicable.

We looked for a total of 1,020 instances of critical symptom indicators on the 120 sites; we found 329 (32%). Of the 329 critical symptom indicators identified, 64% were associated with a recommendation to seek care emergently or urgently (Table 2). Critical symptom indicators lacked clear recommendations for a time frame to seek care and clear recommendations for source of care in 42% and 23% of instances, respectively.

Table 2.

Counts of Recommendations and Locations of the Critical Symptom Indicators (n=329)

	COUNTS (%)
Recommended time frame of care^a
Emergency/urgent	211 (64)
Indefinite or none	138 (42)
Other	43 (13)
Recommended source of care^a
Ambulance/911	93 (28)
Emergency department	71 (22)
Call provider	85 (26)
Appointment	68 (21)
Indefinite	75 (23)
Location of critical symptom indicator on Web page
First page top half	112 (34)
First page bottom half	109 (33)
Linked page	108 (33)

More than one recommended source or care or time frame for the same critical indicator occurred in 23 Web sites, accounting for totals greater than 329.

There was great variability in the word counts and Flesch–Kincaid grade levels among different sites. The median word count was 893 (range, 64–14,479 words). Median grade level of the content was 9.5 (range, 4.6–16.5). Word count and grade level both showed a weak but statistically significant inverse correlation with the proportion of critical symptom indicators present: adjusted R ²=0.038 (p<0.019) and R ²=0.048 (p<0.009), respectively.

Discussion

To our knowledge, this is the first study to look at triage advice contained in symptom-related Web sites. The eight symptoms we examined accounted for 65 million U.S. Google searches yearly, were associated with 26 million U.S. ED visits in 1 year, and carried a high degree of morbidity as measured by hospitalization rates (Table 3). To assess the completeness of triage advice, we used established criteria from telephone triage literature and triage textbooks as a benchmark. In general, we found the sites did not completely address critical symptom indicators necessary to triage the symptom, and many sites did not address any of the critical symptom indicators.

Table 3.

Counts of Yearly Google Searches and Emergency Department Hospitalization Rate by Symptom

SYMPTOM SEARCHED	U.S. GOOGLE SEARCHES YEARLY, IN MILLIONS (APRIL 2010–MARCH 2011)	U.S. ED VISITS IN CALENDAR YEAR 2008, IN MILLIONS (AGE OVER 14 YEARS)	ED VISIT HOSPITALIZATION RATE AGE 18 YEARS AND OVER [% (CI 95%)]
Chest pain	4.5	6.4	33 (30, 37)
Shortness of breath	1.3	3.2	46 (42, 50)
Abdominal pain	3.7	7.9	18 (16, 20)
Headache	10.8	3.0	6 (4, 8)
Fever	27.7	1.5	33 (27, 38)
Diarrhea	12.4	0.36	23 (15, 31)
Nausea and vomiting	0.31	3.2	22 (18, 27)
Fainting	3.9	0.78	35 (29, 41)

CI, confidence interval; ED, emergency department.

The millions of Internet searches for acute symptoms suggest how important the critical symptom indicators may be. According to our data from Google AdWords, even if only one in 1,000 U.S. chest pain or abdominal pain searches was used for a patient with an urgent symptom, there would be over 8,000 people each year who could be helped by site information directing them to timely care. More complete site triage information could also benefit users who do not even have the symptom. Surrogate caregivers or spouses may be an important user group needing information about critical symptom indicators. We know that nearly half of Internet users (48%) who go online for health information say their last search was on behalf of another person.¹ Although we have few data on what these surrogates are looking for, some may be spouses who need advice about the chest pain in their stubborn husbands who insist “it's only heartburn” and won't seek care. It is well known that many patients deny their symptoms, and, in the case of chest pain, this denial can cause a significant delay in care.^19,20 More complete triage advice on the Internet could not only help patients searching on behalf of themselves, but also could help spouses and surrogates looking for advice about a symptomatic husband or aging parent.

Our study shows that most symptom-related sites will have information about a differential diagnosis. However, if you want to know whether you need your symptom evaluated right away, you may not get the information you need. Although higher-ranked sites were more likely to have important triage information about a symptom, no site consistently addressed all the major critical symptom indicators found in triage textbooks. Even when present, the critical symptom indicators were often not readily noticeable because of their location on the Web page. The multiple potentially distracting pop-ups and dynamic ads in 38% of the sites might be an additional barrier to rapidly finding important triage information.

Even when the critical symptom indicator was present, there could be unclear or even potentially conflicting advice about what to do. A small but significant proportion of sites (23%) gave different recommendations for the same symptom indicator. For example, a headache site on one page might recommend an ED evaluation for a sudden severe headache, whereas on another page it might recommend a call to the doctor. We also noted several indefinite recommendations for source of care and time frame for care (Table 2). An example of this was the nonspecific recommendation to “seek care” if a critical symptom indicator was present. With the “seek care” recommendation, there is no definite recommendation for where or from whom to get care and no indication of how quickly to seek care. Sites should pay attention not only to completeness and visibility of critical symptom indicators but also give unambiguous recommendations for time frame and source of care.

More site content (as measured by word count) was not correlated with the presence of critical symptom indicators. In fact, there was a statistically significant inverse correlation of word count to critical symptom indicators. The explanation for this is that many of the content dense sites were aggregator sites, composed of a voluminous aggregation of dictionary definitions of the symptom or a compilation of information about diseases that might cause that symptom. Wikis, although more structured, also tended not to have critical symptom indicators.

This study had limitations. It is unknown how many searches represent actual patients seeking information about active symptoms. However, search volumes suggest that if only one of 1,000 searchers was having some chest pain from cardiac ischemia, there would be about 4,500 Internet searchers yearly who should be getting rapid advice to go the hospital. We did not examine the overall accuracy of the information on these sites, and we did not review sites dedicated to symptoms of children. Our study was designed to capture all the critical symptom indicators viewable within one link of the search engine link. Casual users or those in the throes of an acute symptom would probably not have the patience or focus to search word by word through some of the lengthy sites, as we did in this study. Thus, our study likely overestimates what would be obtained in a real Web-browsing experience. Another limitation is the dynamic nature of the Web itself. Search engines come up with different results according to changing search algorithms, and site content can change frequently.

Further research needs to be done to determine what proportion of the searches for acute symptoms may require urgent care. In the meantime, health information sites should consider enhancing their acute symptom-related Web pages so that a list of critical symptom indicators is readily visible on the site and is associated with unambiguous recommendations.

Footnotes

Disclosure Statement

No competing financial interests exist.

Appendix

Table A1.

Acute Symptoms and Associated Critical Symptom Indicators

SYMPTOM	CRITICAL SYMPTOM INDICATORS^a	TOTAL CRITICAL SYMPTOM INDICATORS EVALUATED
Chest pain	Shortness of breath (4), loss of consciousness, can't stand, or weakness (3), pain quality crushing or pressure (3), diaphoresis (4), radiation of pain to arm or jaw (4), history of blood clots in lungs, legs, or phlebitis (4), recent period prolonged sitting or traveling (3), dizziness/lightheadedness (4), nausea/vomiting (3)	9
Shortness of breath	Severe breathing difficulty (3), cyanosis, blue lips, or color change (3), confusion, decreased level of consciousness (3), inability to swallow (3), history of blood clot in lungs, legs, or phlebitis (3), recent trauma or surgery (3), pregnancy or recent childbirth (3), chest pain (3), fever (4)	9
Abdominal pain	Fainting, lightheadedness, too weak to stand (4), pain severe (4), hematemesis/coffee ground emesis (3), bloody or black stools (4), nausea and vomiting (4), temperature over 101°F or fever (3), pregnant (3), age 60 years or over (3), female with late or missed menses and pelvic pain (3)	9
Headache	Confusion or difficult to awaken (4), focal weakness or numbness (4), difficulty with speech (4), stiff neck (4), "worst" headache (4), severe and sudden onset (3), severe with eye pain or vision change (3), temperature over 103°F (4), immunocompromised (3)	9
Fever	Decreased consciousness (4), difficulty breathing (4), headache and stiff neck (4), temperature over 103°F (4), temperature over 100.5°F with age over 60 years or frail/high risk or immunosuppressed (4), low urine output (3), rash with fever (4), difficulty swallowing (3)	8
Diarrhea	Weakness, dizziness, or faintness (4), decreased consciousness (3), severe abdominal pain (4), fever (3), black or bloody stools (4), low urine output (3)	6
Nausea and vomiting	Diaphoresis/cold or clammy skin (3), decreased consciousness (4), coffee ground or bright red emesis (4), recent head injury (4), decreased urine output (4), persistent abdominal pain (4), fever with age over 60 years or frail or immunosuppressed (3), lightheadedness or fainting (3), toxin/medication or plant ingestion recently (3)	9
Fainting and syncope	Persistent lightheadedness (3), confusion (3), difficulty breathing (3), bleeding/hematemesis/melena (3), chest pain (4), palpitations (3), heart problem history (4), abdominal pain (3), still unconscious (3)	9

Numbers in parentheses are number of triage textbooks endorsing, of four total.^10

–13

References

Fox

. Health information is a popular pursuit online. www.pewinternet.org/Reports/2011/HealthTopics/Part-1/Internet-access.aspx. 2011 May 30.

Health Information National Trends Survey (HINTS) codebook. http://hints.cancer.gov/dataset.jsp. 2011 May 30.

North

, Varkey

, Laing

et al.

Are e-health Web users looking for different symptom information than callers to triage centers?

Telemed J E Health, 2011; 17:19–24.

Eysenbach

, Kohler

. How do consumers search for and appraise health information on the World Wide Web? Qualitative study using focus groups, usability tests, and in-depth interviews. BMJ, 2002; 324:573–577.

Hitwise.com search engine analysis. www.hitwise.com/us/datacenter/main/dashboard-23984.html?gclid=COPd85b4j6kCFQsCbAodLDnBpA. 2011 May 30.

McGinn

, Wyer

, Newman

et al.

Tips for learners of evidence-based medicine: 3. Measures of observer variability (kappa statistic)

CMAJ, 2004; 171:1369–1373.

Harris

, Taylor

, Thielke

et al. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform, 2009; 42:377–381.

Yanovski

, Yanovski

, Malley

et al. Telephone triage by primary care physicians. Pediatrics, 1992; 89:701–706.

Derkx

, Rethans

J-JE

, Maiburg

et al. Quality of communication during telephone triage at Dutch out-of-hours centres. Patient Educ Counsel, 2009; 74:174–178.

10.

Briggs

. Telephone triage protocols for nurses, 3rd. Philadelphia: Lippincott William & Wilkins, 2007.

11.

Katz

. Telephone medicine triage and training for primary care, 2nd. Philadelphia: F.A. Davis, 2001.

12.

Thompson

. Adult telephone protocols, 2nd. Elk Grove Village, IL: American Academy of Pediatrics, 2009.

13.

Wheeler

. Telephone triage protocols for adult populations. New York: McGraw-Hill, 2009.

14.

North

, Varkey

. How serious are the symptoms of callers to a telephone triage call centre? J Telemed Telecare, 2010; 16:383–388.

15.

Sagaram

, Walji

, Meric-Bernstam

et al. Inter-observer agreement for quality measures applied to online health information. Stud Health Technol Inform, 2004; 107:1308–1312.

16.

Google Adwords keyword tool. www.adwords.google.com. 2011 May 30.

17.

National Hospital Ambulatory Medical Care Survey (NHAMCS)

www.cdc.gov/nchs/ahcd.htm. 2011 May 30.

18.

National Hospital Ambulatory Medical Care Survey: 2008 emergency department summary tables. www.cdc.gov/nchs/data/ahcd/nhamcs_emergency/nhamcsed2008.pdf. 2011 May 30.

19.

Galdas

, Cheater

, Marshall

. Men and health help-seeking behaviour: Literature review. J Adv Nurs, 2005; 49:616–623.

20.

White

, Johnson

. Men making sense of their chest pain—niggles, doubts and denials. J Clin Nurs, 2000; 9:534–541.