An assessment of the quality of information on stroke and speech and language difficulty web sites

Abstract

Online healthcare information is used by patients, their carers and families (PCF) experiencing speech and language difficulties (SLD) after a stroke. This information may be of variable quality. Tools have been designed to assess the risk of poor information quality, evaluating both generic and specific concepts, yet none focuses on stroke. The readability of information is also an issue when communication disorders are under consideration. The study investigated the quality and readability of information on 51 web sites pertaining to SLD following a stroke. These were assessed using two generic evaluation tools (DISCERN and the HON Code), readability tests (Flesch Reading Ease and Flesch-Kincaid Grade Level) and a specifically designed Stroke Tool based on the reported information needs of stroke PCF experiencing SLD. The tools themselves were then evaluated for their feasibility, reliability and validity. In common with previous studies, the information quality of the selected web sites was found to be extremely variable with only 59% achieving a score of 50% or more using HON, only 37% using DISCERN, and 49% using the Stroke Tool. Readability is generally very poor with only 6% of web sites scoring below the recommended grade 7. The tools did not correlate well, suggesting that they measure different domains of quality. The Internet may be a valuable resource for stroke PCF; however, the variable quality of information means that cautious use is recommended. The readability of information is a more serious issue that needs addressing by web site developers.

Keywords

carers consumer health information families patients speech and language difficulties stroke

1. Introduction

1.1. Health Information on the Internet

Patients are increasingly involved in decisions regarding their own healthcare and providing appropriate information is an important part of this process. Not only may it contribute to awareness campaigns, it may educate patients in self-care, and provide support for patient decision-making [1]. Information may take a variety of forms: while health professionals are often considered the most appropriate sources of information, leaflets and other materials can play a vital supplementary role [1]. However, the Internet is increasingly used for information delivery.

The amount of information on the World Wide Web is increasing constantly [3]. In 2011, there were an estimated 175 million active web sites on the Internet [4], and digital content is reported to have exceeded 800 billion gigabytes. Healthcare information is part of this and is referred to increasingly, not only by health professionals, but by patients, their families and members of the public [5]. 42% of the UK population are reported to use the Internet to seek health-related information [6]. As the amount of information on the Internet increases, consumers may feel that the desired information must exist somewhere, and that it only takes good enough searching to find. The increasing emphasis on patient involvement in decision-making [7,8] encourages consumers to seek information actively. However, the quality of online health information is an important issue, due to the lack of publishing control on the Internet [9]. Unlike paper-published material, which may be edited, verified and proofread, such processes and protocols do not exist on the Internet; anyone can publish anything they like and claim it to be true. While in some areas this may be relatively harmless, false health information can result in distressing, and even fatal, consequences for patients [10]. The motivation to determine a way to evaluate sources of health information is therefore great.

1.2. Evaluation tools

Evaluation tools are available to assess online healthcare information, and previous studies have used generic tools to examine the quality of web-based healthcare information. Some studies have used generic tools to evaluate web-based information on a specific condition, for example breast cancer [11], depression [12] and genetics [13], while others have developed evaluation tools specific to a condition, which have proved more reliable in assessing how well web sites meet information needs [14 –17]. This raises the issue of whether a generic tool can suffice in evaluating web sites for a specific condition [14]. Studies and tools relating to stroke information on web sites are limited: one study evaluated 30 web sites using previously defined criteria [18, 19].

1.3. Stroke and speech and language difficulties

A stroke, or cerebrovascular accident, is the interruption of blood supply to parts of the brain, usually as the result of a blood clot, haemorrhage or other brain event [20]. A lack of oxygen following a stroke can result in significant brain damage and can lead to death. Depending on the area of the brain affected, brain damage can affect many body’s functions: the brain controls muscle movement and coordination, and many other functions, for example, feelings and communication. The effects of strokes may be evident immediately, changing a person’s life suddenly and permanently.

Strokes are more prevalent in older people [18], and with the increasing older population, the incidence of strokes may increase [21], although decreases in stroke-associated mortality are increasing the numbers of stroke survivors [22, 23]. Stroke is a leading cause of adult disability worldwide [22]. Although there is no complete ‘cure’ for stroke, treatments may improve the clinical prognosis and quality of life. Minimizing the impact of stroke is more important as the stroke incidence increases and to offset the increasing costs of care [24]. One way of limiting this may be a greater awareness of stroke in general, stroke symptoms and risk factors for strokes, for example, smoking and lack of exercise [22]. An improved awareness of what may be done post-stroke to improve quality of life and decrease stress is also important. Access to appropriate and high-quality information is therefore important.

Speech and language difficulties (SLD) are a common result of brain damage following a stroke, with varying degrees of severity [25]. This constitutes one key area of information need for patients, their carers and families (PCF) at all stages following a stroke, from hospitalization, through therapy and rehabilitation, to returning home [26]. Communication problems are distressing for the PCF. While families may have to relearn how to identify the patient’s needs through alternative communication methods, the patient may become completely dependent on their carers to communicate for them and isolated through a reduced ability to comprehend complex situations and communicate to those with little awareness of their condition.

A range of conditions associated with communication, speech and language may occur as a result either of muscle weakness or of reduced brain function following a stroke. Aphasia, or dysphasia [27], is one of the most common language disorders acquired following a stroke, with approximately one-third of stroke patients developing it [25]. It may be defined by a variety of difficulties, encompassing a wide range of aspects of language, varying in severity and over time. It may affect the construction and comprehension of spoken and written language and other non-verbal communication.

As SLD may affect the comprehension of written text, information provision must take account of the readability of information sources. Although research has examined the written information about SLD, and more particularly aphasia, given to PCF following strokes, these concentrate mainly on physical leaflets provided by health professionals [28, 29]. One study examined the quality and accessibility of aphasia web sites; however, this was not specifically in relation to strokes [5]. With the increased prevalence of health information on the Internet, PCF may use it to fulfil any information need that is not met through provision from healthcare professionals.

Studies have evaluated stroke and SLD web sites, ranging from a practical list of resources for use by PCF [30], to those examining the quality and suitability of information on web sites [5, 18]. Although previous studies [5, 18] have a different focus (stroke education vs aphasia), both use quality evaluation criteria [19]. Thirty stroke education web sites scored highly on reliability, as measured in comparison with the National Clinical Guidelines for Stroke [18], but low on readability, as calculated using Flesch, and accountability. Similarly, there was little correlation between the quality of information and the accessibility of five aphasia web sites [5]. If the information was of high quality, it was not necessarily accessible, and vice versa, and quality and accessibility overall were poor.

The accessibility and readability of information are important issues to consider when considering information provision to people with aphasia and other SLD after a stroke [31, 32]. Information provided to stroke patients and carers must be of a suitable reading level. There is little purpose in providing high-quality information if it is not usable by the intended audience [32]. The average reading level of provided written material is often higher than that of the patients and carers concerned: sixth grade is considered maximum for stroke, and especially for aphasia, for patients and their carers [28, 31, 33, 34]. By taking readability and accessibility measures, for example, large font sizes and pictures, into account, the comprehension of information is facilitated [28, 31, 35]. However, lowering readability levels too far may contribute to infantilism of patients [1].

1.4. The information needs of patients with speech and language difficulties following a stroke, their families and carers

Stroke impacts on the patient, their family and carers [47 –49], particularly when speech is affected [47], so it is important to provide families of stroke survivors with appropriate information [38, 48]. Difficulties arise because family members do not necessarily recognize themselves as carers, especially in the early stages following a stroke [50, 51]. Information needs of families and carers are similar not only to each other, but also to those of patients [46]. Improving family support may improve the efficacy of treatments [48].

The information needs of stroke PCF experiencing SLD are substantial and change with time, according to the time period following the stroke [36 –40]. Despite well-documented benefits of providing written health information, stroke patients with SLD do not necessarily receive the information they require from health professionals, and information may be too complex [32, 34, 36, 41, 42], particularly among stroke patients with aphasia [43]. Information provision should be integral to care following a stroke, especially information on the services and benefits available to PCF, in addition to strategies for managing practically with their changed circumstances [34]. Such information can improve knowledge of the condition, increase levels of life satisfaction and reduce patient depression [44]. The patient’s recovery process and outcome are improved [36]; uncertainty and anxiety levels amongst carers are also reduced [45].

Clinical information forms a large area of information need among PCF who have experienced a stroke. PCF wish to know about the stroke itself, the causes and risk factors [25, 26, 33, 34, 36, 38, 39, 41, 46, 47, 49, 50, 52], the risks and prevention of stroke recurrence [25, 33, 34, 36, 38, 39, 41, 46, 53], as well as the consequences of the stroke on the body, for example, the visual field and bladder function, and the management of such effects [25, 26, 33, 34, 36, 38, 39, 41, 46, 48, 49, 52, 53]. The consequences may be emotional and behavioural, for example, depression or memory loss [26, 33, 38, 41], as well as physical, for example, incontinence [33, 38, 41]. Information on medications and treatments is frequently needed, including details on the rehabilitation process [25, 26, 33, 34, 36, 38, 41, 46, 48, 49, 53]. An indication on the prognosis is often required to gauge potential recovery [36, 38, 46, 48, 50].

Practical information forms another information need for PCF, and may include details on the healthcare services available [25, 34, 36]. Patients frequently wish to know which medical professional they should approach for help [26, 36]. Coping mechanisms form another area of practical information need, especially for returning home, and for care that is available [25, 33, 36, 39]. This may include assistive devices and aids around the house to help with walking, balance, etc. [26, 33, 39]. PCF may wish to know about leisure activities and exercise following a stroke, including sexual activity [26, 33, 34, 36, 39, 41, 46, 48, 52]. PCF may also need to access further information resources, for example, information on local community and support groups and clubs, which can provide psycho-emotional support for people. Information is needed on the experiences of others, which offers the reassurance that they are not alone [39, 46]. The patient may need information on advice and support for their partner and carer [39, 46]. Financial information, including available benefits, help with tax and legal matters [25, 34, 36, 39, 46, 48, 49, 53], may also be needed. Patients may also wish to return to work, and may need to know about driving [26, 36]. Carers have concerns about the impact of the stroke on their own lives, such as their relationship with the stroke survivor [40, 52, 54].

Studies have examined the needs of stroke patients more generally; however, those that focus more specifically on people with aphasia and other SLD conclude that, although a need for information on these conditions is important, people will usually be experiencing a wide variety of symptoms as a result of stroke and so have a broad range of information needs [28, 32]. The SLD-related information needs, however, range from general information on the condition to practical help in communicating, and information on further speech and language therapy [26, 37, 40, 46, 48, 52, 53]. PCFs have information needs as regards alternative therapies, independent living, such as job retraining and volunteer opportunities [48]. Carers have more practical information needs, especially regarding services available and who to approach for help [34, 49, 50, 54]. They require information on practical and coping strategies at home, such as falls prevention and privacy issues [45, 42, 53]. They may need information on the experiences of others and advice on adjusting to the carer role [39, 49].

However, the most important information need for all concerned is that information is accurate, up-to-date and accessible [49, 51].

1.5. Research aims and objectives

The aim of this study was to evaluate the quality, content and readability of information on web sites pertaining to SLD following a stroke. The objectives of the study were:

to develop an evaluation tool, specifically designed for stroke and SLD web sites, based on the information needs of PCF experiencing SLD following a stroke;

to identify which web sites hold the best quality and most accessible healthcare information on the practical aspects of living with SLD in stroke patients;

to evaluate the feasibility, reliability and validity of the evaluation tools for measuring information quality.

2. Methodology

2.1. Search strategy

A search for stroke and SLD web sites was carried out using Google Chrome. Web sites were identified using three of the most popular search engines: Google, Yahoo! and Bing [55, 56]. Non-specialist search terms (‘stroke speech and language problems’ and ‘stroke speech and language difficulties’) were used to ensure that sites found would be those that stroke PCF might find themselves. While most consumers do not venture past the first two pages of search results [57], it was decided that a broader sample of sites would be taken from viewing further pages. Consequently, the first 10 pages of search results (n = 100) on each search engine were trawled for suitable sites. While not a complete set of sites, this should have encompassed those most commonly found by consumers, as well as providing an effective sample of relevant sites.

2.2. Sampling and capture of web sites

Selection criteria were developed to determine which web sites identified by the search engines would be suitable for the evaluation. Initial searches revealed only two web sites solely devoted to both stroke and SLD. Relevant sites returned were either stroke web sites with pages on SLD (e.g. The Stroke Association), speech or language web sites with pages on strokes (e.g. Speakability), or general web sites containing pages on strokes and SLD (e.g. Health Online). The web sites eventually identified as being suitable for the study met the following criteria (as adapted from those used in previous studies [16, 58, 59]):

the content of the web site is related to stroke and SLD;

the web site is aimed at the consumer rather than clinicians;

the web site is predominantly in English;

the web site is accessible to all, with no password necessary;

the web sites represented the range of sectors found by search engines.

At the end of the selection process, 51 web sites had been identified (Table 1). All web sites were captured using Offline Explorer, as recommended previously [16], over a 24 hour period (July 2010). This ensured that the information was from the same time point across all sites, which is important owing to the changing nature of the Internet. Although caches and archives are sometimes available for some web sites, they are not necessarily comprehensive, may store incomplete information and would not permit a valid comparison. The top five relevant levels of web sites were captured using the software to provide sufficient information for analysis, while maintaining small enough file sizes for the resources available.

Table 1.

Score (and rank) for each web site according to each tool according to overall rank

Site name (overall rank based on mean of five ranks)	Percentage score for tool (rank)			Readability (rank)
	DISCERN	HON Code	Stroke Tool	FRE Score	FKGL Level
Stroke Association (1)	67 (6)	70 (14)	92 (1)	68.4 (4)	7.2 (4)
BUPA (2)	73 (2=)	100 (1=)	48 (26=)	66.1 (5)	7.6 (6)
Patient UK (3)	55 (13=)	97 (3=)	53 (20=)	64.0 (6)	7.7 (7)
Heart & Stroke Information Point: NHS Tayside (4)	55 (13=)	67 (15)	78 (4=)	60.4 (12)	8.4 (10=)
Revolution Health (5)	76 (1)	100 (1=)	73 (8=)	55.0 (25)	9.2 (20)
MedLine Plus (6)	55 (13=)	97 (3=)	48 (26=)	62.7 (7)	8.0 (8)
eHealth MD (7)	61 (8=)	93 (5=)	53 (20=)	58.5 (17)	9.1 (17=)
Caring.com (8)	50 (17=)	90 (8=)	65 (15)	59.1 (15)	9.1 (17=)
Chest, Heart & Stroke Scotland (9=)	33 (37=)	53 (24=)	88 (2)	62.2 (8)	8.4 (10=)
NIH Senior Health (9=)	34 (33=)	63 (16=)	43 (30)	73.4 (1)	5.9 (1)
Everyday Health (9=)	61 (8=)	90 (8=)	77 (7)	54 (26)	10.2 (32)
@ a stroke (12)	46 (22=)	57 (20=)	55 (18=)	60.7(10=)	8.9 (15=)
Brain and Spine Foundation (13)	39 (29)	93 (5=)	52 (24)	60.9 (9)	9.5 (22=)
Stroke Unit Glos (14)	70 (5)	53 (24=)	70 (11)	55.5 (23)	10.1 (28=)
National Stroke Association (15)	50 (17=)	57 (20=)	82 (3)	51 (33)	9.5 (22=)
About.com (16)	73 (2=)	93 (5=)	67 (12=)	48 (38)	11.4 (42=)
University of Maryland Medical Center (17)	43 (27=)	73 (12=)	37 (32=)	58.3 (18)	8.8 (12=)
NHS Choices (18)	48 (20=)	73 (12=)	78 (4=)	51.8 (30)	10.4 (36=)
My Stroke Journey (19)	36 (31)	47 (31=)	33 (34=)	69.7 (3)	7.4 (5)
UK Connect (20)	45 (24=)	60 (19)	32 (39)	58.6 (16)	8.3 (9)
Family Doctor (21)	51 (16)	50 (27=)	67 (12=)	55.8 (22)	10.3 (33=)
Stroke SA (22)	63 (7)	40 (36=)	73 (8=)	41.7 (45=)	8.9 (15=)
Stroke Recovery Association NSW (23)	33 (37=)	40 (36=)	67 (12=)	59.3 (14)	8.8 (12=)
Camden Council (24)	33 (37=)	47 (31=)	53 (20=)	57.2 (20)	9.1 (17=)
American Stroke Association (25)	61 (8=)	53 (24=)	73 (8=)	41.7 (45=)	11.4 (42=)
NHS Kirklees (26)	33 (37=)	33 (43)	25 (44)	70.2 (2)	6.2 (2)
Speech Therapy on Video (27)	50 (17=)	50 (27=)	50 (25)	49.6 (36)	10 (26=)
Health Talk Online (28)	44 (26)	80 (8=)	55 (18=)	50.1 (35)	11.7 (45)
National Stroke Association of Malaysia (29)	34 (33=)	30 (44=)	45 (28=)	58.1 (19)	8.8 (12=)
Stroke Speech (30)	45 (24=)	50 (27=)	33 (34=)	51.1 (31=)	9.5 (22=)
American Speech–Language–Hearing Association (31)	58 (11)	47 (31=)	40 (31)	47.9 (39)	10.1 (28=)
Children’s Hemiplegia and Stroke Association (32=)	46 (22=)	50 (27=)	78 (4=)	41 (48)	11.8 (46)
Icommunicate (32=)	56 (12)	80 (8=)	45 (28=)	40.4 (49)	12.7 (50)
Stroke Foundation of New Zealand (34)	43 (27=)	43 (35)	60 (17)	49.5 (37)	10.3 (33=)
The DANA Guide to Brain Health (35=)	48 (20=)	63 (16=)	33 (34=)	45.4 (41)	11.4 (42=)
Volunteer Stroke Scheme (35=)	35 (32)	30 (44=)	63 (16)	52.9 (28=)	10.3 (33=)
Dumferline and West Fife CHP: NHS Fife (37)	38 (30)	47 (31=)	28 (41=)	55.3 (24)	10.1 (28=)
Speech Disorder (38=)	70 (4)	57 (20=)	28 (41=)	42.1 (44)	11.9 (47=)
Stroke Watch (38=)	23 (45)	27 (47=)	18 (48)	60.2 (13)	6.6 (3)
Private Speech Language Therapy (40)	34 (33=)	37 (40=)	27 (43)	56.2 (21)	9.7 (25)
Different Strokes (41)	34 (33=)	40 (36=)	53 (20=)	47.3 (40)	10.9 (40)
Speakability (42=)	33 (37=)	30 (44=)	37 (32=)	52.9 (28=)	10.4 (36=)
Stroke in Stoke (42=)	33 (37=)	40 (36=)	33 (34=)	50.8 (34)	10.4 (36=)
Stroke Talk Survivors (44)	19 (48=)	23 (50)	20 (46=)	60.7 (10=)	10 (26=)
The National Aphasia Association (45)	30 (43)	63 (16=)	33 (34=)	41.4 (47)	11(41)
Bristol Area Stroke Foundation (46)	20 (47)	37 (40=)	23 (45)	53.1 (27)	10.1 (28=)
Confederation of Indian Organisations Stroke Project (47)	13 (50)	37 (40=)	17 (49=)	51.1 (31=)	9.4 (21)
Speech–Language Therapists Association (48)	26 (44)	57 (20=)	30 (40)	44.6 (42)	12 (49)
Association of Speech and Language Therapists in Independent Practice (49)	21 (46)	27 (47=)	17 (49=)	42.7 (43)	10.5 (39)
Aphasia Treatment (50)	19 (48=)	13 (51)	20 (46=)	39.4 (50)	11.9 (47=)
Stroke Folk (51)	10 (51)	27 (47=)	7 (51)	31.5 (51)	13 (51)

2.3. Selection of tools to evaluate web sites

Two generic information quality evaluation tools used in previous research [16, 17], DISCERN and the HON Code, were selected to evaluate the quality of information. A measure of readability is important in assessing web sites associated with SLD after a stroke, because people with a stroke frequently experience difficulties in comprehending written text: text on web sites must therefore be of a suitable level for comprehension by the target audience. The Flesch Reading Ease (FRE) and Flesch–Kincaid Grade Level (FKGL) formulae were selected for this study and implemented through the Spelling and Grammar function in Microsoft Word 2007. FRE is based on the average sentence length and the average number of syllables in a word [61], and has been used previously [18, 28, 60, 62]; the scores are recognized generally as a measure of readability. The FKGL, in particular, is effective for measuring readability to the non-specialist. FRE scores range between 0 and 100, higher scores indicating more readable text, and 60–70 is considered ideal for most texts [61]. The FKGL is based on the US grade system, meaning that a score of 9 would necessitate 9 years of education, a lower level denoting that a lower level of reading is required. The recommended grade level is 7–8 for most documents, although a maximum of sixth grade is recommended for healthcare information [63].

A specific tool was developed to assess the content of online information about SLD following a stroke, according to the findings in the literature summarized in Section 1.4. The aim of the Stroke Tool was to assess the extent to which the information on a web site matched the reported information needs of PCF experiencing SLD following a stroke. The tool was not designed to assess the accuracy of the information on web sites as this was covered by the generic tools. First, a detailed investigation of the information needs of PCF experiencing SLD following a stroke was carried out via a review of the literature (Section 1.4). The identified information needs were classified into sections according. An initial tool was designed based on these reported information needs, in two main sections: one for general stroke information and one for SLD information. Finally, a modified version of the HON Code scoring method was devised for the Stroke Tool, as in other studies [14 –17]. Accordingly, there were three possible responses for each question: ‘Yes’, ‘Partly’ and ‘No’, scoring 2, 1, and 0 respectively.

2.4. Evaluation of web sites

The evaluation of the web sites was undertaken from an end user perspective, an approach used previously [14 –17]. Although the lead researcher (R.S.) did not have clinical knowledge, a family member had experienced a stroke, so she was able to consider the needs of a PCF experiencing SLD after a stroke. Furthermore, considerable further knowledge of the condition and the need for information within the defined population was derived from the literature review.

Each of the 51 web sites was evaluated by RS using each tool between 8 and 19 July 2010. Each web site was viewed via Offline Explorer, and evaluated using the two generic tools and the stroke-specific tool, before the readability tests were undertaken.

The responses to each question of each tool were noted according to the scoring method of each tool. These scores were then recorded in Microsoft Excel 2007 and transferred to SPSS for further analyses. The time taken to use each tool on each web site was also recorded so that the tools could be evaluated for feasibility. The time was recorded from initial viewing of the web site to the completion of the final question in the tool being used. To overcome any bias towards the tool being used last being quickest to evaluate, the HON-Code, DISCERN and the Stroke Tools were rotated with each web site.

2.5. Data analysis

Sub-totals and totals of scores for each web site were calculated for the tools. Scores were standardized into a percentage score for subsequent analyses. Web sites were ranked according to scores for each tool, and an overall ranking was obtained from the mean ranking for the five tools.

The internal reliability of each tool was tested using Cronbach’s α Coefficient, a score of 0.8 or above suggesting acceptable internal reliability [64]. Kendall’s coefficient of concordance (W) was used to test the level of agreement among the rankings of the five tools. Spearman’s rank order correlation coefficient was used to test the degree of correlation between each pair of tools. The Kruskal–Wallis test was used to test for differences in the mean time taken to evaluate the web sites for each tool.

3. Results

3.1. Description of the sample of web sites

As UK search engines were used, the top results of searches were UK sites and these made up the majority of the sites assessed in this study (n = 26). Other web sites were based in the USA (n = 17), Australia (n = 2), New Zealand (n = 3), Malaysia (n = 1), South Africa (n = 1) and Ireland (n = 1). The sample of web sites included a range of types, from a two-page community group site (Stroke Folk) giving details of the group’s activities, to Stroke charities (Stroke Association) giving information about various subjects associated with stroke, to more general healthcare information web sites (Patient UK) giving information not only about stroke and related conditions, but also about a full range of healthcare issues. Some sites mainly concerned stroke, with some information on SLD (n = 18); some were mainly on SLD, with some information on stroke (n = 11); only two were specifically designed to cover both areas. The other main category was that of general healthcare sites (n = 19), covering a wide range of subjects including those relevant to the current study. One web site was a local council web site with several pages of information relevant to the study.

3.2. Information quality across the sample

Figure 1 presents a summary of the distribution of percentage scores of the information quality from the web site assessments using the three evaluation tools. Few web sites scored highly for any of the three evaluation tools (i.e. scored ≥75%); however, few web sites performed badly (<25%). The majority of web sites were in the 25–49 or 50–74% range. Nineteen sites (37%) achieved a score of ≥50% using DISCERN; 25 sites (49%) managed a score of ≥50% using the Stroke Tool; and 30 sites (59%) attained a score of ≥50% or greater using HON.

Figure 1.

Distribution of percentage scores for the sample of web sites for the three evaluation tools.

3.3. Ranking of web sites according to each tool

Table 1 shows the percentage scores and the ranking of the web sites for each evaluation tool (columns 2–4) and the overall rank (column 1), based on the mean ranking across the five tools. The highest-ranked web sites were the Stroke Association, Bupa and Patient UK, and the lowest-ranked web sites were the Association of Speech and Language Therapists in Independent Practice, Aphasia Treatment and Stroke Folk. There was some disagreement in the rankings of some web sites, most noticeably between the quality scores and readability scores. For example, Everyday Health, ranked 8, 8 and 7 (DISCERN, HON Code and Stroke Tools, respectively) for information quality, but 26 (FRE) and 32 (FKGL) for readability.

3.4. Readability scores

The readability of the web sites was assessed using the FRE score and the FKGL. Figure 2 shows the distribution of the readability scores across the sample of web sites. Over 30% of web sites have a low score for the FRE, indicating a low level of reading ease. Only 4% of sites had a score of ≥70, indicating a high level of readability. Correspondingly, only 6% of web sites had a FKGL of <7, which has been said to be the average reading level of the general public, meaning that 94% (n = 48) of the web sites assessed would not be accessible to everyone. Table 2 (columns 6 and 7) show the scores and ranks for the individual web sites.

Figure 2.

Distribution of FRE scores and FKGL.

Table 2.

Spearman’s Rho Correlation Coefficient (and p value) for each pair of evaluation tools

Evaluation tool	HON	Stroke Tool	FRE	FKGL
DISCERN	0.704 (<0.001)	0.622 (<0.001)	0.043 (0.765)	−0.083 (0.560)
HON Code		0.494 (<0.001)	0.233 (0.100)	−0.150 (0.295)
Stroke Tool			0.116 (0.416)	−0.115 (0.420)
FRE				−0.873 (<0.001)

3.5. Internal reliability of evaluation tools

The three tools had acceptable levels of internal reliability: DISCERN (α = 0.915); HON Code (α = 0.860) and Stroke Tool (α = 0.922).

3.6. Concordance and correlation among the tools

The level of concordance among all of the tools was fair (W = 0.553; p < 0.001), but was much lower among the information quality tools (HON, DISCERN and Stroke Tool; W = 0.172; p < 0.001). The level of concordance was higher between the HON and DISCERN tools (W = 0.353; p < 0.001). Table 2 shows the level of correlation between each pair of tools: correlations between the web site evaluation tools and between the readability measures were significant, but correlations between the readability scores and information quality scores were not significant.

The lowest correlation was between DISCERN and FKGL. Lower FKGL indicates that a lower reading level is required, whereas higher scores for all the other tools indicate better performance, which is why correlations with FKGL are negative. The highest correlation was between the two readability scores. Regarding the information quality tools, the highest correlation was between DISCERN and HON Code, indicating a good level of correlation between these tools. The correlation between DISCERN and the Stroke Tool was moderate and the weakest correlation was between the HON Code and the Stroke Tool.

3.7. Feasibility of evaluation tools

Table 3 presents the time taken to assess the web sites. There was a significant difference among the three tools in the rankings of the mean time taken to assess the web sites (χ² = 15.06; degrees of freedom = 2; p = 0.001), the HON Code tool had the lowest mean ranking and the Stroke Tool had the highest mean ranking.

Table 3.

Descriptive statistics on time spent assessing web sites

	Time spent assessing web sites (minutes:seconds)
Evaluation tool	Mean	Minimum	Maximum	Standard deviation
DISCERN	04:33	01:10	10:38	01:47
HON Code	04:11	01:31	08:23	01:41
Stroke Tool	06:18	01:55	13:24	03:06

4. Discussion

4.1. Information quality of the web sites

Overall, the quality of information regarding stroke and SLD on the Internet was highly variable, in terms of overall quality, information content and readability. Some web sites had high quality, others were of low quality. This does not augur well for consumers of this information, and corresponds with previous studies that have evaluated web-based information on other conditions, for example, breast cancer [11, 16], multiple sclerosis [17] and Alzheimer’s disease [14, 15], which also found the quality of web-based information to be highly variable. Griffin et al. also found that criteria used to examine referencing practice and currency of information on stroke web sites were not met sufficiently [18]. The current study indicates that some web sites had high overall quality and good information content, for example, the Stroke Association web site scored 67, 70 and 92% using DISCERN, HON and the Stroke Tool respectively. However, other web sites had poor overall quality and limited content, for example, Aphasia Treatment scored 19, 13 and 20% for DISCERN, HON and the Stroke Tool, respectively. In terms of readability, some web sites, for example, NIH Senior Health, had high readability and required a low level of reading ability, while others, for example, About.com, had poor readability and required a high level of reading ability. However, some web sites had variable quality across the three domains, for example, although NIH Senior Health had the highest readability and required the lowest level of reading ability, the overall quality of information and the content was relatively poor. As well as providing a better understanding of the quality of stroke and SLD information on the Internet, this can inform healthcare professionals in their advisory role to PCF, as well as PCF themselves, about finding high-quality information on stroke and SLD online. The results of the study may also help to inform the developers of web sites.

There was some disagreement between the results from different tools. For example, although ‘Patient UK’ ranked third overall, and according to HON, it ranked 13 according to DISCERN and 20 according to the Stroke Tool. This is primarily because different tools measure different domains of quality, that is, DISCERN measures reliability and quality of information on treatment, whereas the Stroke Tool measures how well the web sites meet the information needs of their users.

4.2. To what extent do web sites meet the information needs of stroke PCF experiencing SLD?

Generally, the sample of web sites scored fairly well using the Stroke Tool, with 49% achieving a score of 50% or more, indicating that they meet a proportion of the information needs of stroke PCF experiencing SLD. It is important that the web sites had reasonable evaluations using the Stroke Tool, because it measures the extent to which they meet the information needs of PCF. The importance of relevant information to these people should not be underestimated. Indeed, even if the information is of high-quality as measured by generic tools, it may be of little use if it does not contain the information that users want. Some web sites performed very well, notably the Stroke Association and Chest, Heart and Stroke Scotland, but others performed very poorly. Several of these are relatively small web sites, and the hosting organization may not have the resources to provide fuller information, but are less likely to meet all the information needs specified.

The variability in the evaluations of the web sites was not unexpected, corresponding as it does with previous studies that have evaluated other conditions. For example, over half of the multiple sclerosis sites evaluated in one study scored <50% using the Multiple Sclerosis-specific tool [17]. In addition, only 28% of Breast Cancer sites met >50% of the criteria of their specifically designed tool [16]. Interestingly, 53% of Alzheimer’s disease sites that were examined achieved a score of >50% [14, 15]; however the small sample size (n = 15) limits the generalizability of these results.

The Stroke Tool can be considered as complementary to generic tools in that it measures different domains of information quality that may be of particular interest and relevance to stroke PCF experiencing SLD. It had high internal reliability, and a degree of correlation with other information quality tools, that is, web sites that provide better quality of information tend to have a higher information content. The Stroke Tool developed in this study permits a more thorough and systematic examination of the content of information on stroke and SLD web sites than was previously possible.

4.3. Readability of information on web sites

The information on the sample of web sites was generally not of a sufficiently low reading level for use by all members of the population. Very few (6%) scored a grade level of <7 according to the FKGL, when a maximum of grade 6 is the recommended level for healthcare information aimed at the general public, and especially for an older general public. This is comparable with previous studies that have measured the readability of healthcare information online and in paper format, and especially information relating to stroke and SLD. For example, one study found that Parkinson’s disease information is not at a suitable reading level [62], and another study made similar conclusions regarding information given to people suffering aphasia following a stroke [28]. The vast majority of studies also conclude that the readability of healthcare information is poor. However, the accessibility of information is an important issue, and even more so when the condition, in this case stroke, creates communication difficulties. This means that information may be generally harder to obtain for people who have experienced a stroke, but also that it is more important that the available information is presented in an accessible way. Sites that have high reading levels are not fully accessible, and if the accessibility of information forms one of the needs of the consumers, this means that web sites are not completely meeting information needs even if the quality of their information scores highly.

The low correlation that the readability scores had with other scores raises important issues regarding information quality and accessibility [5], in that sites with high-quality information may not necessarily be accessible to people of all reading levels, and similarly, information that is highly readable may not be of high quality. For example, Revolution Health (site 35) ranked highly against quality and content criteria (ranking 1, 1 and 8), yet much lower for readability (ranking 25 and 20). Conversely, HIN Senior Health (site 20) ranked fairly low in quality and content (33, 16 and 30), yet had the best readability scores of the sample. This highlights the importance of carrying out both quality and readability tests to enable a full picture of the web sites to be seen.

4.4. Feasibility of the evaluation tools

The times taken to evaluate the web sites varied greatly, ranging from 1 minute 10 seconds (on the Stroke Watch web site using DISCERN) to 13 minutes 24 seconds (on the Stroke Association web sites using the Stroke Tool). Generally, the Stroke Tool took longer to use than the others tools (for 57% of web sites), and some web sites took longer, for example, the Stroke Association, to evaluate than others. There are several factors to take into consideration when considering such results, including the number of questions in a tool and the size of the web site. Two web sites that took the longest to evaluate also performed the best against the Stroke Tool, indicating that they contain a large amount of relevant material. Although they may have taken a long to time to assess, this does not necessarily reflect badly on the tools. It is also possible that the first tool to be used in assessing a web site may take longer than subsequent tools, which may benefit to a certain degree from an increased familiarity with the web site and therefore may return shorter times for evaluation. For example, site 5 (NHS Kirklees) was assessed with the HON Code, followed by the Stroke Tool and DISCERN; the times taken using each tool decreased correspondingly. This risk was mitigated by always visiting appropriate pages, regardless of whether the relevant information has been gleaned by use of a previous tool, and by alternating the sequence of tools used to assess each web site across the sample.

4.5. Limitations

First, a limited sample of web sites and tools was used in this assessment. A more comprehensive selection of web sites would enable a more generalized conclusion to be made on the state of information quality relating to stroke and SLD on the Internet. However, it may in fact be impossible to ensure a complete picture of online information owing to the size of the Internet and its constantly changing nature. Only English language sites were examined in this study; it would be of interest to evaluate the situation of other languages as regards this information. After all, stroke is a global phenomenon, and one that affects large numbers of people worldwide.

The limited time and resources available for the study meant that each web site was assessed only once using each of the tools. A more reliable estimate of the quality of information, as well as the reliability of the tools, would be enabled if sites were assessed multiple times.

One researcher (R.S.), with no medical background, undertook the evaluations, owing to time and resource limitations. Individuals may have subjective responses towards specific sites, which can introduce subjectivity into evaluations. Assessments carried out by multiple researchers may give more reliable results. A further difficulty in carrying out research from the point of view of a particular PCF is that information needs may be individual. Even a site that has performed well according to the tools used may not have the particular information that one person is looking for, so that that the results of this study may not be transferable for every person’s need.

4.6. Recommendations for further research

Multiple assessments of web sites by the same, or other, researchers, would help to assess the test–retest and inter-rater reliability of the tools, respectively. A researcher with appropriate clinical knowledge would be helpful to assess the accuracy of information, or whether the sites meet clinical guidelines. As recommended by several other studies, incorporating the views of a user group itself may be useful at several stages of the investigation. This would ensure that appropriate criteria are used in the specifically developed tool; the Stroke Tool may be able to be redeveloped using that information. It may also be of interest to involve users in assessments themselves to evaluate what is most relevant to them.

5. Conclusions

The quality of information for PCF experiencing SLD after a stroke is highly variable, in terms of overall quality, information content and readability. Some web sites are of high quality, some are of poor quality across all three domains, while others are variable across the domains. Use of the Internet by stroke PCF is therefore recommended, but with caution. An awareness of indicators of quality may be helpful, as will guidance by healthcare professionals.

Different tools measure different domains of information quality and a range of information quality indicators may be useful in assessing a web site fully. The readability of online healthcare information remains an important issue. Similar to previous research examining the reading levels of health web sites, stroke and SLD sites are not alone in having poor readability, which has serious implications for the accessibility of information provided. Assessing the extent to which web sites include information that a user needs is important. The low correlation between information quality and content measures and readability measures suggests that it is important to assess all these domains to evaluate a web site fully. Even if a web site holds high-quality information that is relevant to users, it may not be accessible to the intended audience if it is not at an appropriate reading level.

References

Coulter

. Evidence based patient information. British Medical Journal 1998; 317(7153): 225–226.

Burns

. Information for health: An information strategy for the modern NHS 1998–2005. A national strategy for local implementation. Leeds: NHS Executive.

Domain Tools. Domain counts and internet statistics. Domain Tools, http://www.domaintools.com/internet-statistics/ (2010, accessed 17 May 2010).

Netcraft. December 2011 web server survey. Netcraft Ltd, http://news.netcraft.com/archives/2011/ (2012, accessed 03 May 2012).

Ghidella

Murray

Smart

McKenna

Worrall

Aphasia web sites: An examination of their quality and communicative accessibility. Aphasiology 2005; 19(12): 1134–1146.

Office for National Statistics. Internet access: Households and individuals 2009. Newport: Office for National Statistics, http://www.statistics.gov.uk/pdfdir/iahi0809.pdf (2009, accessed 6 August 2010).

Shepperd

Charnock

Gann

. Helping patients access high quality health information. British Medical Journal 1999; 319: 764–766.

Rees

Ford

Sheard

. Evaluating the reliability of DISCERN: A tool for assessing the quality of written patient information on treatment choices. Patient Education and Counseling 2002; 47(3): 273–275.

Gagliardi

Jadad

. Examination of instruments used to rate quality of health information on the internet: Chronicle of a voyage with an unclear destination. British Medical Journal 2002; 324(7337): 569–573.

10.

Eysenbach

Diepgen

. Towards quality management of medical information on the internet: Evaluation, labelling, and filtering of information. British Medical Journal, 1998; 317(7171): 1496–1500.

11.

Meric

Bernstam

Mirza

Hunt

Ames

Ross

Kuerer

Pollock

Musen

Singletary

. Breast cancer on the world wide web: Cross sectional survey of quality of information and popularity of web sites. British Medical Journal 2002; 324(7337): 577–581.

12.

Griffiths

Christensen

. Quality of web based information on treatment of depression: Cross sectional survey. British Medical Journal 2000; 321(7275): 1511–1515.

13.

Shedlosky-Shoemaker

Sturm

Saleem

Kelly

. Tools for assessing readability and quality of health-related web sites. Journal of Genetic Counseling 2009; 18(1): 49–59.

14.

Bouchier

Bath

. Evaluation of web sites that provide information on Alzheimer’s disease. Health Informatics Journal 2003; 9(1): 17–31.

15.

Bath

Bouchier

. Development and application of a tool designed to evaluate web sites providing information on Alzheimer’s disease. Journal of Information Science 2003; 29(4): 279–297.

16.

Hsu

W-C

Bath

. Development of a patient-oriented tool for evaluating the quality of breast cancer information on the Internet. In: Andersen

Klein

Schulz

Aarts

Mazzoleni

(eds) eHealth beyond the horizon – get IT there. Proceedings of MIE2008. Studies in Health Technology and Informatics, Vol. 136. Lansdale, PA: IOS Press, 2008, pp. 297–302.

17.

Harland

Bath

. Assessing the quality of web sites providing information on multiple sclerosis: Evaluating tools and comparing sites. Health Information Journal 2007; 13(3): 207–221.

18.

Griffin

McKenna

Worrall

. Stroke education materials on the World Wide Web: An evaluation of their quality and suitability. Topics in Stroke Rehabilitation 2004; 11(3): 29–40.

19.

Kim

Eng

Deering

Maxfield

. Published criteria for evaluating health related web sites: Review. British Medical Journal 1999; 318(7184): 647–649.

20.

Sims

Muyderman

. Mitochondria, oxidative metabolism and cell death in stroke. Biochimica et Biophysica Acta 2010; 1802: 80–91.

21.

Low

Payne

Roderick

. The impact of stroke on informal carers: A literature review. Social Science and Medicine 1999; 49: 711–725.

22.

Flynn

MacWalter

Doney

. The cost of cerebral ischaemia. Neuropharmacology 2008; 55: 250–256.

23.

Rothwell

Coull

Silver

Fairhead

Giles

Lovelock

Redgrave

Bull

Welch

Cuthbertson

Binney

Gutnikov

Anslow

Banning

Mant

Mehta

; Oxford Vascular Study. Population-based study of event-rate, incidence, case, fatality, and mortality for all acute vascular events in all arterial territories (Oxford Vascular Study). Lancet 2005; 366: 1773–1783.

24.

Donnan

Fisher

Macleod

Davis

. Stroke. Lancet 2008; 371: 1612–1623.

25.

Parr

Byng

Gilpin

. Talking about aphasia: Living with loss of language after a stroke. Maidenhead: Open University Press, 1997.

26.

Hanger

Mulley

. Questions people ask about stroke. Stroke 1993; 24: 536–538.

27.

Crystal

Varley

. Introduction to language pathology. London: Whurr, 1998.

28.

Aleligay

Worrall

Rose

. Readability of written health information provided to people with aphasia. Aphasiology 2008; 22(4): 383–407.

29.

Rose

Worrall

McKenna

Hickon

Hoffman

. Do people with aphasia receive written stroke and aphasia information? Aphasiology 2009; 23(3): 364–392.

30.

Hanif

Raghabir

. Multifaceted Web resources for stroke. Journal of Stroke and Cerebrovascular Diseases 2008; 17(4): 218–225.

31.

Hoffmann

McKenna

. Analysis of stroke patients’ and carers’ reading ability and the content and design of written materials: Recommendations for improving written stroke information. Patient Education and Counseling 2006; 60: 286–293.

32.

Worrall

Rose

Howe

McKenna

Hickson

. Developing an evidence-base for accessibility for people with aphasia. Aphasiology 2007; 21(1): 124–136.

33.

Tooth

Hoffmann

. Patient perceptions of the quality of information provided in a hospital stroke rehabilitation unit. British Journal of Occupational Therapy 2004; 67(3): 111–117.

34.

Eames

McKenna

Worrall

Read

. The suitability of written education materials for stroke survivors and their carers. Topics in Stroke Rehabilitation 2003; 10(3): 70–83.

35.

Brennan

Worrall

McKenna

. The relationship between specific features of aphasia-friendly written material and comprehension of written material for people with aphasia: An exploratory study. Aphasiology 2005; 19(8): 693–711.

36.

Wiles

Pain

Buckland

McLellan

. Providing appropriate information to patients and carers following a stroke. Journal of Advanced Nursing 1998; 28(4): 794–801.

37.

Michallet

Le Dorze

Teatreault

. The needs of spouses caring for severely aphasic persons. Aphasiology 2001; 15(8): 731–747.

38.

Rodgers

Bond

Curless

. Inadequacies in the provision of information to stroke patients and their families. Age and Ageing 2001; 30: 129–133.

39.

Wachters-Kaufmann

Schuling

The

Meyboom-de Jong

. Actual and desired information provision after a stroke. Patient Education and Counseling 2005; 56: 211–217.

40.

Le Dorze

Tremblay

Croteau

. A qualitative longitudinal case study of a daughter’s adaptation process to her father’s aphasia and stroke. Aphasiology 2009; 23(4): 483–502.

41.

van Veenendaal

Grinspun

Adriaanse

. Educational needs of stroke survivors and their family members, as perceived by themselves and by health professionals. Patient Education and Counseling 1996; 28: 265–276.

42.

Rose

Worrall

McKenna

Hickon

Hoffman

. Do people with aphasia receive written stroke and aphasia information? Aphasiology 2009; 23(3): 364–392.

43.

Knight

Worrall

Rose

. The provision of health information to stroke patients within an acute hospital setting: What actually happens and how do patients feel about it? Topics in Stroke Rehabilitation 2006; 13(1): 78–97.

44.

Smith

Forster

House

Knapp

Wright

Young

. Information provision for stroke patients and their caregivers. Cochrane Database of Systematic Reviews 2008; 3.

45.

O’Connell

Baker

. Managing as carers of troke survivors: Strategies from the field. International Journal of Nursing Practice 2004; 10: 121–126.

46.

Garrett

Cowdell

. Information needs of patients and carers following stroke. Nursing Older People 2005; 17(6): 14–16.

47.

Draper

Brocklehurst

. The impact of stroke on the well-being of the patient’s spouse: An exploratory study. Journal of Clinical Nursing 2007; 16: 264–271.

48.

Avent

Glista

Wallace

Jackson

Nishioka

Yip

. Family information needs about aphasia. Aphasiology 2005; 19(3): 365–375.

49.

O’Connell

Baker

Prosser

. The educational needs of caregivers of stroke survivors in acute and community settings. Journal of Neuroscience Nursing 2003; 35(1): 21–28.

50.

Ward

Cavanagh

. A descriptive study of the self-perceived needs of carers for dependants with a range of long-term problems. Journal of Public Health Medicine 1997; 19(3): 281–287.

51.

Jarvis

Worth

. Meeting carers’ information needs. Community Practitioner 2005; 78(9): 322–326.

52.

Cook

Pierce

Hicks

Steiner

. Self-care needs of caregivers dealing with stroke. Journal of Neuroscience Nursing 2006; 38(1): 31–36.

53.

Hinojosa

Rittman

. Stroke caregiver information needs: Comparison of mainland and Puerto Rican caregivers. Journal of Rehabilitation Research and Development 2007; 44(5): 649–658.

54.

Docherty

Owens

Asadi-Lari

Petchey

Williams

Carter

. Knowledge and information needs of informal caregivers in palliative care: A qualitative systematic review. Palliative Medicine 2008; 22: 153–171.

55.

Nielsen. Nielsen insights: Internet. The Nielsen Company, http://en-us.nielsen.com/rankings/insights/rankings/internet (2010, accessed 18 May 2010).

56.

Lawrence

Giles

. Searching the world wide web. Science 1998; 280: 98–100.

57.

Jansen

Spink

. And analysis of web searching by European AlltheWeb.com users. Information Processing and Management 2005; 41(2): 361–381.

58.

Pandolfini

Impicciatore

Bonati

. Parents on the Web: Risks for quality management of cough in children. Pediatrics 2000; 105(1): e1.

59.

Irvin

Guzmán

Bombardier

. Surfing for back pain patients: The nature and quality of back pain information on the Internet. Spine 2001; 26(5): 545–557.

60.

Fitzsimmons

. Readability of patient orientated geriatric health information on the Internet. In: Communications to the spring meeting of the Geriatrics Society: Programme of abstracts, 22–23 April 2010, Edinburgh. London: British Geriatrics Society, p. 35.

61.

Microsoft Office, Test your document’s readability. Microsoft Corporation, http://office.microsoft.com/en-us/word-help/test-your-document-s-readability-HP010148506.aspx (2010, accessed 23 August 2010).

62.

Hulley

. How readable is patient orientated Parkinson’s disease information on the Internet? In: Communications to the spring meeting of the Geriatrics Society: Programme of abstracts, 22–23 April 2010, Edinburgh. London: British Geriatrics Society, p. 95.

63.

Hargrave

Bouffet

. Quality of health information on the Internet in pediatric neuro-oncology. Neuro-oncology 2006; 8: 175–182.

64.

Bryman

Cramer

. Quantitative data analysis with SPSS for windows: A guide for social scientists. London: Routledge, 1997.