How well do websites concerning children’s anxiety answer parents’ questions about treatment choices?

Abstract

The goals of this study were to evaluate the quality of information concerning anxiety disorders in children that is available on the Internet and to evaluate changes in the quality of website information over time. The authors identified websites addressing child anxiety disorders (N = 26) using a Google search and recommendations from an expert in child anxiety. Each website was evaluated on the extent to which it addressed questions that parents consider important, the quality of information, and the reading level. All websites provided adequate information describing treatment options; however, fewer websites had information addressing many questions that are important to parents, including the duration of treatment, what happens when treatment stops, and the benefits and risks of various treatments. Many websites provided inadequate information on pharmacological treatment. Most websites were of moderate quality and had more difficult reading levels than is recommended. Five years after the initial assessment, authors re-analyzed the websites in order to investigate changes in content over time. The content of only six websites had been updated since the original analysis, the majority of which improved on the three aforementioned areas of evaluation. Websites could be strengthened by providing important information that would support parent decision-making.

Keywords

Internet child anxiety disorders parent information quality of information treatment decision-making

Introduction

Large-scale epidemiological research examining the prevalence of psychiatric disorders among children and adolescents demonstrates high rates of past-year anxiety disorders (Costello, Egger, & Angold, 2005; Merikangas, Nakamura, & Kessler, 2009; Merikangas et al., 2010). In a nationally representative study conducted in the United States, anxiety disorders were found to be the most prevalent lifetime psychiatric disorder among adolescents (31.9%), followed by behavior disorders (19.1%), mood disorders (14.3%), and substance use disorders (11.4%) (Merikangas et al., 2010). When parents are considering treatment for their child’s anxiety disorder, they are faced with complex decisions. Psychosocial, pharmacological, and combined treatments are available for each type of anxiety disorder (Klein, 2009). In order to make an informed decision about the most appropriate treatment for their child, parents may wish to consider the advantages and disadvantages of each treatment, as well as the practical constraints such as the cost, time involvement, and duration of treatment. Unfortunately, balanced and evidence-based information pertaining to the treatment of child anxiety disorders is not often readily accessible to parents.

The public is increasingly relying on the Internet as a primary source of information for many things, including their health. Among adults in Europe and the United States, 52% and 61%, respectively, have searched for health information online (Fox & Jones, 2009; Kummervold et al., 2008). Similar findings exist in Canada, with more than one-third of Canadian adults reporting the use of the Internet to search for health information (Underhill & McKeown, 2008). Research also corroborates the increased use of the Internet to search for mental health information. Community studies in Australia indicate that young and middle-aged adults see the Internet as a helpful source of mental health information (Leach, Christensen, Griffiths, Jorm, & Mackinnon, 2007; Oh, Jorm, & Wright, 2009). A review of parents’ use of the Internet indicated that the majority of parents use the Internet for information and social support concerning their parenting strategies and their children’s health (Plantin & Daneback, 2009). The authors of this study also found that parents are most likely to find this information through search engines such as Google, as opposed to recommendations from friends or advertisements (Plantin & Daneback, 2009). More recently, it has been shown that 78% of parents of children attending psychiatric outpatient services use the Internet to search for information concerning symptoms, diagnoses, and treatment of their children’s mental health problems (Turner, Kabashit, Guthriet, Burket, & Turner, 2011). Furthermore, results of this study indicated that 50% of parents reported highly valuing the Internet as a source of mental health information, second only to contact with the physician in their psychiatric clinic (Turner et al., 2011). Online information has been shown to enhance both the quality and quantity of information given by health professionals, support patients in medical decision-making, as well as limit the need for in-person professional consultation (Couper et al., 2010; Gray, Klein, Noyce, Sesselberg, & Cantrill, 2005). The ease of access, anonymity, and vast quantity of information on the Internet allow for the potential to help parents make informed decisions about their child’s treatment. However, in order to do this effectively, websites should provide evidence-based information in an understandable way that answers questions considered important by parents. There is also a downside to information available on the Internet. Methods of quality control, such as the peer review process used by journals, are typically absent. Furthermore, anyone can post information on the Internet, including persons with a commercial or other motivation, and it may be difficult to determine the source of the information. As a consequence, information found on the Internet is highly variable in quality as compared to information gained from professional consultation, or information found in published books or journal articles.

A review of studies examining the quality of websites addressing mental health problems found that 23 out of 31 studies included in the review were of poor quality (Reavley & Jorm, 2011). Information provided in websites concerning mood disorders was generally rated as moderate to high quality (e.g. Barnes et al., 2009; Ferreira-Lay & Miller, 2008). However, inadequate information was evident in websites concerning anxiety disorders (Isper, Dewing, & Stein, 2007; Khazaal, Fernandez, Cochand, Reboh, & Zullino, 2008), pathological gambling (Khazaal, Chatton, Cochand, & Jermann, et al., 2008), cannabis use (Khazaal, Chatton, Cochand, & Zullino, 2008a), cocaine use (Khazaal, Chatton, Cochand, & Zullino, 2008b), eating disorders (Murphy, Frost, Webster, & Schmidt, 2004), psychological trauma (Bremner, Quinn, Quinn, & Veledar, 2006), suicide prevention (Jorm, Fisher, & Oh, 2010), and schizophrenia (Kisely, Ong, & Takyar, 2003). Of further concern, it has been shown that websites containing inadequate information are not significantly changed to improve this information over time. For example, Coquard, Fernandez, Zullino, and Khazaal (2012) re-assessed alcohol-related websites one year after their original analysis and found a lack of change in information.

Although there is considerable evidence suggesting that websites tailored to the mental health problems of adults are of inadequate quality, there is a lack of information concerning the quality of websites focused on the mental health problems of children and adolescents. Furthermore, previous research in the mental health area has focused on the use of standardized website evaluation tools as the sole method of website evaluation, and has not evaluated how well websites answer questions that are considered important by the individuals who are seeking information.

Research conducted by Mak, Hiebert-Murphy, Walker, and Altman (2014) identified questions concerning the treatment of anxiety disorders in children that parents considered to be important. The goal of the current study was to evaluate common websites providing information for parents about anxiety disorders in children with regard to the following criteria: (1) the extent to which they answer questions that parents have identified as being important in making decisions about the treatment of their child’s anxiety; (2) the quality of information they provide, based on the standardized evaluation measure, the DISCERN (Charnock, Shepperd, Needham, & Gann, 1999); and (3) the readability of the information they provide based on the Flesch-Kincaid Grade Level (FKGL) measurement (Ley & Florio, 1996). An additional goal of this study was to re-evaluate websites in 2013 that were originally evaluated in 2008, in order to assess changes in information quality over time.

Methods

Website selection

Google is the most widely used search engine in Canada and the United States; therefore, we used Google.ca (Canada) and Google.com (United States) to select websites for evaluation. Information from industry sources indicates that Google was used in December 2009 for 67% of searches worldwide while its closest competitor, Yahoo, was used for 7% (Sullivan, 2010). A recent report supports that Google remains the most popular search engine, with usage greatly exceeding that of Yahoo and Bing (McGee, 2013). We conducted an initial search in August 2008, with the keywords “Child Anxiety Disorder” in Google.ca. The first 30 websites were evaluated if they did not meet any of the exclusion criteria. Exclusion criteria included duplicate websites, excerpts from books and articles, websites advertising books or products (i.e. Amazon.ca), and websites containing too little information to be viewed as a comprehensive resource (defined as less than 500 words). We chose 30 sites as the target, as research investigating online consumer behavior indicates that 62% of search engine users click on a search result within the first 10 sites, and 90% of search engine users click on a result within the first 30 sites (iProspect, 2006). After applying the exclusion criteria, the initial Google.ca search that retrieved 30 websites resulted in the analysis of 14 websites. Following this initial selection, we conducted a Google.com search to identify websites that were not present in the Canadian Google search. Although there was substantial overlap, five new websites emerged and were evaluated. Finally, the second author (J.W.) who specializes in the treatment of anxiety disorders in children, adolescents, and adults, recommended seven additional websites based on his experience in the field, resulting in a total of 26 websites. The additional seven websites are known among professionals as providing information for public use and may be identified later in the results listings for Internet searches focused on child anxiety disorders. The first and third authors (K.R. and K.W.) evaluated each website independently in August 2008. Websites were not evaluated blindly; however, the authors did not have any biases toward particular websites, and they evaluated websites according to specific criteria. The websites were evaluated again in 2013 to establish whether there were changes in content. When there were changes, website information was re-assessed (between August and October 2013) based on content areas, DISCERN score, and FKGL reading level score, in order to examine changes that occurred since the initial evaluation.

Content area evaluation

Research by Mak et al. (2014) informed the development of questions (items) that were evaluated across websites in the present study. This qualitative study assessed parental views concerning the importance of different types of information regarding the treatment of their child’s anxiety disorder. Information areas that emerged as highly important to parents in this research translated into 40 items that authors grouped into three thematic areas: characteristics of treatment (see Table 2), information concerning pharmacological treatment (see Table 3), and administrative information concerning treatment (see Table 4). Based on the initial evaluation of websites, authors identified one further area for evaluation, the additional resources for parents provided by the website. This area included self-help resources (e.g. suggestions for ways in which parents can help to manage their child’s anxiety at home or at school), other resources for parents (e.g. links to other websites and recommendations for educational books and articles), and interactive tools (e.g. discussion forums, videos, and links to social media such as Facebook and Twitter) (see Table 4).

The authors copied information from each website addressing each question, excluding external links, into a Microsoft Word word-processing file for further evaluation. Information on each item (N = 40) was assessed on a 5-point rating scale ranging from 1 = no information to 3 = adequate information with basic detail to 5 = complete and comprehensive information with detailed discussion (Hall, Howard, & McCaffery, 2008). An overall information score for each website was calculated by summing all of the item scores. Ratings of each item across the 26 websites evaluated were averaged to create mean information scores. The authors assessed interobserver agreement by comparing the ratings of two evaluators (K.R. and K.W.) for each question using the intraclass correlation coefficient (Shrout & Fleiss, 1979).

Measures of website quality

The overall website quality was assessed using the DISCERN scale (Charnock et al., 1999). The DISCERN is a 16-item instrument that is used to assess the quality of written consumer health information concerning treatment choices. The DISCERN scale is widely used and has good psychometric properties (Breckons, Jones, Morris, & Richardson, 2008; Charnock et al., 1999; Khazaal, Chatton, Zullino, & Khan, 2012). Whereas the content area evaluation focused on specific items that parents rated as important pertaining to the treatment of their child’s anxiety disorder, the DISCERN scale contains more general questions pertaining to treatment of health conditions and is a standardized and validated indicator of website quality that has been used in multiple areas of website evaluation and health information research. The advantage of including the DISCERN scale in this study is that it increases the validity of findings, in addition to the degree to which results can be compared with other website evaluation research. Each item of the DISCERN receives a score of 1–5, with 1 indicating no success in meeting the item criteria, 2–4 indicating partial success in meeting the item criteria, and 5 indicating complete fulfillment of the item criteria (see Charnock et al. (1999) for detailed scoring procedures). For each website, the scores of the 16 items were averaged to produce an overall website score out of 5, with 1 indicating low quality with extensive shortcomings, 3 indicating moderate quality with potentially important shortcomings, and 5 indicating high quality with limited shortcomings. To give an indication of areas of strength and weakness across the 26 websites, we created a mean score for each DISCERN item.

Readability

The authors assessed the reading level of each website using the FKGL score, which is available in the word-processing program Microsoft Word (Ley & Florio, 1996). The FKGL measurement analyzes the relative numbers of syllables, words, and sentences in a passage of text to calculate a readability score, which indicates the grade level at which a person would be able to understand the text.

Results

Characteristics of websites

The websites evaluated in this study are described in Table 1. The search strategy identified a variety of websites, including websites from national organizations (e.g. Anxiety and Depression Association of America), websites for parents (e.g. Child Anxiety Network), hospital-based websites (e.g. Children’s Hospital Boston), specialty clinic websites (e.g. Northern County Psychiatric Associates), and university websites (e.g. Duke University Program in Child Affective and Anxiety Disorders). The reading grade level (FKGL) of website content ranged from grade 10.4 to 15.9, demonstrating a wide range in reading levels. The number of content areas addressed (out of 40) ranged from 4 to 30, illustrating great variability in the quantity of information across websites. The degree to which websites answered questions considered important by parents also varied greatly, with overall child anxiety information scores ranging from 46.0 to 105.0 out of a possible total of 200. The total information score for each website correlated moderately with the standardized tool that was designed to assess the quality of health information, the DISCERN (r = .61, p < .01). According to the DISCERN evaluation, there was great variability in the quality of information presented on websites, with scores ranging from 1.9 (low quality) to 3.9 (moderate quality). The assessment of interobserver agreement indicated that the two independent evaluators (K.R. and K.W.) had a high level of agreement on the total information score for each website (r = .93, p < .01).

Table 1.

Characteristics of the 26 websites evaluated in the initial evaluation (2008).

Website	Order in search results^a	Reading grade level	Overall child anxiety information score^b	Number of content areas covered^c	DISCERN score^d
American Academy of Child and Adolescent Psychiatry	US29	12.3	78.0	17	3.0
Anxiety Disorders Association of America	Ca22, US5	11.9	100.0	29	3.5
Anxiety Disorders Association of Canada	S	12.5	91.0	23	3.8
Anxiety UK	S	10.9	127.0	30	3.5
Association for Behavioral and Cognitive Therapies	S	13.2	85.5	24	3.0
Canadian Family Physician	Ca5	13.4	89.0	27	3.5
Canadian Mental Health Association-British Columbia	Ca4	10.6	105.0	25	3.9
Canadian Psychiatric Association	S	10.8	46.0	4	2.5
Canadian Psychological Association	S	11.7	59.0	10	2.8
Centre of Knowledge on Healthy Child Development	Ca6	11.5	68.0	17	3.8
Child Anxiety Network	Ca1, US1	12.8	95.0	20	3.1
Child Development Institute	Ca23, US20	10.8	52.0	12	2.2
Children’s Hospital Boston	US10	14.2	51.0	13	2.0
Children’s Mental Health Ontario	Ca3	11.2	61.0	13	3.2
Duke University Program in Child Affective and Anxiety Disorders	US27	14.9	60.0	13	2.7
Keep Kids Healthy	Ca6, US4	15.2	69.5	18	2.6
Mental Health America	US25	13.0	46.0	15	3.2
Merck Manuals Online Medical Library	Ca10, US19	12.0	76.5	22	2.7
Native Remedies	Ca15, US28	14.0	67.0	24	2.0
Northern County Psychiatric Associates	Ca24, US8	10.4	90.5	25	2.8
Selective Mutism Group Childhood Anxiety Network	Ca15, US11	11.5	81.0	16	3.3
Shyness Institute	S	12.7	81.0	19	3.1
Solutions for Child Problems	Ca7	11.0	59.5	10	1.9
Substance Abuse and Mental Health Services Administration (US)	Ca16, US6	12.7	77.5	21	3.9
UCSD Veryshy.com	S	15.9	54.0	8	2.8
Worrywise Kids	US30	12.0	94.0	27	3.1

UCSD: University of California San Diego.

Order is from the Google.ca (Canadian) search or from the Google.com (United States) search. Some sites appeared in both searches. Sites labeled with the prefix Ca were generated in the Canadian search and sites labeled with the prefix US were generated in the US search. Sites labeled with the prefix S were selected to be evaluated by the second author (J.W.) who is a specialist in anxiety disorders, based on sites that may be recommended by professionals.

The overall information score is a sum of the content area scores on a 1–5 rating across 40 content areas (minimum possible score of 40, maximum possible score of 200).

Maximum number of topics covered = 40.

The DISCERN score is a 1–5 rating averaged across 16 questions.

Content area evaluation

A description of the amount and quality of information provided by websites to address parents’ questions is provided in Tables 2 to 4. These tables present the proportion of websites providing information on that item, the mean information score (1–5) with the 95% confidence interval (CI), and the intraclass correlation coefficient comparing two raters (K.R. and K.W.). The mean score was based on the ratings of K.R. across the 26 websites. All of the websites contained information concerning treatment choices and methods of treatment. On average, these items were adequately addressed across websites, with mean information scores of 3.8 (treatment choices) and 3.4 (treatment methods). There was great variability in the degree to which websites addressed questions concerning more specific characteristics of treatment. Ratings of information about the specific roles of the child, health-care professional, and parent during treatment; the goals of treatment; the advantages of treatment; treatment effectiveness; how treatment works; and the format of treatment ranged from 1.8 to 3.0. Questions pertaining to what happens when treatment stops, effectiveness of treatment across childhood and adolescence, disadvantages of treatment, side effects of treatment, and the length of time that treatment takes to produce results demonstrated even lower mean ratings across websites, ranging from 1.3 to 1.7 (see Table 2).

Table 2.

Amount of information on the characteristics of treatment for child anxiety disorders across the websites.

Items	Proportion of websites with information (N = 26)	Mean information score (95% CI)	Intraclass correlation coefficient comparing two raters
Treatment choices	100%	3.8 [3.4–4.2]	.82
Treatment methods	100%	3.4 [2.9–3.8]	.80
Format of treatment	88%	3.0 [2.6–3.5]	.87
How treatment works	85%	3.0 [2.5–3.5]	.91
Effectiveness of treatment	85%	2.6 [2.2–3.1]	.77
Advantages of treatment	73%	2.3 [1.8–2.8]	.75
What health-care provider will be doing in treatment	69%	2.5 [2.0–3.1]	.80
Goal of treatment	65%	2.3 [1.8–2.8]	.74
Parent’s role in treatment	62%	2.0 [1.5–2.4]	.82
What child will be doing in treatment	54%	2.2 [1.6–2.7]	.95
Child’s role in treatment	46%	1.8 [1.3–2.2]	.86
Length of time to produce results	46%	1.7 [1.3–2.0]	.81
Side effects	46%	1.7 [1.3–2.0]	.89
Duration of treatment	42%	1.7 [1.3–2.1]	.78
Disadvantages of treatment	35%	1.6 [1.2–1.9]	.95
What parent will be doing in treatment	35%	1.6 [1.2–2.0]	.95
Effectiveness of treatment across ages	27%	1.5 [1.1–1.9]	.97
What happens when treatment stops	27%	1.3 [1.0–1.4]	.66
Generalizability of treatment skills	23%	1.4 [1.1–1.8]	.93

CI: confidence interval.

Information score is on a 5-point rating scale ranging from 1-no information to 3-adequate information with basic detail to 5-complete and comprehensive information.

All correlations were statistically significant at the p < .001 level.

Table 3.

Information concerning pharmacological treatment for child anxiety disorders: amount of information on each item across the websites.

Items	Proportion of websites with information (N = 26)	Mean information score (95% CI)	Intraclass correlation coefficient comparing two raters
Medications used to treat anxiety	69%	2.8 [2.1–3.4]	.87
Advantages/Disadvantages of medication	62%	2.2 [1.7–2.6]	.93
Common side effects of medication	42%	1.7 [1.3–2.1]	.89
How doctor decides on medication	38%	1.8 [1.3–2.2]	.86
What happens when medication is stopped	27%	1.5 [1.1–1.9]	.86
Risk for dependency on medication	27%	1.3 [1.1–1.6]	.82
Uncommon side effects of medication	19%	1.3 [1.0–1.5]	.85
Limits in knowledge about the effects of medication on children	19%	1.4 [1.0–1.7]	.99
Amount of medication and how often	15%	1.4 [0.9–1.8]	.87
How long medication has been used with children	8%	1.1 [1.0–1.2]	.80

CI: confidence interval.

Information score is on a 5-point rating scale ranging from 1-no information to 3-adequate information with basic detail to 5-complete and comprehensive information. All correlations were statistically significant at the p < 001 level.

Table 4.

Administrative information concerning treatment for child anxiety disorders.

Items	Proportion of websites with information (N = 26)	Mean information score (95% CI)	Intraclass correlation coefficient comparing two raters
Training and profession of treatment provider	69%	2.6 [2.0–3.2]	.85
Which treatment option is recommended for a particular child and why	54%	1.9 [1.6–2.3]	.86
Location of treatment	45%	2.2 [1.5–2.8]	.87
Cost of treatment	38%	1.9 [1.3–2.5]	.91
Health-care provider’s experience in treating anxiety disorders	31%	1.5 [1.1–1.8]	.79
Frequency of treatment	23%	1.4 [1.0–1.8]	.69
When treatment will occur	12%	1.3 [0.9–1.6]	.69
How will parent be informed on child’s progress throughout treatment	8%	1.1 [1.0–1.1]	.92
Additional resources for parents
Other resources	81%	3.3 [2.7–3.9]	.89
Interactive tools	50%	2.2 [1.7–2.8]	.91
Self-help	42%	1.9 [1.4–2.5]	.82

CI: confidence interval.

Table 3 describes information concerning the pharmacological treatment of child anxiety disorders. Mean information scores on these items ranged from 1.1 (how long medication has been used with children) to 2.8 (medications used to treat anxiety). Of immediate concern, only 42% of websites addressed the common side effects of medication, doing so poorly, with very little information (mean information score of 1.7), and only 19% of websites included information about uncommon side effects of medication (mean information score of 1.3). More than half of the analyzed websites failed to include information about how the doctor decides on medication treatment for anxiety disorders in children, what might happen when the medication treatment is stopped, risks for dependency on medication, and any limits in the evidence on the effects of medication on children and adolescents.

The administrative information regarding treatment and the additional resources for parents that websites provided were also assessed (see Table 4). Administrative information included the training and profession of the treatment provider and their expertise in treating child anxiety disorders, as well as specific information regarding the cost, location, and frequency of treatment, scheduling treatment sessions, which treatment option is recommended for a particular child and why, and how parents will be informed about their child’s progress in treatment. Overall, websites provided limited information concerning administrative questions that parents raised about their child’s treatment, with mean ratings ranging from 1.1 to 2.6.

The availability of additional resources for parents, including self-help, other resources, and interactive tools, varied considerably across websites, with more than half of the websites analyzed not providing any information on self-help resources. The comparison of ratings across the independent raters (Tables 2 to 4) suggests that there was an acceptable level of agreement between the two raters on each content area.

Information quality evaluation

To further examine how website information concerning child anxiety could be strengthened, it is useful to consider the standardized DISCERN scale criteria. Table 5 lists the criteria of the DISCERN scale for written patient information concerning treatment choices and the average scores across websites. Most websites clearly stated their aims, resulting in an average score of 4.5 across websites. Most websites also provided adequate information concerning more than one possible treatment option (average score of 3.8). Many websites did not report the sources of information that were used to create the website (average score of 3.2), nor did they indicate the date that the specific information was produced (average score of 2.7). It would be helpful to parents if more websites addressed areas of uncertainty regarding treatment, how treatment choice may affect the child’s overall quality of life, what would happen if no treatment were used, and the risks associated with each treatment (average scores ranging from 1.4 to 1.8). The intraclass correlation coefficient suggests that there was an acceptable level of agreement between the two raters on each criteria of the DISCERN.

Table 5.

Mean scores on the items of the DISCERN scale across the websites.

DISCERN question	Mean Score (95% CI)	Intraclass correlation coefficient comparing two raters
Are the aims clear?	4.5 [4.1–4.9]	.80
Does it achieve its aims?	3.8 [3.4–4.2]	.80
Is it relevant?	3.9 [3.5–4.3]	.54
Is it clear what sources of information were used to compile the publication (other than the author or producer)?	3.2 [2.5–3.8]	.81
Is it clear when the information used or reported in the publication was produced?	2.7 [2.0–3.4]	.81
Is it balanced and unbiased?	3.9 [3.4–4.3]	.50
Does it provide details of additional sources of support and information?	3.9 [3.3–4.6]	.70
Does it refer to areas of uncertainty?	1.4 [1.1–1.8]	.80
Does it describe how each treatment works?	3.2 [2.7–3.6]	.66
Does it describe the benefits of each treatment?	2.3 [1.8–2.8]	.82
Does it describe the risks of each treatment?	1.8 [1.4–2.2]	.82
Does it describe what would happen if no treatment is used?	1.7 [1.2–2.2]	.91
Does it describe how the treatment choices affect overall quality of life?	1.6 [1.3–1.9]	.80
Is it clear that there may be more than one possible treatment choice?	3.8 [3.3–4.2]	.54
Does it provide support for shared decision-making?	2.8 [2.3–3.3]	.63
Based on the answers to all of the above questions, rate the overall quality of the publication as a source of information about treatment choices	3.0 [2.7–3.3]	.80

CI: confidence interval.

Each question is rated on a 5-point scale from 1-question criterion not fulfilled to 2–4-criterion met to some extent to 5-criterion completely fulfilled. All correlations were statistically significant at the p < .05 level.

Website re-assessment

As indicated above, K.R. re-evaluated websites that were initially analyzed in August 2008 from August 2013 to October 2013. The text of the 26 websites collected in the initial evaluation was compared to the current information on the websites. K.R. examined the changes in website content, paying attention to the revised copyright dates indicated on websites. The content of only six websites had been changed since the initial evaluation, and a revised copyright date had been posted. These six websites were re-evaluated according to the content areas, the DISCERN scale criteria, and the readability measure (see Table 6).

Table 6.

Revised website characteristics: 2013 re-assessment (N = 6).

Website	Date updated	Reading grade level	Overall child anxiety information score^a	Number of content areas covered^b	DISCERN score^c
American Academy of Child and Adolescent Psychiatry	2013	10.9	96.0	27	3.3
Anxiety and Depression Association of America	2012–2013	10.6	114.0	32	3.9
Association of Behavioral and Cognitive Therapies	2012	12.0	125.0	27	4.1
Children’s Hospital Boston	2012	11.4	125.0	25	3.8
Mental Health America	2013	11.4	59	13	2.7
Worrywise Kids	2013	11.5	99	23	3.4

The overall information score is a sum of the content area scores on a 1–5 rating across 40 content areas (minimum possible score of 40, maximum possible score of 200).

Maximum number of topics covered = 40.

The DISCERN score is a 1–5 rating averaged across 16 questions.

The date of website revisions ranged from 2012 to 2013. Overall, the websites that were revised after the initial evaluation date improved in all of the areas assessed. With regard to the FKGL score, the reading grade levels of website information decreased in all of the websites examined. The overall child anxiety information scores improved from initial evaluation to re-evaluation, with the greatest improvement being evident for Children’s Hospital Boston, increasing from 51.0 to 125.0. When examining specific content areas that improved across the revised websites, it was evident that websites enhanced their information on the characteristics of treatment for child anxiety disorders, including treatment choices, effectiveness of treatment, and the roles of parents and children during treatment. Websites also improved in providing information on the common and uncommon side effects of treatment, in addition to enhancing their interactive tools, self-help information, and other resources for parents. With the exception of two websites, websites also improved in addressing more content areas than they had in the initial analysis. Finally, for all but one website, DISCERN scale scores improved across the revised websites. Specific areas of improvement on the DISCERN scale included increased clarity regarding the sources of information that were used to create the website, providing the date the information was produced, and providing increased support for shared decision-making.

Discussion

The goals of this study were to evaluate the quality of information concerning anxiety disorders in children that is available on the Internet, in addition to evaluating changes in the quality of website information over time. To the best of the authors’ knowledge, this is the first website evaluation study focused on children’s anxiety, and the first to evaluate website information based on questions that have been identified as important by parents in prior research. Similar to previous website evaluation studies (e.g. Isper et al., 2007; Khazaal, Chatton, Cochand, Jermann, et al., 2008; Khazaal et al., 2008a, 2008b; Reavley & Jorm, 2011), the findings of the current research demonstrated tremendous variability in both the quality and quantity of information across websites. Websites that had the highest total information scores in the initial analysis (2008) included Anxiety United Kingdom (127.0), Anxiety Disorders Association of America (now the Anxiety and Depression Association of America) (100.0), Canadian Mental Health Association-British Columbia (105.0), and the Child Anxiety Network (95.0). These websites also scored well according to the DISCERN criteria. It is important to note that the websites providing the highest quality of information were often not among the first 10 identified by the Google search engine. Given that the majority of individuals searching for information online select a search result within the first 10 websites, this finding is a concern. However, it is important to note that the order of websites is variable, and may have changed since the initial evaluation.

Content areas that were adequately addressed by websites (mean rating of 3.0 or higher) included the treatment choices available for children and adolescents, methods of treatment, a description of how the treatment works, the format of treatment, and other resources for parents. The remaining content areas that parents considered important in previous research were inadequately addressed by websites. There has been a modest amount of research on the effectiveness of pharmacological and psychosocial treatments for anxiety disorders among children and adolescents, and it will be possible to improve the information provided to parents in this area with additional research and synthesis of existing research. Much less information on questions such as the duration of treatment, side effects of treatment, what happens when treatment is stopped, and the long-term outcome of treatment is readily available. The authors are working to assemble information to answer parents’ questions using knowledge synthesis approaches.

Websites also provided limited information concerning administrative questions that parents raised about their child’s treatment. Specific administrative information about the training and experience of treatment providers and the cost, location, and frequency of treatment is likely to vary by clinical setting, and one would not expect that national or regional websites would provide information to this degree of specificity. However, it would be reasonable for websites to suggest questions that parents could ask their treatment provider, in order to help them make informed treatment decisions.

Half of the websites that were analyzed included interactive tools such as video information from professionals, parent testimonials, ask a psychologist/doctor question box, links to social media websites, and an intra-site search engine. Allowing readers to engage with information in a more interactive way may promote understanding and recall of information (Pitkethly, MacGillivray, & Ryan, 2008).

There was a broad range in readability scores across websites, which was skewed toward a higher grade level (average of 12.4 across websites). It is generally recommended that health information for the public be targeted at a reading level of about Grade 8 (Shedlosky-Shoemaker, Sturm, Saleem, & Kelly, 2009). In practice, however, it is difficult to achieve this grade level for websites that include medical terminology, even when it is clearly explained. The website with the highest overall information score (Anxiety UK) had one of the lowest reading levels (10.9), suggesting that this is perhaps a more reasonable reading level that websites could aim to achieve. Information concerning treatment for child anxiety disorders could benefit from being tailored in a manner that is easily understood by parents (Raynor et al., 2007).

In line with findings from research by Coquard et al. (2012) indicating a lack of change in website information on alcohol dependence over a one-year period, authors of the current study found that only six out of 26 websites modified their content from the date of the original analysis (2008) to the date of the revised analysis (2013). This is concerning given the demonstrated need for websites to make improvements in the quality of information. Overall, the six websites that revised their content after the initial evaluation date improved in all of the areas assessed. The limited change in the content of the majority of the websites over this five-year period may indicate the challenge that organizations face in updating their website content. This may also provide an opportunity for researchers to collaborate with these organizations in making evidence-based materials available that answer important consumer questions. It may be best to coordinate these efforts through national organizations of researchers and health service providers interested in providing high-quality information to the public.

This study has a number of limitations. First, authors focused on reviewing common websites that were identified within three pages of results from the most widely used search engine in North America (as well as some websites identified by an expert in the area of anxiety disorders). As such, it was not intended to be an exhaustive review of all websites with information concerning child anxiety. Previous studies have reviewed larger numbers of websites and used more than one search engine (e.g. Isper et al., 2007; Khazaal, Chatton, Cochand, Jermann, et al., 2008; Khazaal et al., 2008a, 2008b); however, these studies have come to similar conclusions to those outlined in the current study. A search carried out in another country or using multiple search engines would have identified different websites. In spite of this difference in website selection, we believe that the websites evaluated in this study provide a good representation of the material that parents are likely to encounter when searching for information on child anxiety disorders on the Internet. Second, the initial identification of websites through the Google search engine occurred in 2008 and a search carried out at another time point may identify different websites, or contain a different ordering of websites. These limitations should be considered as readers evaluate the findings of this research.

The use of the Internet as a source of health information is steadily increasing, allowing the opportunity for health promotion, early detection of disorders, facilitation of help-seeking, and potential improvement of health outcomes (Christensen et al., 2010; Oh et al., 2009; Santor, Poulin, LeBlanc, & Kusumakar, 2007). High-quality information on the Internet that facilitates parental understanding has the potential to complement the information provided in consultation with health professionals and to allow parents to be more active in caring for their child’s physical and mental health. Strengthening the information available on websites to answer questions that parents consider to be important will assist parents in making informed treatment decisions. The use of published standards may be helpful for those developing websites that focus on providing information to the public (Health On the Net Foundation, 2013). Health professionals should consider routinely directing their patients to high-quality websites in their particular area of concern. At the same time, health professionals may also be able to provide consumers with clear information about evaluating the quality of information they find on the Internet.

Given the increase in the public’s reliance on the Internet for health information, the methodological framework used in this study has the potential to be applied to the evaluation of websites concerning a variety of mental and physical health problems across the lifespan, with the ultimate goal of empowering individuals in their treatment decision-making. A key step in this process is identifying the questions that consumers consider to be important when making health-care decisions.

Footnotes

Acknowledgements

Members of the Mobilizing Mind Research Group (mobilizingminds.ca) include the following (in alphabetical order): Young Adult Partners: Chris Amini, Meagan DeJong, Pauline Fogarty, Mark Leonhart, Kristin Reynolds, Tara Syed, Alex Yaeger; Community Partners: Maria Luisa Contursi, Christine Garinger, Heather Miko-Kelly, and Melissa Taylor-Gates from mindyourmind (mindyourmind.ca); and Research Partners: Lynne Angus, Chuck Cunningham, John D. Eastwood, Jack Ferrari, Emma Firsten-Kaufman, Patricia Furer, Madalyn Marcus, Jennifer McPhee, David Phipps, Linda Rose-Krasnor, Kim Ryan-Nicholls, Richard Swinson, John Walker, and Henny Westra.

Funding

Preparation of this article was supported by a team grant from the Canadian Institutes of Health Research (CIHR) and the Mental Health Commission of Canada (#88666).

Author biographies

Kristin A Reynolds, MA, is a PhD Candidate in the Department of Psychology at the University of Manitoba. She is especially interested in research in the areas of knowledge translation of health information, website evaluation, and aging and mental health.

John R Walker is a Professor in the Department of Clinical Health Psychology at the University of Manitoba. He is particularly interested in self-help approaches to mental health treatment. He has coauthored a book focused on health anxiety titled Treating Health Anxiety and Fear of Death (2007), and a self-help book for adults titled Triumph Over Shyness (2009).

Kate Walsh, BA (Honours), is a research assistant and study coordinator in the Department of Clinical Health Psychology at the University of Manitoba.

References

Barnes

Harvey

Wilde

Hadzi-Pavlovic

Wilhelm

Mitchell

(2009). Review of the quality of information on bipolar disorder on the Internet. Australian and New Zealand Journal of Psychiatry, 43, 934–945.

Breckons

Jones

Morris

Richardson

(2008). What do evaluation instruments tell us about the quality of complementary medicine information on the Internet? Journal of Medical Internet Research, 10, e3.

Bremner

J. D.

Quinn

Veledar

(2006). Surfing the net for medical information about psychological trauma: An empirical study of the quality and accuracy of trauma-related websites. Journal of Medical Internet Research, 31, 227–236.

Charnock

Shepperd

Needham

Gann

(1999). DISCERN: An instrument for judging the quality of written consumer health information on treatment choices. Journal of Epidemiology & Community Health, 53, 105–111.

Christensen

Murray

Calear

A. L.

Bennett

Griffiths

K. M.

(2010). Beacon: A web portal to high-quality mental health websites for use by health professionals and the public. The Medical Journal of Australia, 192, S40–S44.

Coquard

Fernandez

Zullino

Khazaal

(2012). A follow-up study on the quality of alcohol dependence-related information on the web. Substance Abuse Treatment, Prevention, and Policy, 6, 13.

Costello

E. J.

Egger

Angold

(2005). 10-Year research update review: The epidemiology of child and adolescent psychiatric disorders: I. Methods and public health burden. Journal of the American Academy of Child & Adolescent Psychiatry, 44, 972–986.

Couper

M. P.

Singer

Levin

C. A.

Fowler

F. J.

Fagerlin

Zikmund-Fisher

B. J.

(2010). Use of the Internet and ratings of information sources for medical decisions: Results from the DECISIONS survey. Medical Decision Making, 30, 106S–114S.

Ferreira-Lay

Miller

(2008). The quality of Internet information on depression for lay people. Psychiatric Bulletin, 32, 170–173.

10.

Fox

Jones

(2009, June). The social life of health information. Pew/Internet & American Life Project. Retrieved from http://www.pewinternet.org/Reports/2009/8-The-Social-Life-of-HealthInformation.aspx

11.

Gray

N. J.

Klein

J. D.

Noyce

P. R.

Sesselberg

T. S.

Cantrill

J. A.

(2005). Health information seeking behaviour in adolescence: The place of the Internet. Social Science & Medicine, 60, 1467–1478.

12.

Hall

Howard

McCaffery

(2008). Do cervical cancer screening patient information leaflets meet the HPV information needs of women? Patient Education and Counseling, 72, 78–87.

13.

Health On the Net Foundation. (2013, June). HONcode certification: Improving the quality of online health information. Retrieved from http://www.hon.ch/HONcode/Webmasters/

14.

iProspect. (2006, April). Search Engine User Behaviour Study. Retrieved from http://district4.extension.ifas.ufl.edu/Tech/TechPubs/WhitePaper_2006_SearchEngineUserBehavior.pdf

15.

Isper

J. C.

Dewing

Stein

D. J.

(2007). A systematic review of the quality of information on the treatment of anxiety disorders on the Internet. Current Psychiatry Reports, 9, 303–309.

16.

Jorm

A. F.

Fisher

(2010). Effect of feedback on the quality of suicide prevention websites: Randomised control trial. British Journal of Psychiatry, 197, 73–74.

17.

Khazaal

Chatton

Cochand

Jermann

Osiek

Bondolfi

Zullino

(2008). Quality of web-based information on pathological gambling. Journal of Gambling Studies, 24, 357–366.

18.

Khazaal

Chatton

Cochand

Zullino

(2008a). Quality of web-based information on cannabis addiction. Journal of Drug Education, 38, 97–107.

19.

Khazaal

Chatton

Cochand

Zullino

(2008b). Quality of web-based information on cocaine addiction. Parent Education and Counseling, 72, 336–341.

20.

Khazaal

Chatton

Zullino

Khan

(2012). HON label and DISCERN as content quality indicators of health-related websites. Psychiatric Quarterly, 83, 15–27.

21.

Khazaal

Fernandez

Cochand

Reboh

Zullino

(2008). Quality of web-based information on social phobia: A cross-sectional study. Depression and Anxiety, 25, 461–465.

22.

Kisely

Ong

Takyar

(2003). A survey of the quality of web based information on the treatment of schizophrenia and Attention Deficit Hyperactivity Disorder. Australian and New Zealand Journal of Psychiatry, 37, 85–91.

23.

Klein

R. G.

(2009). Anxiety disorders. Journal of Child Psychology and Psychiatry, 50, 153–162.

24.

Kummervold

P. E.

Chronaki

C. E.

Lausen

Prokosch

H.-U.

Rasmussen

Santana

. . . Wangverg

S. C.

(2008). eHealth trends in Europe 2005–2007: A population-based survey. Journal of Medical Internet Research, 10, e42.

25.

Leach

L. S.

Christensen

Griffiths

K. M.

Jorm

A. F.

Mackinnon

A. J.

(2007). Websites as a mode of delivering mental health information: Perceptions from the Australian public. Social Psychiatry & Psychiatric Epidemiology, 42, 167–172.

26.

Ley

Florio

(1996). The use of readability formulas in health care. Psychology, Health & Medicine, 1, 7–28.

27.

Mak

Hiebert-Murphy

Walker

J. R.

Altman

(2014). Parents’ decision making and their information needs concerning treatments for child anxiety: Implications for family-centered practice. Journal of Family Social Work, 17, 51–67.

28.

McGee

(2013, August). Google dwarfs Bing and Yahoo as traffic source for major news sites. Retrieved from http://searchengineland.com/google-dwarfs-bing-yahoo-as-traffic-source-for-major-news-sites-170660

29.

Merikangas

K. R.

J. P.

Burstein

Swanson

S. A.

Avenevoli

Cui

. . . Swendsen

(2010). Lifetime prevalence of mental disorders in U.S. Adolescents: Results from the National Comorbidity Survey Replication—Adolescent Supplement (NCS-A). Journal of the American Academy of Child & Adolescent Psychiatry, 49, 980–989.

30.

Merikangas

K. R. J.

Nakamura

E. F.

Kessler

R. C.

(2009). Epidemiology of mental disorders in children and adolescents. Dialogues in Clinical Neurosciences, 11, 7–20.

31.

Murphy

Frost

Webster

Schmidt

(2004). An evaluation of web-based information. International Journal of Eating Disorders, 35, 145–154.

32.

Jorm

A. F.

Wright

(2009). Perceived helpfulness of websites for mental health information: A national survey of young Australians. Social Psychiatry & Psychiatric Epidemiology, 44, 293–299.

33.

Pitkethly

MacGillivray

Ryan

(2008). Recordings or summaries of consultations for people with cancer. Cochrane Database of Systematic Reviews, 3.

34.

Plantin

Daneback

(2009). Parenthood, information and support on the Internet: A literature review of research on parents and professionals online. BMC Family Practice, 18, 10–34.

35.

Raynor

D. K.

Blenkinsopp

Knapp

Grime

Nicolson

D. J.

Pollock

. . . Spoor

(2007). A systematic review of quantitative and qualitative research on the role and effectiveness of written information available to patients about individual medicines. Health Technology Assessment, 11(5), 1–160.

36.

Reavley

N. J. P.

Jorm

A. F.

(2011). The quality of mental disorder information websites: A review. Patient Education and Counseling, 85, e16–e25.

37.

Santor

D. A.

Poulin

LeBlanc

J. C.

Kusumakar

(2007). Online health promotion, early identification of difficulties, and help seeking in young people. Journal of the American Academy of Child & Adolescent Psychiatry, 46, 50–59.

38.

Shedlosky-Shoemaker

Sturm

A. C.

Saleem

Kelly

K. M.

(2009). Tools for assessing readability and quality of health-related web sites. Journal of Genetic Counseling, 18, 49–59.

39.

Shrout

P. E.

Fleiss

J. L.

(1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428.

40.

Sullivan

(2010, January). ComScore: US has most searches; China slowest growth; Google tops worldwide in 2009. Retrieved from http://searchengineland.com/comscore-us-most-searches-china-slowest-34217

41.

Turner

Kabashit

Guthriet

Burket

Turner

(2011). Use and value of information sources by parents of child psychiatric patients. Health Information and Libraries Journal, 28, 101–109.

42.

Underhill

McKeown

(2008). Getting a second opinion: Health information and the Internet (Catalogue no. 82–003-X Health Reports, 19, 65–69). Retrieved from Statistics Canada website: http://www.statcan.gc.ca/pub/82-003-x/2008001/article/10515-eng.htm