Abstract
Anxiety is common in youth with autism spectrum disorder. Despite this common co-occurrence, studies targeting anxiety in this population are hindered by the under-developed state of measures in youth with autism spectrum disorder. Content validity (the extent to which an instrument measures the domain of interest) and an instrument’s relevance to the patient population are key components of measurement development. This article describes the application of qualitative research methods in the initial development of a parent-rated instrument of anxiety symptoms in youth with autism spectrum disorder. Overall, 48 parents of 45 children (aged 3–17 years) with autism spectrum disorder and at least mild anxiety participated in one of six focus groups at two sites (three groups per site). Systematic coding of the focus group transcripts identified broad themes reflecting the situations and events that trigger anxiety in children with autism spectrum disorder, the behavioral manifestations of anxiety in children with autism spectrum disorder, the parent and the child’s own response to anxiety, and broad behavioral patterns that could be associated with anxiety. From the focus group data, investigators generated 52 candidate items for a parent-rating of anxiety in youth with autism spectrum disorder. This report provides a detailed description of these early steps in developing a patient-oriented outcome measure.
Keywords
Autism spectrum disorder (ASD) is a chronic neurodevelopmental condition of early childhood onset marked by social communication deficits and repetitive behavior (American Psychiatric Association, 2013). A recent review of 33 epidemiological studies estimated a worldwide ASD prevalence of 62 per 10,000 children (Elsabbagh et al., 2012). Given its chronicity and associated disability, ASD is a major public health problem.
In addition to the core features, meta-analysis, detailed reviews, and population-based studies indicate that 30%–42% of youth with ASD also meet diagnostic criteria for an anxiety disorder (Simonoff et al., 2008; Van Steensel et al., 2011; White et al., 2009). There are a growing number of psychotherapeutic studies targeting anxiety in youth with ASD; however, most studies have focused on higher-functioning populations (Reaven et al., 2012; Storch et al., 2013; White et al., 2013; Wood et al., 2009). Pharmacological trials are few in number (Scahill et al., 2014).
A major barrier to the design and testing of interventions for anxiety in youth with ASD is the under-developed state of outcome measurement (Grondhuis and Aman, 2012; Lecavalier et al., 2014). A work group empanelled by Autism Speaks concluded that none of the available outcome measures of anxiety are fully appropriate for use in children with ASD (Lecavalier et al., 2014). By default, investigators have adopted self- and parent-report measures of anxiety validated in youth with anxiety disorders uncomplicated by ASD, for example, the Pediatric Anxiety Rating Scale and the Screen for Child Anxiety Related Emotional Disorders (SCARED) (Birmaher et al., 1999; Research Unit on Pediatric Psychopharmacology Anxiety Study Group, 2001; Walkup et al., 2008).
The parent-rated Child and Adolescent Symptom Inventory (CASI) – Anxiety scale, which covers generalized anxiety, separation anxiety, panic, simple phobia, and social phobia, has been used to explore anxiety symptom severity in children with ASD (Hallett et al., 2013). This 20-item scale is part of a 132-item, parent-rated, Diagnostic and Statistical Manual of Mental Disorders (4th ed.; DSM-IV) referenced instrument designed to screen for child psychiatric disorders (Gadow and Sprafkin, 2002). In a sample of 415 children (aged 4–17 years) with ASD enrolled in one of four multisite trials, Hallett et al. (2013) showed that on the CASI Anxiety scale, some children were low on anxiety, others were rated in the mild-to-moderate range, and still others had scores suggesting high anxiety. However, items beginning with “worries” or “complains” were seldom endorsed in this sample, suggesting that items reliant on language may not be useful in children with ASD. The issue of reliance on language was especially evident for intellectually disabled children. Parents of children with an intelligence quotient below 70 rarely endorsed items reliant on language. Thus, compared to children with IQ above 70, children with IQ below 70 had lower mean scores on the parent-rated measure. The coverage of the 20-item measure also raised concerns about the use of this measure in ASD. For example, eight items focus on separation anxiety, but only two items address social phobia. Although the 20-item CASI anxiety scale provides information on the distribution of anxiety symptoms of children with ASD, the incomplete symptom coverage as well as the language and cognitive delays in this population may limit the utility of this measure in children with ASD.
There is also considerable debate about whether anxiety is a separate co-occurring problem in children with ASD, whether anxiety symptoms overlap with ASD, or whether anxiety (e.g. social anxiety) is a complication of ASD (Kerns and Kendall, 2012). At the center of these debates is whether the manifestations of anxiety are similar or different in youth with ASD. Nonetheless, the development of a relevant, reliable, and valid measure of anxiety across the full range of intellectual functioning in youth with ASD is a prerequisite for assessing pre-treatment symptom severity and for monitoring outcomes over time.
Because outcome measures may be used to support a claim for approval of new medications or a new indication for approved medications, the US Food and Drug Administration (FDA, 2009) provided guidelines on instrument development in a monograph: “Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims” (http://www.fda.gov/cder/guidance). This monograph places high value on incorporating the voice of the patient constituency in the development of a given measure. In the case of children with developmental disabilities, parents are necessary and acceptable surrogates. First, investigators are encouraged to determine whether an existing instrument is a reliable and valid measure for the clinical domain of interest. If one is not available, the FDA recommends that investigators design a new instrument and document the development process in order to support the relevance, reliability, and validity of the final product.
Key components of validity include the extent to which the instrument measures the domain of interest (content validity) and is important to the patient population (FDA, 2009). Qualitative research is one way to ensure that patient concerns are captured in the instrument (FDA, 2009; Nassar-McMillan et al., 2010; Rowan and Wulff, 2007; Vogt et al., 2004). Qualitative methods employing focus groups foster in-depth exploration of the day-to-day experience of patients and their families. This approach has been used to examine the experience of individuals with ASD and anxiety (aged 7–35 years) and their caregivers (Ozsivadjian et al., 2012; Trembath et al., 2012). In this study, we describe the process of using focus groups to generate candidate items for a parent-rated instrument of anxiety symptoms in youth with ASD. Item analysis, reliability, and validity will be presented in future reports.
Methods
Qualitative data were collected at two sites (Emory University and Ohio State University) over a 4-month period (April–August 2013). The study was approved by the Institutional Review Boards of both sites prior to data collection.
The study involved the collaboration of two complementary work groups. A Core Team included six investigators: four with clinical and research expertise in ASD and anxiety and two experts in qualitative research. Both qualitative methods experts had a track record of conducting studies using qualitative methodology (Remley et al., 2010; Whittemore and Dixon, 2008; Whittemore et al., 2013; Williams et al., 2008). This team was directly involved in the collection and analysis of focus group data and item generation. To ensure construct validity, we engaged six external consultants with expertise in ASD, anxiety disorders, clinical trials, and instrument design. These consultants provided periodic advice during the data collection period and feedback on the wording and content of item drafts. The generated items were included in a large-scale, web-based, parent survey on children with ASD. The results of this survey will be reported in the future.
Participants
We used purposive sampling to include parents with higher- and lower-functioning children with ASD across a range of verbal skills, levels of anxiety, and age (3–17 years). We also sought families of diverse racial, ethnic, and socioeconomic backgrounds. Parents were recruited from existing subject data banks, outreach to local autism parent groups and websites at each institution. Parents were interviewed by telephone to confirm that their child had received a community diagnosis of autistic disorder, Asperger’s disorder, or pervasive developmental disorder—not otherwise specified (American Psychiatric Association, 2000). At the beginning of the focus group, parents also completed the Social Communication Questionnaire (SCQ; Rutter et al., 2003), which is a brief instrument that evaluates communication skills and social functioning in children who may have autism. Because our interest was to obtain information on children with a range of anxiety symptoms, we asked parents to indicate that their affected child had at least mild anxiety.
Overall, 48 parents participated in one of six focus groups (mean group size = 7; range = 5–12) (Table 1). The age of parents ranged from 26 to 55 years (mean = 41.5; SD = 6.8) and most participants were females (85%). The age of children of participants (32 boys and 13 girls) ranged from 3 to 17 years (mean = 10.4; SD = 3.5); 66.7% were White, not Hispanic; 17.8% African-American, and 11.1% were Other. Parents reported that 46.7% of the children had been diagnosed with autistic disorder and most (68.9%) were placed in a regular education classroom. The mean SCQ score for the whole group of children was 22.4 (SD = 6.9).
Demographic characteristics of participants and their children.
SCQ: Social Communication Questionnaire; PDD-NOS: pervasive developmental disorder, not otherwise specified.
N = 44 for household income as four parent-dyads participated in the focus groups.
Procedures
Focus group design
Qualitative research, and specifically focus groups, for instrument development is a commonly used approach (Morgan, 1998). Focus groups elicit the perspectives of the target population in order to identify important elements of the phenomenon of interest. The focus group transcripts can be used to generate meaningful and understandable items for outcome measures (FDA, 2009; Nassar-McMillan et al., 2010; Rowan and Wulff, 2007; Willgerodt, 2003).
Data saturation
The number of focus groups required for a project can be estimated a priori. Recruitment of participants, however, should continue until little new information is obtained. This concept, known as saturation, usually occurs after 4–6 focus groups with 6–10 participants in each group (Millward, 2012). For this study, transcripts and impressions were reviewed following each focus group to make certain that salient issues from a previous group could be reviewed in the subsequent groups. This re-introduction of issues from prior groups aides the assessment of saturation.
We predicted that four focus groups (two per site) would be sufficient to achieve saturation of themes. Based on review of the transcripts of the first four groups, however, we concluded that two additional focus groups (one per site) would be needed to reach saturation of the thematic content. For example, several parents commented on a perceived connection between anger outbursts and anxiety. Following discussions in the Core Team and with external consultants, we concluded that further exploration of this issue was warranted.
Focus group interview guide
Interview guides provide context and structure for focus group discussions. Core Team investigators drafted a semi-structured interview guide with questions intended to elicit observations and perceptions from parents about: (a) manifestations of anxiety in their children with ASD, (b) events and situations that are associated with the anxiety, (c) how children managed anxiety, and (d) the impact of the children’s anxiety on the family. The resulting guide included four open-ended questions, each with several probes. For example, the question, “What does anxiety look like in children with ASD?” was followed by probes such as “What situations or events bring about anxiety in your child?” and “How do you respond to your child’s anxiety?” Finally, we asked parents to comment on the relevance of the 20-item CASI Anxiety scale to children with ASD and whether important elements were missing. As noted, these 20 items were drawn directly from DSM-IV (Gadow and Sprafkin, 2002; Hallett et al., 2013).
Consistent with the constant comparative method (Creswell, 2007), the interview guide was incrementally modified during the data collection process to foster clarification on behavioral manifestations of anxiety in children with ASD. This approach may alter how questions are asked in subsequent focus groups in order to explore unexpected material in a prior group or to elicit greater detail about specific material. As noted above, parents in early focus groups often attributed anger outbursts (e.g. tantrums or aggression) to anxiety. In subsequent focus groups, we included probes designed to clarify how parents differentiated outbursts driven by anxiety versus angry outbursts for other reasons.
Moderation of focus groups
We used a semi-structured focus group design in which the target population (parents of children with ASD) discussed the phenomenon of concern (anxiety in children with ASD) guided by a moderator (Krueger and Casey, 2008). Interaction among the participants facilitated in-depth discussion and elicited multiple perspectives. Contrasting viewpoints expressed during focus groups provided more detail about the domain of interest and contributed to item generation (O’Brien, 1993). Moderating a focus group, however, poses challenges. Some participants seem all too ready to speak, while others need encouragement to contribute. The discussion can get bogged down in a detail or go off on tangents. To ensure consistency across the two study sites, the qualitative researchers and group moderators reviewed strategies on how to redirect tangential conversations back to the domain of interest, how to manage talkative group members, and how to draw out less talkative participants. Debriefing sessions after each focus group within and across sites reviewed the flow and content of the group and considered strategies for the effective conduct of future groups.
Focus group implementation
We planned to recruit six to eight participants per focus group to foster interaction among participants, provide all members the opportunity to participate, and to avoid audio recording problems created by too many voices (Millward, 2012). Parents were offered $25 compensation for participation. Each session lasted 75–90 min, including informed consent, demographic data collection, the guided focus group discussion, and a review of the 20 CASI items. During the consent process, parents received a detailed description of focus group expectations, including the importance of mutual respect and confidentiality. Each focus group was recorded with two digital recording devices with external microphones. The digital recordings were transcribed verbatim within a week. A research coordinator at each site also kept notes of the conversation to aid the interpretation of transcripts.
Focus group transcript coding
The two investigators on the Core Team who moderated the focus groups (KB and MGA) at each site also completed the detailed coding of all six transcripts, guided by the two qualitative experts. Web-based teleconferences were held after coding each of the six transcripts to assess consistency across the two coders, generate consensus in the coding of text segments, and refine the codebook as needed.
The first round of data analysis, known as open coding, began with each coder independently reviewing the first focus group transcript in order to identify and define emerging keywords (or codes) that captured mutually exclusive dominant themes (Corbin and Strauss, 1990). In consultation with the qualitative experts, the two coders then generated agreed-upon initial codes and definitions of themes that were documented in a codebook intended to guide analysis of the subsequent transcripts. Using the first iteration of the codebook, independent line-by-line open coding continued with the subsequent five transcripts. Any discrepancies between coders were resolved by consensus. When new themes emerged, new code words and definitions were added to the codebook. Thus, the codebook was iteratively refined after review of each focus group transcript. Following discussions and refinement of coding category definitions, percent agreement between coders reached 85%.
After all transcripts were coded and validated via the open coding process, the coded transcripts were imported into Ethnograph software to prepare for the second round of analysis, called axial coding. This software program is designed to store coded data and tag relevant text segments for retrieval.
In axial coding, the broad themes from the open coding were explored to identify subthemes. Text segments coded within a particular theme were grouped together and recoded. The two coders independently coded the text segments into subthemes within the broader themes. As with the open coding, any discrepancies between coders on the independently coded subthemes were resolved by consensus.
Item generation
During the conduct of the focus groups, the six Core Team members discussed recurring topics and themes from the transcripts. These discussions also considered the challenges of translating certain common themes into items on the measure. For example, children with ASD often have difficulty with change in the daily routine and may react with angry outbursts, aggression, or self-injury. In the view of some parents, these reactions were driven by anxiety. Although this may be true for some children, parents also acknowledged that tantrums, aggression, and self-injurious behavior can occur for other reasons as well. Similarly, some parents expressed the view that repetitive behaviors are manifestations of anxiety. Here again, parents acknowledged that repetitive behavior may also occur in the absence of anxiety.
During the open and axial coding, these issues were discussed with our consultants in a series of conference calls. Consultants echoed concerns expressed by the Core Team that certain observable behaviors (e.g. tantrums, aggression, self-injury, repetitive behavior) require parental inference that these behaviors reflect anxiety. The Core Team and consultants agreed that, because these same behaviors can occur in children with ASD for reasons unrelated to anxiety, items reflecting anger outbursts or repetitive behavior should not regarded as specific to anxiety. On the other hand, we agreed that the impressions of parents should not be dismissed altogether. Discussions between the Core Team and consultants also compared and contrasted the manifestations of anxiety in children with ASD and anxious children without ASD.
Another issue that emerged in discussions within the Core Team and with consultants concerned the differentiation of fears (thunderstorms, sirens) from anxiety. We also discussed anxiety disorders as discrete categories (generalized anxiety, separation anxiety, social anxiety) versus a dimensional phenomenon with blurred boundaries between categories. These discussions were intended to avoid departure from contemporary understanding of anxiety in children without ASD (e.g. excessive worry about everyday matters, fears about separation, social phobia, etc.) and to anchor scale items on observable behaviors.
Reflecting on the conference calls with consultants, the Core Team considered several practical measurement issues, including the format of scoring (e.g. none, mild, moderate, severe vs. frequency metrics such as none, rarely, sometimes, always) and phrasing of questions (e.g. use of words such as “extremely” and “excessively”). For example, some parents identified frequent requests for reassurance as a manifestation of anxiety in the child. However, routine requests for reassurance may not be a symptom of anxiety. We agreed that terms such as “extremely” or “excessively” should be used sparingly as needed to qualify behaviors such as requesting reassurance that in mild form may not reflect anxiety.
A summary review of the transcripts and final coding schemes as well as discussions within the Core Team and feedback from consultants led to a draft of 25 candidate items. This list of 25 items was reviewed by consultants, who made suggestions regarding wording and overall reading level of items. Using this feedback from the consultants, the Core Team refined the wording of some items and generated several new items, and items with alternative wording for a second draft that included 52 new candidate items, to which we added the original 20 CASI items.
Results
Themes of parental perceptions of anxiety
The qualitative analysis included 25 web-based conference calls over 7 months. The systematic coding of the transcripts from the six focus groups resulted in six broad themes, containing 45 subthemes: Triggers, (13 subthemes); Setting Events, (5 subthemes); Observable Behaviors, (7 subthemes); Coping, (9 subthemes); Parental Management, (6 subthemes); Hold In–Escalation–Release, (5 subthemes). Table 2 provides definitions for all six themes as well as a listing of the accompanying subthemes.
Six themes with definitions and listing of subthemes.
Parent feedback on the 20-item CASI
Eight of 20 CASI items were rated by a majority (⩾50%) of parents as “not relevant to my child.” Notably, three of these items used “complains” in the question and four included “worries.” This feedback mirrors findings from Hallett et al. (2013) where items with “worries” or “complains” were endorsed at a lower frequency. The item “nightmares about being separated from parent” was deemed “not relevant” by several parents.
Item generation
Table 3 illustrates how the coding scheme and text segments from the focus groups informed item generation. All but one theme (Hold in—Escalation—Release) contributed to the pool of items. Triggers and Coping themes contributed the most new items (22 and 15, respectively). Two items not directly connected to text segments in the transcripts were generated by investigators based on broader issues raised by parents about the presence of repetitive and self-critical behaviors in their children. Based on the Flesch–Kincaid Grade level index, the readability of the 72 items was estimated at grade 7.1.
Illustration of translation from text example to item on the draft parent-rated measure of anxiety.
Discussion
In addition to usual demands of reliability, validity, and sensitivity to change with treatment, new outcome measures are expected to incorporate input from patients or, in the case of children with ASD, from parents (FDA, 2009). To this end, we conducted six focus groups with parents of children with ASD. We also incorporated input from expert consultants. For outcome measures intended to support FDA approval for a medical treatment, investigators are also advised to document the steps in measure development.
This article describes the application of qualitative research methods in the early phases of developing a parent-rated anxiety measure for children with ASD. The process from initiating focus groups to generation of candidate items took 13 months. This timeline was influenced by the number of focus groups conducted and the resulting volume of qualitative data produced. The goal was to achieve data saturation in order to avoid missing relevant content.
The 52 new candidate items incorporated a wide range of input from parents who participated the focus groups, 50 of which were pulled directly from comments made by parents during the focus groups. The draft items were generated following the identification of six themes. Of these, all but one theme contributed to the list of candidate items. The exclusion of items generated from the “Hold in—Escalation—Release” theme was due to the uncertain attribution of anger outbursts to anxiety. Co-occurring anxiety can cause acute distress, amplify the core symptoms of ASD, and trigger behavioral difficulties including anger outbursts, reactive aggression, and tantrums (Bubier and Drabick, 2009; Hallett et al., 2013; Moskowitz et al., 2013). However, children with ASD also exhibit challenging behavior to escape demands, for attention or to obtain tangible items such as a favorite food or preferred toy (Matson et al., 2011; Reese et al., 2005). Thus, an item on tantrums would likely get endorsed for multiple reasons. Moreover, comments from the Hold in—Escalation—Release theme often involved a complex sequence of behavior that could be difficult to capture in a single item. For example, one parent stated:
My son tends to let it build up, and you never know what’s going to be the final straw, and it’s usually not things of date, the anxiety is usually about something else, then all of sudden he starts to escalate into physical behavior.
The use of qualitative methods to develop a parent-rated outcome measure for anxiety in children with ASD is uncharted territory. This was a team effort involving investigators familiar with qualitative methodology as well as investigators with research and clinical experience in children with ASD. In addition, we benefited from periodic input from external consultants with expertise in anxiety in children without ASD, anxiety in children with ASD, and measurement development. The discourse at multiple levels fostered examination of opposing views that enriched the discussion and promoted resolution. For example, there was extensive discussion on whether anxiety is better understood as a categorical (disorder-based) or dimensional phenomenon. At this point in the process, we suggest that categorical versus dimensional is an empirical question that item analysis of the large-scale survey with the 72-item draft measure may answer (52 new items and 20 CASI items). Similarly, there was a considerable deliberation on whether anxiety is a co-occurring problem in children with ASD, a complication of ASD or in some way convergent with ASD (Hallett et al., 2013; Lecavalier et al., 2014; Kerns et al., 2014; Kerns and Kendall, 2012; Wood and Gadow, 2010). If anxiety is a co-occurring problem, then anxiety and anxiety disorders in children with ASD would not be different from children without ASD. The results from the focus group suggest that anxiety may have somewhat unique behavioral manifestations in children with ASD. Social anxiety offers an example of anxiety as a complication of ASD. Children with ASD may recognize their social disability, which may amplify their reluctance to enter social situations. The long-standing observation that children with ASD insist on routines and may over-react to changes in routine offers a model on convergence of anxiety and ASD. In our focus groups, parents noted that some children with ASD who are insistent on routines also seem to be constantly vigilant about a change in the routine.
This study is the first in a series of steps toward the development of a parent-rated measure of anxiety in children with ASD. Limitations of our approach include acceptance of a community diagnosis of ASD and reliance on parent report of at least mild anxiety without formal assessment of either. We submit that this level of characterization was appropriate for this first stage of measurement development.
The next step is to evaluate parental response on the 72-item draft instrument (52 new items and 20 items from the CASI) in a large sample of children with ASD collected via web-based survey. Within the 52 new items, there is deliberate repetition with subtle differences in wording. For example, the current pool of items includes “gets upset if someone breaks the rules” and “gets upset when others (children or adults) do not follow rules.” This subtle variation may or may not be important, but the items were derived directly from focus group material. This variation and several others may not have occurred if we relied solely on clinical experts for item generation. We expect that subsequent factor and item analysis will pare the number of items and take on the issue of a single dimension versus diagnostic categories. Subsequent steps will also include evaluation of the reliability and validity of the new measure in a well-characterized sample of children with ASD.
Conclusion
In keeping with the FDA monograph on the development of patient reported outcome measures, eliciting parental perspectives on anxiety in children with ASD via focus groups was key to the generation of items. Our investigative team and external consultants examined themes and subthemes identified in the qualitative data analysis procedures. Clinical expertise and experience with measurement construction aided interpretation of parental descriptions and item generation. By design, however, our attention focused on the voices of the respondents as the primary source of information.
Footnotes
Acknowledgements
We would like to thank our consultants Ken Gadow, Matthew Goodwin, John Herrington, Robert Schultz, John Walkup, and Susan White for their expertise and input during this project.
Funding
This grant was supported by a grant from the National Institute of Mental Health [NIMH R01 MH099021] to Dr Scahill.