Feasibility of Virtual Reality Environments for Adolescent Social Anxiety Disorder

Abstract

Purpose:

This study assessed the feasibility of virtual reality (VR) exposure as an assessment and treatment modality for youth with social anxiety disorder (SAD).

Methods:

Forty-one adolescents, 20 of which were identified as having SAD, were recruited from a community sample. Youth with and without SAD were exposed to two social virtual environments—party and public speaking—and two neutral virtual environments.

Results:

All youth reported significantly higher ratings on the Subjective Units of Distress Scale (SUDS) during the party and public speaking scenarios compared to the two neutral environments, while youth with SAD reported significantly higher SUDS in the public speaking and party environments than those without SAD. Youth also demonstrated acceptable levels of presence and immersion in the VR environments.

Discussion:

VR exposure treatment appears to be feasible for youth with SAD, highlighting the need for further research on its development.

Keywords

virtual reality social anxiety disorder social phobia youth adolescent treatment

Social anxiety disorder (SAD) is a chronic mental health condition characterized by a fear of negative evaluation, excessive self-consciousness, and avoidance of social contacts (Stein & Stein, 2008). SAD ranks among the top 10 chronic disorders, mental or physical, that affect health-related quality of life (Saarni et al., 2007), and is believed to be one of the most prevalent psychiatric conditions, with a lifetime prevalence rate of up to 12% (Chavira, Stein, Bailey, & Stein, 2004; Herbert et al., 2009; Stein & Stein, 2008). SAD is also highly comorbid with major depression and has been identified as a strong risk factor for the subsequent development of alcohol dependency (Buckner et al., 2008; Buckner & Turner, 2009; Grant et al., 2005; Kessler, Chiu, Demler, & Walters, 2005).

The typical onset of SAD is 15.5 years, with comparable proportions of youth experiencing this disorder as adults (Schneier, Johnson, Hornig, Liebowitz, & Weissman, 1992). Adolescents with SAD have fewer friends and may have difficulty with school attendance and substance use disorders (Essau, Conradt, & Petermann, 1999; Last & Strauss, 1990). While the bulk of research has focused on adult SAD, the early onset, chronicity, comorbidity, and impairment associated with this disorder underscore the importance of early intervention. Three main social anxiety interventions have been developed and tested for children and adolescents (Garcia-Lopez, Olivares, Beidel, Albano, & Rosa, 2006). However, all three consist of 12–16 group sessions or 29 individual sessions and typically require large amounts of structured homework, highlighting the need for more feasible, brief interventions with greater dissemination potential (Garcia-Lopez et al. 2006; Khalid-Khan, Santibanez, McMicken, & Rynn, 2007; Sweeney & Rapee, 2005). Moreover, existing social anxiety interventions for youth have only established “probably efficacious treatment” designation, suggesting a need for further research on efficacious treatments (Sarver, Beidel, & Spitalnick, 2014). Virtual reality (VR) interventions provide multiple opportunities to expose adolescents to environments that mimic real-world settings, as well as an opportunity to practice social skills in a safe environment. As such, VR treatment may yield more powerful effects in fewer sessions and be more acceptable to adolescents, given their comfort with technology and the opportunity to confront their fears in less threatening virtual social settings with fewer social risks.

Exposure Therapy: In Vivo Versus VR

Exposure therapy, where a client is systematically exposed to specific situations or objects that elicit anxiety in an effort to reduce the fear response, has been identified as an efficacious approach for treating anxiety disorders, including panic disorders, agoraphobia, social phobia, and obsessive-compulsive disorders (Barlow, Ellard, Hainsworth, Jones, & Fisher, 2005; Deacon & Abromowitz, 2004; Feske & Chambless, 1995; Landon & Barlow, 2004). There are currently three approaches for implementing exposure therapy: (1) in vivo (direct contact with the feared object or situation in a real-world context), (2) imagery (having the client imagine the feared object or situation), and (3) more recently, the use of VR (direct contact with the feared object or situation in a virtual environment; Repetto et al., 2011).

Virtual reality exposure therapy (VRET) has been used to successfully treat other adult anxiety disorders, including SAD (Anderson et al., 2006; Anderson, Jacobs, & Rothbaum, 2004; Anderson, Rothbaum, & Hodges, 2003; Anderson, Zimand, Hodges, & Rothbaum, 2005; Anderson, Zimand, Schmertz, & Ferrer, 2007; Botella et al., 1998; Carlin, Hoffman, & Weghorst, 1997; Coelho, Waters, Hine, & Wallis, 2009; Klinger et al., 2005; Powers & Emmelkamp, 2008; Rothbaum et al., 2006; Rothbaum, Hodges, Kooper, & Opdyke, 1995; Rothbaum, Hodges, Ready, Graap, & Alarcon, 2001; Rothbaum, Hodges, Smith, Lee, & Price, 2000; Rothbaum & Schwartz, 2002; Safir, Wallach, & Bar-Zvi, 2012). The potential advantages of the use of VR include the following: (1) an alternative option for clients who are unwilling or too fearful of in vivo exposure or who have difficulty imagining the feared situation; (2) the ability of the therapist to have complete control over the fear stimulus and the hierarchical order of the feared stimuli; (3) the option of the therapist to immediately stop exposure if it is too overwhelming for the client; and (4) the convenience, reduced time, and reduced cost of creating the context of the feared stimulus within the office as compared to real-world in vivo exposure (Repetto et al., 2011).

VR and VRET

VR provides a human–computer interaction that allows clients to feel a sense of presence and immersion in a virtual environment, offering an opportunity to expose clinically anxious individuals to realistic life scenarios thereby reducing their reactivity to anxiety provoking cues and helping them to learn meaningful coping skills that can be transferred to real-world settings. Given the many challenges and stresses that adolescents experience within the peer social context, VR may provide an optimal medium to teach youth the effective ways of coping with social anxiety before practicing them in real social settings. VR systems utilize a head mounted display (HMD) with a tracking system that detects user movements and adapts by changing the displayed environment in real time, as if someone were looking around a real-world environment. Graphics, directional stereo audio, vibration patterns, and scent cues can all enhance the VR environment to portray an environment that most closely matches the sensory cues present in a real-world setting.

VR exposure treatment is similar to other kinds of graded exposure therapy (in vivo, imaginal), where the goal is to reduce anxiety symptoms by introducing the client to the feared situation/object in small amounts over time (Krijn, Emmelkamp, Olafsson, & Biemond, 2004). As the client begins to habituate to this situation/object, they become more comfortable and less anxious. VR technology has been identified as an effective medium for exposure treatment and appears to be promising for integration into traditional cognitive behavioral therapy treatment modalities (Anderson et al., 2006; Bordnick, Traylor, Carter, & Graap, 2012; Scozzari & Gamberini, 2011). If exposure and skill development can occur in relevant, realistic, and feasible social contexts where the maladaptive behavior was learned (e.g., VR), such maladaptive behavior can be unlearned and generalized into real social settings (Bordnick et al., 2012). Based on past research (Powers & Emmelkamp, 2008), VR environments elicit realistic emotional responses, providing a novel clinical opportunity for exposure and social skill development, as well as the evaluation of practice outcomes using ongoing measurements of anxiety and stress obtained during VR exposure. Although VR has been found to be efficacious among adults with various anxiety disorders (Powers & Emmelkamp, 2008) and elicits responses from socially anxious participants (Garau, Slater, Pertaub, & Razzaque, 2005), the authors are unaware of a study that has assessed the feasibility of these methods with socially anxious adolescents. It should be noted, however, that Rizzo and colleagues (2009) are currently in the process of developing a virtual classroom for socially anxious youth.

Despite the growing research on VRET and anxiety disorders, there is a clear need for additional research on the potential promise of VR exposure to treat SAD (Krijn, et al, 2004; Opris et al., 2012), especially since it has not been assessed for the treatment of adolescents specifically (Harris, Kemmerling, & North, 2002). Thus far, VR exposure for SAD has only been assessed with younger children in a feasibility study. While this study only included 17 children and an anxiety outcome measure was not reported, the findings of this study suggested that VR for children with childhood SAD was feasible and acceptable for children and their family (Sarver et al., 2014).

To address the lack of research on VRET as an assessment and treatment modality for adolescents with SAD, this study assessed the feasibility and acceptability of VR for socially anxious adolescents. The logical first step in this line of research was to determine whether VR environments can evoke anxiety in adolescents with SAD. As such, the primary goal of this study was to examine whether adolescents with SAD react to VR social cues differently than nonsocially anxious youth and the degree to which these environments elicit responses to social cues as they would within real-world environments. This study had the following three aims: (1) Test the feasibility of a VR Exposure System for differentiating between adolescents with SAD and those without SAD; (2) Test the feasibility of a VR Exposure System for differentiating between socially reactive VR environments and neutral VR environments; and (3) Test whether adolescents with SAD react differently to a public speaking VR scenario when compared to a social party VR scenario. We hypothesized that (a) adolescents with SAD would report higher Subjective Units of Distress Scale (SUDS) scores during their immersion in the socially reactive VR environments and (b) that the VR Exposure System would differentiate between socially reactive VR environments and neutral VR environments. If these hypotheses are supported, it would inform future research concerning the development of VR exposure treatment for adolescents, as well as establish known-groups and discriminant validity of the VR Exposure System (using the SUDS or SUDS during exposure) as an assessment protocol during treatment. A secondary aim of this study was to assess the acceptability of VR among adolescents.

Method

Participants

The study utilized a community sample of 41 adolescents between the ages of 13–18. The majority of participants were female (65.9%) and an average of 16 years of age (SD = 1.65). Participants self-identified as “Caucasian” (n = 16, 42%), “Black or African American” (n = 14, 37%), “Asian American” (n = 4, 10.5%), “Native Hawaiian or Other Pacific Islander” (n = 2, 5.3%), and “Hispanic/Latino” (n = 2, 5.3%). Three participants did not answer the question about their racial/ethnic identity. Twenty of the 41 adolescents were identified as socially anxious as they had a score above the cutoff of 29.5 on the Liebowitz Social Anxiety Scale for Children and Adolescents (LSAS-CA; Masia-Warner et al., 2003). Socially anxious youth were compared with the remaining 21 nonsocially anxious youth in the sample for purposes of this study. As shown in Table 1, there were not any significant demographic differences between those with and without SAD.

Table 1.
Comparison of Youth With and Without Social Anxiety on Background Demographics, SESS Score and SUDS Ratings.

Socially Anxious Nonsocially Anxious χ²/t

(n = 20) (n = 21)

N % N % p

Gender χ² = .299 .585

Male 6 30.0 8 38.1

Female 14 70.0 13 61.9

Race χ² = .991 .609

White 8 40.0 8 40.0

African American 7 38.9 6 30.0

Other 3 16.7 6 30.0

Hispanic

Yes 9 45.0 12 57.1 χ² = .606 .436

No 11 55.0 9 42.9

M SD M SD

Age 16.15 1.76 15.81 1.57 t = .655, df = 39 .516

SESS^a 53.90 14.90 70.61 10.85 t = −.4.05, df = 38 <.001

Baseline SUDS 8.57 15.38 2.00 4.71 t = 1.83, df = 22.37 .080

LSAS-CA^b 56.65 20.98 17.57 7.33 t = 7.89, df = 23.37 <.001

Note: N = 41. SUDS = subjective units of distress scale; LSAS-CA = Liebowitz Social Anxiety Scale for Children and Adolescents; SESS = Self-Efficacy for Social Situations Scale.

^aHigher scores on the SESS represent greater self-efficacy in social situations.

^bThis scale was used to assign participants to group with a cutoff of 29.5.

Community recruitment methods were used, including flyers on local university campuses and throughout the community, social networking sites, a community agency, and an online classifieds advertisement. Two advertisements were used—one specifically targeted socially anxious youth and another targeted all youth interested in participating in a community study. To incentivize participation, US$30 gift cards were offered. To augment these methods, snowball sampling was used and incentivized with an additional US$10 gift card for each referred participant. Each recruitment method invited potential participants and parents to contact research staff by phone to learn more about the study and complete the initial screening process. Verbal permission by a parent or guardian was required for eligibility screening if the potential participant was a minor. Participants aged 13–18 were eligible unless they endorsed: (1) a current or past Diagnostic and Statistical Manual of Mental Disorders IV psychiatric diagnosis of severe mental illness or alcohol or drug dependency, (2) recent use of psychotropic medications in the past 30 days, (3) being a member of the same household as another participant in the study, (4) prior use of illicit drugs, (5) fear of closed spaces or inability to wear VR helmet, and/or (6) specific health concerns, including visual problems that affect viewing VR materials, history of seizures or seizure disorders, serious health problems, or pregnancy.

Eligible participants were scheduled for an appointment in the VR lab, where the research assistant provided informed consent and completed assent and parental permission forms. This protocol took approximately 1.5 hours and was approved by the University of Houston’s Committee for the Protection of Human Subjects.

Research Design and VR Exposure Protocol/Materials

The LSAS-CA was used to identify youth with SAD for group assignment. Although there are two cutoff scores that indicate the presence of social phobia using the LSAS-CA, the more conservative score (29.5) was selected to better distinguish social phobia from other coexisting anxiety disorders (Masia-Warner et al., 2003). Participants and research assistants were blind to social anxiety group assignment, as the instrument was not formally scored until after the research protocol was completed for each participant.

After completing baseline measures, all participants were randomized to two possible VR exposure paths to prevent an order effect (see Figure 1). Each participant wore a VR tracker helmet (HMD; VFX-3D Interactive Imagining Systems, Rochester, New York; customized with optics from the Emagin z800 HMD, Emagin, Bellevue, Washington) that was adjusted to fit properly and sat in a comfortable, stationary chair. Prior to beginning the VR exposure, participants were exposed to a nonrelated VR setting to become acclimated with the proper use of the helmet and the VR experience. All procedures for the VR protocol were standardized, including the scripts read by the research assistants prior to and during VR exposure.

Figure 1.
Randomized paths through virtual reality (VR) environments.

Participants were asked to rate their level of anxiety at four equally spaced time points (approximately every 75 s) during each 5-minute VR exposure using the SUDS (Wolpe, 1990). The average SUDS score across each scenario for each individual was used for the final analysis, as there was not a significant difference between the four time points of either primary VR environment for all participants or the socially anxious group. The SUDS served as the primary dependent variable for the analysis. After each VR exposure, the participant was asked to take off the HMD helmet to rest for 5 minutes before the next VR environment.

After finishing the party and speech environments, participants completed the neutral environment, with SUDS collected at the same time points as the initial neutral environment. Participants were asked to complete the modified Imagery Realism Presence Questionnaire (PQ) and The Immersion Questionnaire (IQ). Participants were then debriefed and asked to describe their reactions to each of the VR environments and provide suggestions to improve the VR environments.

VR Environments

As shown in Figure 1, each participant began with the same neutral environment, which included 5 minutes in a VR fish gallery. Participants were then exposed to the party and public speaking environments. To counterbalance exposure to the order of party and public speaking environments, we used randomization. Both of these scenarios were approximately 5 minutes. The protocol ended with the same neutral environment from the first exposure.

VR neutral environment

The VR neutral environment consisted of a virtual art gallery where participants viewed fish and aquatic environments on the walls. The neutral environment was devoid of social interactions or other SAD cues.

VR public speaking environment

To engender anxiety, research staff asked participants to take approximately 3–4 minutes to prepare a 4-minute speech on a topic of their choice (designed to elicit anticipatory anxiety). Next, the participant was moved into a virtual classroom where an audience of 30 was waiting (see Figure 2). After arriving in the virtual environment, the participant was asked to begin the speech. The audience was prompted to look sleepy, distracted (e.g., person answering ringing phone), as though they disagreed, and were puzzled in a consistent manner during the speech for each participant. At the end of the scenario, the audience was prompted to clap politely. These environments were developed by Virtually Better, Inc., Decatur, Georgia.

Figure 2.
Screenshots of party and public speaking virtual reality (VR) environments.

VR party environment

In the party environment (see Figure 2), participants were asked to provide their first SUDS rating while on the walkway outside a house party where party music was playing (in the head set) and individuals can be seen visiting inside an open front door at the party (designed to elicit anticipatory anxiety). Participants were encouraged to interact naturally with others. This environment runs on an automatic timer, moving the participant through various parts of the home where the participant is exposed to several social interactions with individuals. The VR party was developed by Bordnick (2011) of the VRCRL at the University of Houston and World Viz; Santa Barbara, California, 2011.

Measures

Baseline assessment

The self-administered baseline questionnaire was completed by the participant, which included background and demographic questions, many of which were taken from the 2009 State Youth Questionnaire. The SESS and LSAS-CA were also administered at baseline and are described subsequently.

Self-Efficacy for Social Situations Scale (SESS)

The SESS is a 9-item self-report measure that was originally developed to assess self-efficacy in social situations for adults (Gaudiano & Herbert, 2003; Gaudiano & Herbert, 2007) and that has since been adapted for adolescents and shown good internal consistency (.82), as well as construct and criterion-related validity (Gaudiano & Herbert, 2007). A sample item includes “How often do you think your thoughts and worries bother you during social situations?” Participants rate each item using a Likert-type scale ranging from 1 to 10. Three items are reverse scored, and higher scores represent higher levels of self-efficacy in social situations.

The LSAS-CA

The LSAS-CA is interviewer-administered, has high internal consistency (.97) and test–retest reliability, and constructs validity (Masia-Warner et al., 2003). The LSAS-CA consists of 24 items, 12 social interaction situations, and 12 performance situations, and participants are asked to rate their level of anxiety in each situation (0 = none, 1 = mild, and 3 = severe). Although there are two cutoff scores that indicate the presence of social phobia using the LSAS-CA, the more conservative score (29.5) was selected to better distinguish social phobia from other coexisting anxiety disorders (Masia-Warner et al., 2003).

Measures obtained during VR environments

The SUDS was administered 4 times during the VR environment exposure, and Imagery Realism PQ and IQ were administered following VR exposure to assess participants’ views of how real the VR environment was and how immersed they felt within this environment.

SUDS

The SUDS is a measure of subjective anxiety, where participants rate their experience from 0 = no distress to 100 = most distress (Wolpe, 1990). This measure has correlated highly with psychological measures of distress (Thyer, Papsdorf, Davis, & Vallecorsa, 1984), and its validity and sensitivity to change has been established in college students and adult populations (Kaplan, Smith, & Coons, 1995; Kim, Bae, & Park, 2008; Tanner, 2012). The SUDS has also been used with anxiety-disordered youth (Benjamin, O’Neil, Crawley, & Beidas, 2010).

Realism/presence

A shorter, modified version of the Imagery Realism PQ, based on the original scale (Witmer & Singer, 1998), was used to measure the extent of realism, interactions, involvement, and naturalness of the VR experience. Witmer and Singer (1998, p. 225) define presence as “the subjective experience of being in one place or environment, even when one is physically situated in another.” Participants rated 19 questions on 7-point scales, with total scores ranging from 19 to 133 and higher scores indicating a greater sense of presence in the virtual context. An example of an item from this scale is “How much did your experiences in the virtual environment seem consistent with your real world experiences?”

The IQ

The IQ (Zimand, Anderson, Rothbaum, & Hodges, 2001) measures the extent of presence and immersion in the virtual context. The IQ consists of 7 items rated on 7-point scales, with total scores ranging from 7 to 49 and higher scores representing greater immersion. A sample item of the IQ is “How easy was it for you to convince yourself that the exposure was ‘for real’?”

Data Analysis

SPSS 18.0 was used to conduct all statistical analyses. Missing data were within acceptable levels at less than 1% of any one case or variable. For all primary and secondary analyses, we assessed normality using histograms and skewness and kurtosis statistics for each dependent variable, homogeneity of variance using Levene’s test, and violations of sphericity using Mauchly’s test. While Levene’s test identified unequal variances for three of the four dependent variables, the F test is fairly robust against unequal variances when the sample sizes are similar, as in this study.

The purpose of this study was to assess the feasibility of VRET and assessment for socially anxious youth by comparing them to nonsocially anxious youth with regard to social anxiety cue reactivity, as measured by self-rating on the SUDS. A two (with and without social anxiety) by four (VR context) analysis of variance (ANOVA) with repeated measures was utilized to compare the differences in SUDS ratings across the four VR contexts (first neutral condition, presentation, party, and second neutral condition) using average SUDS rating for each condition as the dependent variable. Since there were issues with skewness and lack of sphericity in the data, we utilized a bootstrap to test the mixed ANOVA results, and they were confirmed.

The study sample was divided into two groups—socially anxious and nonsocially anxious—using the LSAS-CA clinical cutoff score of 29.5, which distinguishes youth with SAD from those with other anxiety disorders (Masi-Warner, et al 2003). Group assignment based on the LSAS-CA was further validated by comparing the groups on SESS scores. As shown in Table 1, the socially anxious group was significantly more likely to have lower self-efficacy scores in social situations (p < .001). As shown in Table 1, the socially anxious and nonsocially anxious groups did not differ on key background variables or baseline SUDS ratings in the first neutral condition. For this reason, the inclusion of a covariate was unnecessary for this analysis.

Results

Sample

Participant characteristics by group are reported in Table 1. Overall, 66% of participants were female and the average age was 16 years (SD = 1.65). Fifty-one percent of the sample identified as Hispanic. With regard to race, of the 38 that reported their racial background, 42% were White, 37% African American or Black, 11% Asian, 5% Native Hawaiian or Other Pacific Islander, and 5% Hispanic.

Assessment of Order Effect

A repeated measures multivariate analysis of variance was used to assess differences in order of VR environments between cue and neutral VR contexts, with path as the between subjects factor. There was neither a main effect for path (p = .77) nor a significant interaction between environment and path (p = .428); therefore, data were collapsed across paths.

Primary Analysis

A two (with and without social anxiety) by four (VR context) ANOVA with repeated measures on the second factor was used initially to examine the differences over the four contexts. Results indicated that the two groups (with and without social anxiety) differed on the average of the SUDS ratings, between subjects effect; F(1, 39) = 12.84; p = .0009. In addition, there was a significant difference between the VR contexts when group was ignored, within subjects effect; F(3, 117) = 60.42; p < .0001, using the Geisser-Greehouse correction. More importantly, however, there was a significant interaction between group and context, F(3, 117) = 8.36; p = .0012, using the Geissser-Greenhouse correction, meaning the group differences depended on which context was considered. Simple main effects comparing the group at each context indicated that while the group did not differ during the first or second neutral contexts, they did differ in the presentation context, F(1, 39) = 12.97; p = .0009, d = 1.13, and the party context, F(1, 39) = 7.95; p = .0075, d = .87, with the socially anxious group having higher scores on the SUDS at both of these time points (see Figure 3).

Figure 3.
Comparison of mean subjective units of distress scale (SUDS) rating of distress by virtual reality (VR) context between socially anxious and nonsocially anxious adolescents.

Since the SUDS scores were positively skewed, the next step was to examine the results after controlling for this positive skew. Typically, a nonlinear model would be used, but the sample was too small. Therefore, we used a bootstrapping approach to assess whether these aformentioned results would be supported. Bootstrapping works by taking a sample of size n (where n is the observed sample size) from the data “with replacement” so that in some samples particular subjects will be repeated and in others they may not be represented. By taking a large number of these replications (5,000 in our case), an aproximation to the sample distribution can be determined without assuming any particular distributional form. In our case, since we have repeated measures, we created six pairwise comparisons between contexts, calculated the difference between the groups for each comparison for each of 5,000 bootstrap samples and determined the 99% confidence intervals (CIs) for each comparison to provide some control for multiple tests.The results were consistent with those of the repeated measures ANOVA. The socially anxious group showed a greater increase in the SUDS score between the first neutral context and the presentation context (99% CI: [4.13, 46.72]), between the second neutral context and the presentation context (7.44, 51.48), and between the second neutral and the party contexts (2.94, 24.57). The difference between the groups comparing the first neutral to the party context (95% CI: [0.14, 19.83]) and between the party and presentation contexts (95% CI: [1.85, 32.45]) would have been significant without the more stringent α, but just failed to reach significance when the number of comparisons was controlled. There was no difference between the groups for the comparison of the first and second neutral contexts (95% CI: [−10.38, 1.52]).

Presence and Immersion in the VR Environments

Participants’ average score on the PQ questionnaire was 89.61 (SD = 22.76) with a per item mean of 4.72 (SD = 1.20), which is above the mid-point of the scale. The average score on the IQ was 30.76 (SD = 6.40) with a per item mean of 4.39 (SD = .91), also above the mid-point. The socially anxious and nonsocially anxious youth differed on their scores on the PQ (t = −2.082, df = 39, p = .04) but not the IQ (t = −.392, df = 39, p = .697). While the nonsocially anxious group reported higher levels of presence (M = 93.52, SD = 24.48), the socially anxious group still reported acceptable levels and less within group variation (M = 79.00, SD = 19.81) overall. As shown in Figure 4, socially anxious and nonsocially anxious youth did not differ with regard to key presence and immersion items and only differed on 3 of 21 presence and immersion items overall. Two of these items had to do with VR equipment (delay between actions/outcomes and control devices interfering with performing tasks).

Figure 4.
Comparison of average PQ and IQ item ratings of socially anxious and nonsocially anxious youth with standard deviation bar.

Acceptability of VR by Youth

During the debriefing session, participants described the neutral environment, which included movement through a gallery of fish tanks, primarily as relaxing, calm, or boring. Some also described it as “real,” with one person sharing, “I felt like I was in at an aquarium.” Another individual shared that the second neutral environment was helpful “to calm down.”

Sixty-one percent (n = 24) of the youth described their reaction to the presentation environment using one of the following words: scared, nervous, worried, embarrassing, or afraid. Some of the participants shared how anxious it made them that people were watching them. Most of the remaining (n = 14) participants had a more benign experience with the presentation environment, but shared it seemed real, using words such as fun, unprepared, genuine, weird, and cool. Three youth described the environment as unrealistic, and one participant suggested he would be more nervous if the avatars were his own age. Nearly every participant (except one) rated the presentation environment as more anxiety provoking than the party environment.

In the party environment, eight of the participants used the word real or normal to describe their experience. Most of the other youth used words that would typically be used to describe a party environment such as “fun and interesting,” “cool,” “mildly entertaining,” “not too rowdy,” and “wanted to dance.” One participant shared, “It felt like a real party…[the] food was making me hungry.” Another commented, “Felt real, the people and the pool looked real and reminded me of a party I’ve been to.” Interestingly, several comments suggested social anxiety may have been higher if the environment demanded more social interaction, such as “A little unphased because I didn’t feel like I was interacting” and “A little more relaxed than I thought it would be. Seemed like it was easier than in real life.” Many shared suggestions for making a VR party scenario more applicable to youth including taking the news off the TV and adding dancing, alcohol, and updated music.

Discussion and Applications to Social Work Practice

The primary aim of this study was to assess the feasibility of VRET and assessment for adolescents with SAD. The results support the feasibility and potential promise of this modality with socially anxious adolescents. Socially anxious youth responded with higher levels of distress in virtual environments requiring public presentation and a social party environment compared to neutral virtual environments without social cues. In addition, socially anxious youth reported significantly higher levels of distress compared to nonsocially anxious youth in each of the VR social environments, further suggesting that these environments provide valid social cues and elicit social anxiety that may serve as an analog to real-world social settings. For this reason, VR social environments may have promise for both the treatment of social anxiety in youth and the assessment of anxiety outcomes during VR treatment.

These findings also support the potential feasibility and acceptability of VRET for youth with social anxiety. Specifically, the data demonstrate the suitability of VR for use with socially anxious adolescents given the clinically significant SUDS ratings between the two groups (socially anxious vs. not) in the expected direction within the VR social scenarios. Youth in this study also experienced acceptable levels of presence and immersion, suggesting that the VR environments were believable and engaging. While theory suggests there is a relationship between presence and treatment outcomes for specific phobias, this research is very limited. Two studies found presence to be unrelated to treatment outcome for VRET for specific phobias or social phobia (Krijn, Emmelkamp, Biemod, et al., 2004; Price & Anderson, 2007). In contrast, involvement, a specific aspect of presence, has been found to correlate with VR exposure treatment outcomes (Price, Mehta, Tone, & Anderson, 2011).

It is also interesting that, while there was no difference between study groups with regard to immersion or key items related to presence or immersion (Figure 4), the overall presence score was higher among nonsocially anxious youth than socially anxious youth. This is especially notable since socially anxious youth had significantly higher SUDS ratings in both VR environments. This conflicts with findings from Price and Anderson (2007) and Price, Mehta, Tone, and Anderson (2011) that identified a relationship between presence scores and VR session peak SUDS ratings. The reasons for this discrepancy may lie in the different measures of presence used (similar in the Price & Anderson, 2007 study; different from the Price et al., 2011 study), a focus on peak SUDS compared to average SUDS ratings used in this study or the disparate populations targeted (youth vs. adult). Moreover, Price et al. (2011) only found a relationship between the presence construct of realness and peak SUDS score but did not find a relationship between spatial or involvement presence constructs and SUDS ratings. In reviewing our scale, it does seem to be more oriented toward spatial presence and involvement. Regardless of this potential discrepancy and its reasons, it is clear that additional research is needed to understand the necessity of presence for VR anxiety treatment for adults and youth, as well as ways to increase presence if it found to be an important mediator of treatment outcome (Price et al., 2011).

In addition to clinical measures, qualitative comments from participants further support the utility and feasibility of VRET. Some of the comments also supported recent findings from Powers and colleagues (2013), which suggest that real time interaction with avatars within the VR environment may make these settings more realistic and useful for social anxiety treatment.

Given the feasibility and acceptability of VR with youth demonstrated in this study, future research should focus on developing cost-effective VR treatment and assessment modalities for adolescents with SAD and compare them with in vivo exposure treatment protocols. If comparable or superior to in vivo methods, VRET may become a preferred treatment of choice for youth for several reasons. First, VR provides a private, controlled method in a clinical setting to desensitize youth to social stimuli with low risk, where they are not required to take real social risks among their peers during the exposure process. Second, VR can offer youth multiple realistic opportunities to practice their social skills and build self-efficacy in a safe, virtual environment prior to transferring these skills into the real world. VR also provides the therapist with more easily accessible social scenarios, as in vivo situations often require a great deal of planning and out of office travel. In addition, the therapist can provide more detailed coaching for youth as they experience various social scenarios, as youth are able to hear the therapist through the speakers in the VR helmet while immersed in various VR environments. Finally, the cost of purchasing a VR system is now significantly less expensive. A system that performs similarly to the system used in this study would cost a treatment center approximately US$5,000 and could serve many clients simultaneously each week.

Future research should also seek to examine the feasibility and acceptability of using VR to assess and treat other adolescent anxiety disorders that have been efficacious with adults, such as fear of flying or other phobias. Likewise, additional VR scenarios should be designed more specifically for youth to further increase acceptability and realness.

There are several limitations of this pilot study. First, the generalizability of this study is limited, as it relies on a small sample of adolescents that met inclusion criteria. As such, future studies should replicate this study to ascertain whether different populations of socially anxious adolescents—with and without comorbid conditions—respond in a similar manner and how disparate socially anxious youth perceive and react to varied VR social scenarios. Likewise, while research suggests the SUDS is highly correlated with anxiety related physiological data (Thyer et al., 1984), future studies should also incorporate physiological measures of social anxiety such as heart rate and skin conductance. Similarly, while there may be concerns regarding the self-report nature of the SUDS and the potentially reactive nature of the data collection process, this approach offered the most valid and reasonable option for data collection during exposure to the VR environment, given its brevity and validity. Future studies, especially if focused on treatment outcome (which ours does not), should incorporate other less reactive anxiety outcome measures.

This study also relied on the LSAS-CA to determine whether individuals were identified with or without SAD. Although this may be problematic for a treatment study given the lack of a structured clinical interview, we were not implementing a treatment study and did not have the resources to conduct a lengthy clinical interview. However, despite this potential limitation, the LSAS and LSAS-CA were designed to support the diagnosis of SAD, and the convergent validity of the LSAS (upon which the LSAS-CA was developed) has been established with clinician interview methods (Heimburg et al., 1999).

Finally, some may criticize the ecological validity of the use of an adult audience for the presentation environment with youth. While not ideal, we believe that any public speaking scenario (adults and peers alike) is likely to serve as a useful exposure modality, especially given our findings that socially anxious youth had significantly higher SUDS ratings compared to nonsocially anxious youth. Despite these limitations, this study provides the first known study that has tested the feasibility of VRET and assessment for socially anxious adolescents. The findings were very promising, suggesting future study and VR environment development for adolescents are warranted.

VR appears to be a promising, cost-effective, feasible, and acceptable technology for providing social anxiety exposure for assessment and treatment of adolescents. VR technologies, such as those used in this study, may increase access to assessment and treatment of vulnerable populations. VR simulations may also increase the efficiency of practice, particularly when resources are limited. Future research should assess its use with diverse youth populations, as well as compare the efficacy of VR and in vivo exposure social anxiety treatment approaches. If VR approximates in vivo exposure among socially anxious youth, future research should focus on developing and testing cost-effective cognitive–behavioral VR exposure assessment and treatment modalities that prevent chronic social anxiety symptoms and the development of other highly comorbid conditions among youth that persist into adulthood.

	Socially Anxious	Nonsocially Anxious	χ²/t
Gender					χ² = .299	.585
Male	6	30.0	8	38.1
Female	14	70.0	13	61.9
Race					χ² = .991	.609
White	8	40.0	8	40.0
African American	7	38.9	6	30.0
Other	3	16.7	6	30.0
Hispanic
Yes	9	45.0	12	57.1	χ² = .606	.436
No	11	55.0	9	42.9
	M	SD	M	SD
Age	16.15	1.76	15.81	1.57	t = .655, df = 39	.516
SESS^a	53.90	14.90	70.61	10.85	t = −.4.05, df = 38	<.001
Baseline SUDS	8.57	15.38	2.00	4.71	t = 1.83, df = 22.37	.080
LSAS-CA^b	56.65	20.98	17.57	7.33	t = 7.89, df = 23.37	<.001

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by a New Faculty Grant at the University of Houston (PI: Danielle Parrish, Ph.D.).

Acknowledgments

Edgar Ramirez, a bachelor’s student in the Psychology department at the University of Houston, provided assistance as a volunteer research assistant during the data collection process. The Virtual Reality Lab at the University of Houston also donated its equipment and lab time to support this study.

References

Anderson

Jacobs

C. H.

Lindner

G. K.

Edwards

Zimand

Hodges

Rothbaum

B. O.

(2006). Cognitive behavior therapy for fear of flying: Sustainability of treatment gains after September 11. Behavior Therapy, 37, 91–97. doi:https://dx-doi-org.web.bisu.edu.cn/10.1016/j.beth.2005.05.001

Anderson

Jacobs

Rothbaum

B. O.

(2004). Computer-supported cognitive behavioral treatment of anxiety disorders. Journal of Clinical Psychology, 60, 253–267. doi:10.1002/jclp.10262

Anderson

Rothbaum

B. O.

Hodges

L. F.

(2003). Virtual reality exposure in the treatment of social anxiety. Cognitive and Behavioral Practice, 10, 240–247. doi:https://dx-doi-org.web.bisu.edu.cn/10.1016/S1077-7229(03)80036-6

Anderson

P. L.

Zimand

Hodges

L. F.

Rothbaum

B. O.

(2005). Cognitive behavioral therapy for public-speaking anxiety using virtual reality for exposure. Depression and Anxiety, 22, 156–158. doi:10.1002/da.20090

Anderson

Zimand

Schmertz

S. K.

Ferrer

(2007). Usability and utility of a computerized cognitive-behavioral self-help program for public speaking anxiety. Cognitive and Behavioral Practice, 14, 198–207. doi:https://dx-doi-org.web.bisu.edu.cn/10.1016/j.cbpra.2006.02.006

Barlow

J. H.

Ellard

D. R.

Hainsworth

J. M.

Jones

F. R.

Fisher

(2005). A review of self-management interventions for panic disorders, phobias and obsessive-compulsive disorders. Acta Psychiatrica Scandinavica, 111, 272–285. doi:10.1111/j.1600-0447.2005.00499.x6

Benjamin

C. L.

O’Neil

K. A.

Crawley

S. A.

Beidas

R. S.

(2010). Patterns and predictors of subjective units of distress in anxious youth. Behavioural and Cognitive Psychotherapy, 38, 497–504.

Bordnick

P. S.

(2011). Virtual reality alcohol party craving assessment, (Version 1.0) [PC], Santa Barbara, CA: World Viz, LLC. VIZARD.

Bordnick

P. S.

Traylor

A. C.

Carter

B. L.

Graap

K. M.

(2012). A feasibility study of virtual reality-based coping skills training for nicotine dependence. Research on Social Work Practice, 22, 293–300.

10.

Botella

Banos

Perpina

Villa

Alcaniz

Rey

(1998). Virtual reality treatment of claustrophobia: A case report. Behaviour Research and Therapy, 36, 239–246.

11.

Buckner

J. D.

Schmidt

N. B.

Lang

A. R.

Small

J. W.

Schlauch

R. C.

Lewinsohn

P. M.

(2008). Specificity of social anxiety disorder as a risk factor for alcohol and cannabis dependence. Journal of Psychiatric Research, 42, 230–239.

12.

Buckner

J. D.

Turner

R. J.

(2009). Social anxiety disorder as a risk factor for alcohol use disorders: A prospective examination of parental and peer influences. Drug and Alcohol Dependence, 100, 128–137.

13.

Carlin

A. S.

Hoffman

H. G.

Weghorst

(1997). Virtual reality and tactile augmentation in the treatment of spider phobia: A case report. Behaviour Research and Therapy, 35, 153–158.

14.

Chavira

D. A.

Stein

M. B.

Bailey

Stein

M. T.

(2004). Child anxiety in primary care: Prevalent but untreated. Depression and Anxiety, 20, 155–164.

15.

Coelho

C. M.

Waters

A. M.

Hine

T. J.

Wallis

(2009). The use of virtual reality in acrophobia research and treatment. Journal of Anxiety Disorders, 23, 563–574.

16.

Deacon

B. J.

Abramowitz

J. S.

(2004). Cognitive and behavioral treatmens for anxiety disorders: A reviwe of meta-analytic findings. Journal of Clinical Psychology, 60, 429–441.

17.

Essau

C. A.

Conradt

Petermann

(1999). Frequency and comorbidity of social phobia and social fears in adolescents. Behaviour Research and Therapy, 37, 831–843.

18.

Feske

Chambless

(1995). Cognitive behavioral versus exposure only treatment for social phobia: A meta-analysis. Behavior Therapy, 26, 695–720.

19.

Garau

Slater

Pertaub

D.-P.

Razzaque

(2005). The responses of people to virtual humans in an immersive virtual environment. Presence: Teleoperators and Virtual Environments, 14, 104–116. doi:10.1162/1054746053890242

20.

Garcia-Lopez

Olivares

Beidel

Albano

Turner

Rosa

(2006). Efficacy of three treatmetn protocols for adolescents with social anxiety disorder: A 5-year follow-up assessment. Journal of Anxiety Disorders, 20, 175–191.

21.

Gaudiano

B. A.

Herbert

J. D.

(2003). Preliminary psychometric evaluation of a new self-efficacy scale and its relationship to treatment outcome in social anxiety disorder. Cognitive Therapy and Research, 27, 537–555.

22.

Gaudiano

B. A.

Herbert

J. D.

(2006). Self-efficacy for social situations in adolescents with generalized social anxiety disorder. Behavioural and Cognitive Psychotherapy, 35, 209–223.

23.

Grant

B. F.

Hasin

D. S.

Blanco

Stinson

F. S.

Chou

S. P.

Goldstein

R. B.

Huang

(2005). The epidemiology of social anxiety disorder in the United States: Results from the National Epidemiologic Survey on Alcohol and Related Conditions. Journal of Clinical Psychiatry, 66, 1351–1361.

24.

Harris

S. R.

Kemmerling

R. L.

North

M. M.

(2002). Brief virtual reality therapy for public speaking anxiety. CyberPsychology & Behavior, 5, 543–550.

25.

Heimburg

R. G.

Horner

K. J.

Juster

H. R.

Safran

S. A.

Brown

E. J.

Schneier

F. R.

Liebowitz

M. R.

(1999). Psychometric properties of the Liebowitz Social Anxiety Scale. Psychological Medicine, 29, 199–212.

26.

Herbert

J. D.

Gaudiano

B. A.

Rheingold

A. A.

Moitra

Myers

V. H.

Dalrymple

K. L.

Brandsma

L. L.

(2009). Cognitive behavior therapy for generalized social anxiety disorder in adolescents: A randomized controlled trial. Journal of Anxiety Disorders, 23, 167.

27.

Kaplan

D. M.

Smith

Coons

(1995). A validity study of the subjective unit of discomfort (SUD) score. Measurement and Evaluation in Counseling and Development, 27, 195–199.

28.

Kessler

R. C.

Chiu

W. T.

Demler

Walters

E. E.

(2005). Prevalence, severity, and comorbidity of 12-month DSM-IV disorders in the National Comorbidity Survey Replication. Archives of General Psychiatry, 62, 617.

29.

Khalid-Khan

Santibanez

McMichen

Rynn

M. A.

(2007). Social anxiety disorder in children and adolescents: Epidemiology, diagnosis, and treatment. Pediatric Drugs, 9, 227–237.

30.

Kim

Bae

Park

Y. C.

(2008). Validity of the ubjective Units of Disturbance Scale (SUDS) in EMDR. Journal of EMDR Practice and Research, 1, 67–72.

31.

Klinger

Bouchard

Légeron

Roy

Lauer

Chemin

Nugues

(2005). Virtual reality therapy versus cognitive behavior therapy for social phobia: A preliminary controlled study. CyberPsychology & Behavior, 8, 76–88.

32.

Krijn

Emmelkamp

P. M. G.

Biemond

de Wilde de Ligny

Schuemie

M. J.

van der Mast

C. A. P. G.

(2004). Treatment of acrophobia in virtual reality: The role of immersion and presence. Behavior Research and Therapy, 42, 229–239.

33.

Krijn

Emmelkamp

P. M. G.

Olafsson

R. P.

Biemond

(2004). Virtual reality exposure therapy of anxiety disorders: A review. Clinical Psychology Review, 24, 259–281. doi:https://dx-doi-org.web.bisu.edu.cn/10.1016/j.cpr.2004.04.001

34.

Landon

T. M.

Barlow

D. H

, (2004). Cognitive-behavioral treatment fo panic disorder: Current status. Journal of Psychiatric Practice, 10, 211–226.

35.

Last

C. G.

Strauss

C. C.

(1990). School refusal in anxiety-disordered children and adolescents. Journal of the American Academy of Child and Adolescent Psychiatry, 29, 31–35.

36.

Masia-Warner

Storch

E. A.

Pincus

D. B.

Klein

R. G.

Heimberg

R. G.

Liebowitz

M. R.

(2003). The Liebowitz social anxiety scale for children and adolescents: An initial psychometric investigation. Journal of the American Academy of Child & Adolescent Psychiatry, 42, 1076–1084.

37.

Opris

Pintea

Garcia-Palacios

Botella

SzamoskÃzi

David

(2012). Virtual reality exposure therapy in anxiety disorders: A quantitative meta-analysis. Depression & Anxiety (1091-4269), 29, 85–93. doi:10.1002/da.20910

38.

Powers

M. B.

Briceno

N. F.

Gresham

Jouriles

E. N.

Emmelkamp

P. M. G.

Smits

J. A. J.

(2013). Do conversatins with virtual avatoars increase feeligns of social anxiety? Journal of Anxiety Disorders, 27, 398–403.

39.

Powers

M. B.

Emmelkamp

P. M.

(2008). Virtual reality exposure therapy for anxiety disorders: A meta-analysis. Journal of Anxiety Disorders, 22, 561–569.

40.

Price

Anderson

(2007). The role of presence in virtual reality exposure therapy. Journal of Anxiety Disorders, 21, 742–751.

41.

Price

Mehta

Tone

E. B.

Anderson

P. L.

(2011). Does engagement yield better outcomes? Components of presence as a predictor of treatment response for virtual reality exposure therapy for social phobia. Journal of Anxiety Disorders, 25, 763–770.

42.

Repetto

Gaggioli

Pallavicini

Cipresso

Raspelli

Riva

(2011). Virtual reality and mobile phones in treatment of generalized anxiety disorders: A phase-2 clinical trial. Perssonal and Ubiquitious Computing, 17, 253–260.

43.

Rizzo

A. A.

Klimchuk

Mitura

Bowerly

Buckwalter

J. G.

Kerns

Yeh

(2009). A virtual reality scenario for all seasons: The virtual classroom. Paper presented at the 11th International Conference on Human Computer Interaction, Bonn, Germany. Retrieved from http://ict.usc.edu/pubs/A%20Virtual%20Reality%20Scenario%20for%20All%20Seasons-%20The%20Virtual%20Classroom%20%28HCI%29.pdf

44.

Rothbaum

B. O.

Anderson

Zimand

Hodges

Lang

Wilson

(2006). Virtual reality exposure therapy and standard (in vivo) exposure therapy in the treatment of fear of flying. Behavior Therapy, 37, 80–90.

45.

Rothbaum

B. O.

Hodges

L. F.

Kooper

Opdyke

(1995). Effectiveness of computer-generated (virtual reality) graded exposure in the treatment of acrophobia. The American Journal of Psychiatry, 152, 626–628.

46.

Rothbaum

B. O.

Hodges

L. F.

Ready

Graap

Alarcon

R. D.

(2001). Virtual reality exposure therapy for Vietnam veterans with posttraumatic stress disorder. Journal of Clinical Psychiatry, 62, 617–622.

47.

Rothbaum

B. O.

Hodges

Smith

Lee

J. H.

Price

(2000). A controlled study of virtual reality exposure therapy for the fear of flying. Journal of consulting and Clinical Psychology, 68, 1020–1026.

48.

Rothbaum

B. O.

Schwartz

A. C.

(2002). Exposure therapy for posttraumatic stress disorder. American Journal of Psychotherapy, 56, 59–75.

49.

Saarni

S. I.

Suvisaari

Sintonen

Pirkola

Koskinen

Aromaa

Lönnqvist

(2007). Impact of psychiatric disorders on health-related quality of life: General population survey. The British Journal of Psychiatry, 190, 326–332.

50.

Safir

M. P.

Wallach

H. S.

Bar-Zvi

(2012). Virtual reality cognitive-behavior therapy for public speaking anxiety: One-year follow-up. Behavior Modification, 36, 235–246. doi:10.1177/0145445511429999

51.

Sarver

N. W.

Beidel

D. C.

Spitalnick

J. S.

(2014). The feasiblity and acceptability of virtual envirobnments in the treatmetn of childhood social anxiety disorder. Journal of Clinical Child & Adolescent Psychology, 43, 63–73. doi:10.1080/15374416.2013.843461

52.

Schneier

F. R.

Johnson

Hornig

C. D.

Liebowitz

M. R.

Weissman

M. M.

(1992). Social phobia: Comorbidity and morbidity in an epidemiologic sample. Archives of General Psychiatry, 49, 282.

53.

Scozzari

Gamberini

(2011). Virtual reality as a tool for cognitive behavioral therapy. In Brahnam

Jain

L. C.

(Eds.), Advanced computational intelligence paradigms in healthcare 6. Virtual reality in psychotherapy, rehabilitation, and assessment (pp. 63–108). Heidelberg, Germany: Springer Berlin.

54.

Stein

M. B.

Stein

D. J.

(2008). Social anxiety disorder. Lancet, 371, 1115–1125.

55.

Sweeney

Rapee

R. M.

(2005). Social anxiety in children and adolescents: Psychological treatments. In Crozier

W. R.

Alden

L. E.

(Eds.), The essential handbook of social anxiety for clinicians (pp. 153–165). West Sussex, England: John Wiley.

56.

Tanner

B. A.

(2012). Validity of global physical and emotional SUDS. Applied Psychophysiology Biofeedback, 37, 31–34.

57.

Thyer

B. A.

Papsdorf

J. D.

Davis

Vallecorsa

(1984). Autonomic correlates of the subjective anxiety scale. Journal of Behavior Therapy and Experimental Psychiatry, 15, 3–7.

58.

Witmer

B. G.

Singer

M. J.

(1998). Measuring presence in virtual environments: A presence questionnaire. Presence: Teleoperators and Virtual Environments, 7, 225–240.

59.

Wolpe

(1990). The practice of behavior therapy (4th ed.). New York, NY: Pergamon Press.

60.

Zimand

Anderson

Rothbaum

Hodges

(2001). The immersion questionnaire. Unpublished manuscript.

	Socially Anxious		Nonsocially Anxious		χ²/t
	(n = 20)		(n = 21)
	N	%	N	%		p
Gender					χ² = .299	.585
Male	6	30.0	8	38.1
Female	14	70.0	13	61.9
Race					χ² = .991	.609
White	8	40.0	8	40.0
African American	7	38.9	6	30.0
Other	3	16.7	6	30.0
Hispanic
Yes	9	45.0	12	57.1	χ² = .606	.436
No	11	55.0	9	42.9
	M	SD	M	SD
Age	16.15	1.76	15.81	1.57	t = .655, df = 39	.516
SESS^a	53.90	14.90	70.61	10.85	t = −.4.05, df = 38	<.001
Baseline SUDS	8.57	15.38	2.00	4.71	t = 1.83, df = 22.37	.080
LSAS-CA^b	56.65	20.98	17.57	7.33	t = 7.89, df = 23.37	<.001