Web-Based Acceptance and Commitment Therapy for Mental Health Problems in College Students

Abstract

There are significant challenges in addressing the mental health needs of college students. The current study tested an acceptance and commitment therapy (ACT), web-based self-help program to treat a broad range of psychological problems students struggle with. A sample of 79 college students was randomized to web-based ACT or a waitlist condition, with assessments at baseline and posttreatment. Results indicated adequate acceptability and program engagement for the ACT website. Relative to waitlist, participants receiving ACT improved on overall distress, general anxiety, social anxiety, depression, academic concerns, and positive mental health. There were no between-group effects on eating concerns, alcohol use, or hostility, or on some key ACT process of change measures. ACT participants improved more on mindful acceptance and obstruction to valued living, both of which mediated treatment outcomes. Results are discussed in the context of lessons learned with the website prototype, and areas for further research are presented.

Keywords

acceptance and commitment therapy self-help web-based transdiagnostic college students

Mental health problems are a significant issue among students at universities. Epidemiological studies indicate that up to almost 50% of students have a diagnosable psychological disorder each year (Blanco et al., 2008). Possibly, as a result of this prevalence, there are challenges with mental health service delivery in universities. Surveys of counseling center directors indicate that demands for services continue to increase along with growing severity/complexity of cases seen (Gallagher, 2014). Yet, counseling center resources have not increased to match these demands, with an approximate average of one counselor position per 1,459 students on campus (Association for University and College Counseling Center Directors [AUCCD], 2014). Further complicating matters, many students who would benefit from treatment choose not to seek it, due to factors such as lack of time and stigma (Blanco et al., 2008; Hunt & Eisenberg, 2010).

Web-based self-help interventions have the potential to improve mental health service delivery in universities. Such interventions can be easily disseminated to an entire student body, providing a practical and cost-effective means to scale up services to help match the rates of problems seen. Providing this additional treatment option may serve to reduce the high service demands placed on counseling centers, while also increasing the overall numbers receiving treatment. For example, students who are not willing to seek face-to-face counseling due to issues such as lack of time, inconvenience, or perceived stigma, may be willing to use a self-help program that can be accessed privately at a time that is convenient. Web-based self-help may also increase access to students in rural settings where services are more limited (e.g., distance education), students on the waitlist to receive counseling, and students who have completed counseling but want additional support.

One particularly promising approach for providing web-based self-help to college students is acceptance and commitment therapy (ACT; Hayes, Strosahl, & Wilson, 2012). ACT is a contextual cognitive behavioral therapy that focuses on a combination of acceptance, mindfulness, values, and traditional behavior change methods. These treatment methods are used in ACT to reduce a core pathological process called psychological inflexibility, in which behaviors are rigidly guided by thoughts, feelings, and other psychological experiences, rather than one’s values (what would be meaningful) or direct contingencies (what would be effective). By reducing psychological inflexibility, ACT seeks to treat a very broad range of psychological problems. Consistent with this, ACT has been found in more than 100 randomized controlled trials (RCTs; Hooper & Larsson, 2015) to effectively treat problems relevant to college students, including depression, anxiety, stress, substance abuse, eating disorders, test and public speaking anxiety, obsessive compulsive disorder, and coping with health-related problems, among many other areas (A-Tjak et al., 2015; Bluett, Homan, Morrison, Levin, & Twohig, 2014; Lee, An, Levin, & Twohig, 2016).

The transdiagnostic focus of ACT fits particularly well for the college setting (Hayes, Pistorello, & Levin, 2012). Although students are likely to experience specific psychological disorders, counseling centers typically treat a broader range of problems and do not necessarily focus on treating specific diagnoses (AUCCD, 2014; Gallagher, 2014). This is due to the breadth of problems typically seen in the college setting, including academic challenges, relationship problems, family of origin concerns, subthreshold disorders, general stress, coping with discrimination and diversity concerns, and so on. A transdiagnostic approach such as ACT can help ensure broad coverage of psychological problems typically seen. This has the added benefit for implementation by providing “one stop shopping” for referring and accessing mental health services—rather than navigating a suite of programs based on one’s specific presenting problem.

A series of recent studies have begun to evaluate the efficacy of ACT in web-based self-help formats. For example, RCTs have found ACT to be effective in a web-based self-help format for depression (e.g., Lappalainen et al., 2014), smoking (e.g., Bricker, Wyszynski, Comstock, & Heffner, 2013), and chronic pain (e.g., Trompetter, Bohlmeijer, Veehof, & Schreurs, 2015). However, these studies have focused on testing ACT for more specific-problem areas, rather than evaluating its transdiagnostic impact.

Three published studies have been conducted on web-based ACT for college students specifically (Chase et al., 2013; Levin, Pistorello, Hayes, & Seeley, 2014; Levin, Pistorello, Hayes, Seeley, & Levin, 2015). These studies found promising results suggesting web-based ACT may be acceptable and helpful to students for improving academic success (Chase et al., 2013), preventing mental health problems (Levin et al., 2014), and as an adjunctive treatment along with counseling (Levin et al., 2015). The two studies testing ACT as a transdiagnostic treatment found positive effects on depression and anxiety, but did not assess a broader range of problems (Levin et al., 2014; Levin et al., 2015). Furthermore, all these studies tested more limited ACT programs that were brief (1-3 sessions) and did not target all the key components of ACT (particularly leaving out some of the mindfulness-focused components such as cognitive defusion and present moment awareness). No studies to date have specifically evaluated a complete ACT web-based self-help program for treating the range of problems students might struggle with.

The current study, thus, sought to evaluate a transdiagnostic web-based self-help program that includes all the components of ACT and is designed to treat a wide range of problems among college students. The program was evaluated in a pilot randomized trial relative to a waitlist condition. Predictions included that the ACT website would be acceptable to college students, would improve a wide range of outcomes and processes of change relative to waitlist, and that changes in ACT processes would mediate treatment outcomes.

Method

Participants

The sample consisted of 79 college students who were 18 years of age or older and currently enrolled at a mid-size university in the Mountain West region of the United States. The sample was 66% female with a mean age of 20.51 (SD = 2.73 years, mode = 18). The sample was predominantly White (88%) with 3% reporting being Asian, 3% Hispanic, 3% multiracial, 1% African American, and 1% Native Hawaiian or other Pacific Islander; 14% of the sample described themselves as Hispanic/Latino ethnicity. The majority of participants were in their first year of college (47%), with 20% in their second year, 15% in their third year, 17% in their fourth or fifth year, and 1% in graduate school. Participants were not excluded for being in other concurrent treatment, with 13% reporting seeing a therapist and 19% being on a psychological medication in the past 6 weeks (24% were receiving at least one of these treatments).

The study specifically sought to recruit those who self-identified as being in distress and interested in receiving web-based self-help, as emphasized in recruitment materials. However, no formal screening was provided on these variables. This was done to maximize the breadth of participants included in the study that might benefit from web-based self-help and those who would potentially be offered such services (i.e., anyone interested in going to such a website and using it). The Counseling Center Assessment of Psychological Symptoms (CCAPS-34; Center for Collegiate Mental Health [CCMH], 2012) provides validated cutoff scores for mildly severe and clinically significant problems across a range of areas students might struggle with. Based on this information at baseline, 87% of participants had at least mildly severe scores on at least one CCAPS subscale and 68% had scores in the range indicating a clinically significant problem on at least one subscale. The sample varied in terms of types of clinically significant problems: 45% were in the clinical range for depression, 40% social anxiety, 30% general anxiety, 30% eating problems, 24% academic concerns, 22% hostility, and 14% alcohol problems.

The study was run in two cohorts across the fall 2014 (n = 30) and spring 2015 semesters (n = 49). Measures were added in the spring cohort to further examine level of interest in web-based self-help and reasons for participating (86% reported participating for extra credit, with only 6% participating for help with a psychological problem/personal distress, and 8% for both extra credit and help with distress). Consistent with this, 86% reported learning of the study through the web-based research participant management platform, Sona Systems (for extra credit assignments), and only one participant learned of the study from the counseling or student health center (despite active attempts to recruit in these settings). Finally, the majority of participants agreed with the statement “I am interested in using the self-help website being tested in this study” (67% reported a 5 = slightly agree or higher on a 7-point scale; M = 5.13, SD = 1.25), but 29% gave a neutral neither agree nor disagree response. Thus, the study appeared generally successful with regard to recruiting at least a minimally distressed sample of students. However, the participants primarily were recruited through the Sona research platform and participated for extra credit rather than specifically for help with a psychological problem.

Procedures

Participants were recruited through an online research posting on the Sona research platform, brief in-class presentations, and flyers posted throughout campus. Relevant student service departments/offices (i.e., the health center and counseling center) were also informed of the study and provided with recruitment materials.

All participation was completed remotely online through email and Qualtrics (a web-based survey platform) to enhance external validity in testing a self-help website and to increase feasibility for students to participate in rural settings. Interested students were provided with a link to the brief online screening, which asked if the individual was 18 years or older. Those who were above the age of 18 continued to an online consent form and baseline survey that included assessments of demographics, psychological symptoms, and ACT-related therapeutic processes of change.

After completing the baseline survey, participants were automatically randomized (in blocks of 10) through Qualtrics to either the waitlist or ACT condition. Participants in the ACT condition were directed to complete all six self-help sessions over the following 4 weeks, while those in the waitlist condition were asked to wait 4 weeks before completing the posttreatment assessment, after which they received unrestricted access to the ACT website. Program usage for participants in the ACT condition was regularly monitored by research assistants through the Qualtrics system to track progress in completing the self-help sessions. Research assistants gave reminder prompts via email and phone to those who were falling behind on completing a session every 4 days (i.e., 2 days late email, 4 days late phone call, 7 days late email, 10 days late phone call, 14 days late final email).

All participants were directed to an online posttreatment assessment 4 weeks after baseline, which included the same measures as baseline. Those in the ACT condition were also asked to complete measures of program usability and satisfaction. All participants enrolled in courses offering extra credit received 2 research credits as compensation for their time.

The participant flow diagram outlines the number of participants completing each of these study steps (see Figure 1). Note that one participant was removed after completing baseline several times and being assigned to both conditions (due to repeated random assignments), which reduced the sample size from 80 to the reported 79 participants.

Figure 1.

Participant flow diagram.

ACT Self-Help Website

The web-based ACT program was developed and hosted on Qualtrics, a flexible and secure web-based survey development program. The program content was adapted from empirically validated ACT and self-help protocols, which provide a host of therapeutic exercises, metaphors, and techniques that translate well into a web-based format. The website format and content was refined based on initial usability testing and think-aloud feedback collected from college students viewing earlier iterations of the program.

Participants completed a series of six web-based sessions in a tunneled sequence (i.e., sessions had to be completed in a particular order). Each session focused on a specific ACT component: Session 1 (costs of experiential avoidance), Session 2 (defusion), Session 3 (mindfulness), Session 4 (acceptance of difficult emotions), Session 5 (clarifying personal values), and Session 6 (committed action and goal setting). A transdiagnostic approach was taken in which sessions taught core ACT skills found to be helpful for a range of disorders rather than focusing on more disorder-specific issues and applications. A range of examples were provided in each session to help clarify how these skills might apply to specific problems students encounter. The program also heavily emphasized reflecting on personal experiences and applying skills in one’s life to support generalization of these transdiagnostic ACT skills.

After completing each session, participants were provided a brief therapeutic homework assignment (e.g., practicing a breathing mindfulness exercise) and were directed to wait 4 days before moving on to the next session. Follow-up emails were sent with the link for the next session along with the download links for any multimedia or worksheets provided within the previous session.

The program sessions were primarily text-based, but also made heavy use of multimedia (e.g., images, audio recordings, and videos for experiential exercises) and interactive features (e.g., worksheets, assessments with tailored feedback, expandable text/popup features). These multimedia and interactive features were implemented throughout each session, in an aim to maintain engagement levels and provide experiential training in ACT skills.

Of note, this study used Qualtrics, an existing web-based platform, for developing the intervention. This provides a highly cost-effective and efficient method for prototyping web-based self-help interventions, while minimizing the notable technical expertise, development costs, and time typically taken to create even relatively short web-based interventions. Although Qualtrics is designed for survey administration, it provides a host of key features and tools that are often costly to implement in a secure and stable website (e.g., a range of interactive questions/tools, database integration for tracking user behavior, ability to tailor information and carry data forward in a session, and stable and consistent performance across browsers and devices, including mobile phones). If such an approach is acceptable to participants, it could highlight a very feasible method for behavioral scientists to test prototype web-based interventions relatively quickly and at little to no cost.

Measures

CCAPS-34

The 34-item version of the CCAPS served as the primary outcome for this study. The CCAPS was developed specifically to assess a broad range of mental health problems among college students with subscales for depression, general anxiety, social anxiety, academic concern, eating concerns, hostility, alcohol use, and an overall total score of distress. Items are rated on a 5-point scale ranging from 0 (not at all like me) to 4 (extremely like me). The CCAPS has been found to have good reliability and validity in past studies with distressed college students (CCMH, 2012). The CCAPS had adequate internal consistency in the current study: Distress Total Score, α = .92; Depression, α = .91; Eating Concerns, α = .91; Alcohol, α = .88; Social Anxiety, α = .83; General Anxiety, α = .79; Hostility, α = .79; and Academic Concern, α = .73.

Mental Health Continuum–Short Form (MHC-SF)

The MHC-SF is a 14-item measure of positive mental health, composed of three subscales assessing emotional, psychological, and social well-being. Items assess dimensions of well-being, including positive affect, satisfaction with life, social integration, social contribution, autonomy, personal growth, purpose in life, and self-acceptance. This was a key secondary measure given the focus in ACT on positive functioning outcomes. Items are rated on a 6-point scale ranging from never to every day. The MHC-SF has been found to have adequate reliability and validity in past research (Keyes, 2005) and to be sensitive to detecting the effects of ACT (Bohlmeijer, Lamers, & Fledderus, 2015). The internal consistency of the MHC was adequate in the current study: MHC Total Score, α = .93; MHC Emotional Well-Being, α = .89; MHC Social Well-Being, α = .84; and MHC Psychological Well-Being, α = .85.

Acceptance and Action Questionnaire–II (AAQ-II)

The seven-item AAQ-II served as the primary process measure. This is the most frequently used measure of psychological inflexibility, the key process targeted in ACT. Items are rated on a 7-point scale ranging from 1 (never true) to 7 (always true). The AAQ-II has been found to have adequate reliability and validity in past studies with college students (Bond et al., 2011). The internal consistency of the AAQ-II in the current study was α = .93.

Cognitive Fusion Questionnaire (CFQ)

The CFQ is a seven-item measure of cognitive fusion, an important sub-process that contributes to psychological inflexibility. Items are rated on a 7-point scale ranging from 1 (never true) to 7 (always true). The CFQ is a new measure, but the initial validation study found adequate reliability and validity with populations, including college students (Gillanders et al., 2014). The internal consistency of the CFQ in the current study was α = .95.

Valuing Questionnaire (VQ)

The VQ is a 10-item measure of values, another key sub-process related to psychological inflexibility. The VQ includes two subscales assessing progress in valued living and obstruction to valued living. Each item is rated on a 7-point scale ranging from 0 (not at all true) to 6 (completely true). The VQ is also a relatively new measure, but initial validation results indicate adequate reliability and validity (Smout, Davies, Burns, & Christie, 2014). The internal consistency for the VQ was adequate in the current study: VQ Progress, α = .89 and VQ Obstruction, α = .84.

Philadelphia Mindfulness Scale (PHLMS)

The PHLMS is a 20-item measure of mindfulness with two subscales assessing present moment awareness and acceptance. Deficits in awareness and acceptance are key sub-processes that contribute to psychological inflexibility. Items are rated on a 5-point scale ranging from 1 (never) to 5 (very often). The PHLMS has been found to have adequate reliability and validity in past studies (Cardaciotto, Herbert, Forman, Moitra, & Farrow, 2008). The internal consistency for the PHLMS was adequate in the current study: PHLMS Acceptance, α = .87 and PHLMS Awareness, α = .84.

System Usability Scale (SUS)

The SUS is a 10-item scale designed to assess program usability and acceptability (Tullis & Albert, 2008). Each item is rated on a 5-point scale ranging from 1 (strongly disagree) to 5 (strongly agree). Research summarized across 206 studies indicates that the SUS items load onto a single latent factor, have a high level of internal consistency (Cronbach’s α = .91) and can distinguish between more and less usable programs (Bangor, Kortum, & Miller, 2008). Additional program satisfaction items were used to assess features such as whether participants liked the program, found it helpful, and would recommend it to others, as well as how to further improve the program, based on items used in previous studies (Levin et al., 2014). The internal consistency of the SUS in the current study was α = .84.

Program usage

Qualtrics automatically collected data on participants’ use of the ACT website intervention, including time spent on pages, overall time spent within a session, and number of pages viewed. These data were utilized to examine program engagement and usage across all six sessions.

Analysis Plan

A total of 62 participants completed the posttreatment survey (78%), with 32 (80%) completing posttreatment in the ACT condition and 30 (77%) completing posttreatment in the waitlist condition. There was no difference in completion rates between conditions, χ² = .111, p = .74. An intent to treat approach was taken in which all participants were included in analyses. Missing data were modeled using mixed-model repeated-measures analyses (MMRM; for between-group effects) and maximum likelihood estimation (ML; for mediational analyses).

MMRM analyses were conducted using unstructured covariance matrices to examine time by condition effects on outcome and process measures. This method models missing data such that the full intent to treat sample is included, irrespective of missing data for some participants and time points. MMRM analyses involve a specific form of ML estimation that accounts for correlations among repeated measurements within-subjects. ML relies on the multivariate distribution of observed variables to arrive at a hypothetical covariance structure for the full data set. ML is preferred over traditional imputation techniques for missing data, such as last observation carried forward or multiple imputation under normal assumptions (Black, Harel, & McCoach, 2010; Enders, 2012).

The cross product of coefficient test with bootstrapping was used to test mediators of treatment effects. Analyses were conducted separately for each process measure that significantly improved in the ACT condition relative to waitlist. These analyses were conducted with each outcome measure in which there was a between-group effect. Mediation analyses used posttreatment outcome and posttreatment mediator scores controlling for both baseline outcome and baseline mediator as covariates. For mediational analyses, the ML algorithm in MPlus (Version 7.0; Muthén & Muthén, 2012) was used to estimate the covariance structure in the context of missing data.

Results

Preliminary Analyses

Independent sample t tests compared baseline scores on all outcome and process of change measures between conditions (total scores and subscales for the CCAPS, MHC, AAQ-II, CFQ, PHLMS, and VQ). Of 18 comparisons, a statistically significant difference was found on one measure, CCAPS Academic Concern subscale, t(76) = 2.21, p = .03, such that those in the ACT condition had higher academic concerns at baseline (ACT: M = 2.01, SD = 1.00; Waitlist: M = 1.57; SD = 0.72). The baseline academic concern measure was included as a covariate for MMRM analyses to control for this baseline difference between conditions (of note, results are very similar with or without the inclusion of this covariate). There were no other significant differences between conditions on subscale or total scores.

Each outcome and process of change measure was examined for assumptions of normality through skewness and kurtosis statistics. Only the CCAPS Alcohol subscale had a notable nonnormal distribution (baseline skewness = 2.34, kurtosis = 5.26; posttreatment skewness = 2.02, kurtosis = 2.86), which was addressed using an exponential transformation.

ACT Program Usage and Satisfaction

The majority of participants completed at least one session (83% completed Session 1) and at least half of the ACT program (75% completed Session 3). However, only 55% completed all six self-help sessions. Participants spent on average 73.58 min in the ACT program (SD = 83.28). The average time spent in each session that a participant completed was 17.19 min (SD = 16.20), with the longest time spent in Session 1 (M = 24.83, SD = 18.63) and the shortest time spent in Session 3 (M = 10.50, SD = 12.10).

System usability ratings were in the adequate range (M = 71.13, SD = 16.78). Individual satisfaction items indicated adequate satisfaction: “Overall, I was satisfied with the quality of the program” (M = 4.52, SD = 1.06, 90% rated 4 “slightly agree” or higher), “The program was easy to use” (M = 5.00, SD = 1.00, 90% rated 4 or higher), “The program was helpful to me” (M = 4.26, SD = 1.26, 77% rated 4 or higher), “I felt the program was made for someone like me” (M = 3.94, SD = 1.32, 74% rated 4 or higher), “I would like to use the program again in the future” (M = 3.87, SD = 1.18, 61% rated 4 or higher), “I think the program would be helpful for distressed college students” (M = 4.81, SD = 1.22, 87% rated 4 or higher), and “I would recommend the program to other college students who are distressed and/or struggling” (M = 4.68, SD = 1.14, 90% rated 4 or higher).

Participants in the ACT condition were asked at posttreatment “What did you like least about the program?” These open responses were reviewed for common themes. The most frequent negative comment was that the program was too long and/or too repetitive (60% reported this). Other concerns included that they did not have the time or privacy to use the program (7%), that there was inadequate coverage for certain, specific mental health problems (7%), and that they did not like the web-based and self-help format for receiving services (3%).

Mental Health Outcomes

Does ACT improve mental health outcomes?

Descriptive data for each condition and time point on mental health outcomes are provided in Table 1. MMRM analyses examined Time × Condition effects when including baseline academic concern as a covariate (to control for differences between conditions at baseline). Analyses indicated significant Time × Condition interactions on outcome measures, including CCAPS Total Distress, F(1, 60.18) = 6.62, p = .013, Cohen’s d = .66; CCAPS Social Anxiety, F(1, 59.79) = 9.12, p = .004, Cohen’s d = .78; CCAPS Academic Concern, F(1, 60.25) = 5.75, p = .020, Cohen’s d = .62; MHC Total Score, F(1, 61.72) = 5.16, p = .027, Cohen’s d = .58; and MHC Social Well-Being, F(1, 62.27) = 7.36, p = .009, Cohen’s d = .69. In addition, there were trends for a Time × Condition interaction on CCAPS Depression, F(1, 64.12) = 3.95, p = .051, Cohen’s d = .50, and CCAPS General Anxiety, F(1, 60.81) = 3.38, p = .071, Cohen’s d = .47.

Table 1.

Means and Standard Deviations With Available Data Between Conditions on Process and Outcome Measures.

	ACT		Waitlist
	Pre M (SD)	Post M (SD)	Pre M (SD)	Post M (SD)
Outcome measures
CCAPS-Total	1.68 (0.81)	1.52 (0.91)	1.50 (0.76)	1.60 (0.78)
CCAPS-Depression	1.63 (1.13)	1.51 (1.16)	1.51 (1.05)	1.59 (1.00)
CCAPS-Anxiety	1.66 (0.88)	1.49 (0.91)	1.53 (0.91)	1.63 (0.97)
CCAPS-Social Anx.	2.20 (0.94)	1.88 (1.13)	2.15 (1.04)	2.23 (0.91)
CCAPS-Academic	2.01 (1.00)	1.80 (0.97)	1.57 (0.73)	1.69 (0.82)
CCAPS-Eating	1.09 (1.26)	0.98 (1.26)	1.18 (1.18)	1.12 (1.11)
CCAPS-Hostility	0.78 (0.73)	0.83 (0.82)	0.75 (0.58)	0.79 (0.73)
CCAPS-Alcohol	0.46 (0.93)	0.59 (1.05)	0.33 (0.65)	0.34 (0.78)
MHC-Total	51.13 (13.34)	56.34 (17.25)	55.95 (12.65)	55.77 (13.23)
MHC-Emotional	12.68 (3.13)	12.97 (4.05)	13.62 (2.83)	13.43 (3.21)
MHC-Social	15.80 (5.53)	18.59 (7.54)	17.85 (5.26)	17.50 (5.36)
MHC-Psychological	22.66 (5.98)	24.78 (6.80)	24.49 (5.96)	24.83 (5.88)
Process measures
AAQ-II	26.83 (9.51)	24.44 (9.02)	25.97 (10.70)	24.47 (10.47)
PHLMS-Acceptance	35.23 (8.07)	31.59 (7.43)	33.28 (6.68)	32.47 (6.89)
PHLMS-Awareness	33.23 (6.83)	34.02 (6.90)	34.70 (6.31)	35.17 (6.46)
CFQ	29.45 (10.65)	25.38 (9.90)	28.44 (10.19)	26.83 (10.06)
VQ-Obstruction	20.98 (6.92)	17.13 (6.00)	19.64 (6.20)	18.80 (6.60)
VQ-Progress	20.05 (6.27)	22.06 (6.33)	22.13 (6.09)	22.33 (5.76)

Note. This table only reports non-missing baseline and posttreatment data, but MMRM analyses also modeled the missing data. CCAPS = Counseling Center Assessment of Psychological Symptoms; MHC = Mental Health Continuum; AAQ-II = Acceptance and Action Questionnaire–II; PHLMS = Philadelphia Mindfulness Scale; CFQ = Cognitive Fusion Questionnaire; VQ = Valuing Questionnaire.

Post hoc analyses of each Time × Condition interaction indicated that outcomes only improved from pre- to posttreatment in the ACT condition: CCAPS Total Distress, slope estimate = −0.28, t(60.24) = −2.93, p = .005, Cohen’s d = .52; CCAPS Depression, slope estimate = −0.29, t(64.29) = −2.32, p = .024, Cohen’s d = .40; CCAPS General Anxiety, slope estimate = −0.28, t(60.92) = −2.21, p = .031, Cohen’s d = .39; CCAPS Social Anxiety, slope estimate = −0.39, t(59.84) = −3.85, p < .001, Cohen’s d = .69; CCAPS Academic Concern, slope estimate = −0.29, t(60.30) = −2.54, p = .014, Cohen’s d = .45; MHC Total Score, slope estimate = 5.90, t(61.67) = 3.45, p = .001, Cohen’s d = .60; and MHC Social Well-Being, slope estimate = 2.97, t(62.23) = 3.66, p = .001, Cohen’s d = .64. There were no significant pre- to post improvements on outcome measures in the waitlist condition (p > .10).

There were no significant Time × Condition interactions for CCAPS Eating Concerns, CCAPS Hostility, CCAPS Alcohol, MHC Emotional Well-Being subscale, or MHC Psychological Well-Being (p > .10).

Processes of Change

Does ACT improve psychological flexibility processes?

Another series of MMRM analyses examined Time × Condition interactions on ACT process of change measures (Table 1). Significant Time × Condition interactions were found for PHLMS Acceptance, F(1, 64.52) = 4.49, p = .038, Cohen’s d = .53, and VQ Obstruction, F(1, 62.50) = 6.64, p = .012, Cohen’s d = .65. In both cases, post hoc analyses indicated that ACT processes only improved from pre- to posttreatment in the ACT condition: PHLMS Acceptance, slope estimate = −3.44, t(64.45) = −3.56, p = .001, Cohen’s d = .62, and VQ Obstruction, slope estimate = −4.34, t(62.45) = −4.68, p < .001, Cohen’s d = .82. There were no significant Time × Condition interactions on the AAQ, CFQ, PHLMS Awareness, or VQ Progress subscale (p > .10).

Are treatment effects mediated by psychological flexibility?

Mediation analyses were conducted using the cross product of coefficients test with each psychological inflexibility process and mental health outcome that was found to improve with ACT relative to the waitlist condition (see Table 2). Significant mediation effects were found with both the PHLMS Acceptance and VQ Obstruction scales for CCAPS Total Distress, Depression, General Anxiety, and Social Anxiety. VQ Obstruction was also a significant mediator of CCAPS Academic Concern, MHC Total Positive Mental Health, and MHC Social Well-Being, but PHLMS Acceptance did not mediate these outcomes.

Table 2.

Mediation Analysis Results for Process and Outcome Measures That Significantly Improved in the ACT Versus Waitlist Condition.

	a path	b path	c path	c′ path	Products of coefficients			Proportion mediated (1 − c′ / c)
	X-M	M(X)-Y	X-Y	X(M)Y	Point estimate	SE	Bootstrapping 95% CI	Proportion mediated (1 − c′ / c)
PHLMS acceptance mediation model
CCAPS-Total	1.88^†	3.14**	2.28*	1.90^†	0.07	0.04	[0.007, 0.168]	20.06%
CCAPS-Depression	1.88^†	2.68**	2.03*	1.71^†	0.08	0.06	[0.009, 0.242]	22.13%
CCAPS-Anxiety	1.88^†	3.12**	1.67^†	1.39	0.09	0.06	[0.006, 0.235]	28.57%
CCAPS-Social Anxiety	1.88^†	3.82***	3.00**	2.87**	0.07	0.04	[0.007, 0.170]	16.05%
CCAPS-Academic	1.88^†	1.12	1.72^†	1.58	0.03	0.04	[−0.016, 0.151]
MHC	1.73^†	1.85^†	1.83^†	1.57	0.78	0.70	[−0.029, 2.980]
MHC-Social	1.73^†	1.69^†	2.19*	2.04*	0.32	0.31	[−0.027, 1.336]
VQ obstruction mediation model
CCAPS-Total	2.67*	4.37***	2.28*	1.25	0.16	0.07	[0.049, 0.327]	48.61%
CCAPS-Depression	2.67*	5.25***	2.03*	0.97	0.21	0.10	[0.053, 0.419]	57.94%
CCAPS-Anxiety	2.67*	3.69***	1.67^†	1.02	0.17	0.08	[0.052, 0.367]	50.44%
CCAPS-Social Anxiety	2.67*	3.96***	3.00**	2.60**	0.14	0.06	[0.042, 0.275]	29.18%
CCAPS-Academic	2.67*	1.96*	1.72^†	0.96	0.11	0.07	[0.008, 0.320]	38.99%
MHC	2.32*	3.88***	1.83^†	0.92	2.48	1.34	[0.219, 5.680]	53.19%
MHC-Social	2.32*	4.07***	2.19*	1.57	1.04	0.55	[0.118, 2.343]	37.84%

Note. ACT = acceptance and commitment therapy; X-M = treatment condition and mediator; M(X)-Y = mediator and outcome controlling for treatment condition; X-Y = treatment condition and outcome; X(M)Y = treatment condition and outcome controlling for mediator; CI = confidence interval; PHLMS = Philadelphia Mindfulness Scale; CCAPS = Counseling Center Assessment of Psychological Symptoms; MHC = Mental Health Continuum; VQ = Valuing Questionnaire. t-test values are reported for paths tested.

†

p < .10. *p < .05. **p < .01. ***p < .001.

Of note, most of these effects demonstrated full mediation (such that there was no longer a significant relationship of intervention on outcome after including the mediational path). The main exception was for social anxiety for which VQ Obstruction and PHLMS Acceptance were both only partial mediators. Proportion-mediated effect sizes for significant mediation effects ranged between 16.05% and 57.94%.

Discussion

This study sought to evaluate the acceptability and efficacy of a transdiagnostic web-based ACT program for college students. Results indicated that the ACT website was reasonably acceptable to students: Most completed at least half of the program, and satisfaction ratings were relatively high. The ACT website was found to improve a range of outcomes relative to a waitlist condition, although significant effects were not found on all outcomes (hostility, eating concerns, alcohol problems). Although the ACT website did not lead to improvements on all psychological flexibility measures, those processes that did improve significantly mediated treatment effects.

This study found transdiagnostic effects for the ACT website with improvements relative to waitlist on general distress, social anxiety, academic concerns, positive mental health, depression, and general anxiety. A transdiagnostic web-based approach such as this could provide a central resource for students struggling with a range of problems. This could help simplify triaging and referral to web-based self-help, providing a single website that students can visit and faculty/staff could refer to. Furthermore, such a program could provide greater flexibility in addressing the range of psychological problems, subthreshold disorders, and comorbid issues students might struggle with, all within the same intervention.

However, research on such a broadly transdiagnostic ACT approach is still too preliminary to clearly indicate whether it would actually be helpful for the range of problems students might seek treatment. For example, the current study did find some limiting areas where there were no between-group differences: eating concerns, hostility, alcohol, and some aspects of positive mental health. These null findings tended to occur with rarer problems, suggesting potential floor effects for detecting changes (e.g., eating concerns, alcohol problems in a predominantly religious culture that prohibits drinking). Alternatively, problem areas such as eating concerns and alcohol use might represent issues that require a more tailored approach to treatment and that cannot be effectively treated using a more general transdiagnostic method. Further research is needed to determine whether a transdiagnostic ACT approach focused on more general skills training does effectively treat each of a range of problems. For example, including larger sample sizes with adequate representation of each problem area to conduct moderation analyses and to test effects with specific subgroups on relevant outcome measures.

The transdiagnostic ACT approach is based on the theory that psychological inflexibility contributes to a very broad range of problems and that ACT can reduce psychological inflexibility to improve these outcomes. Thus, the process of change analyses are key to determining the potential efficacy of the ACT program. Results for processes of change were somewhat mixed. The AAQ-II did not significantly change relative to the waitlist. Some previous web-based ACT research with college students has similarly found a lack of effect with the AAQ-II suggesting it may be less sensitive to detecting treatment effects in this context (Levin et al., 2014). Some other process measures also did not improve, including mindful awareness, progress in valued living, and cognitive fusion. In contrast, obstruction in valued living and mindful acceptance both improved significantly from ACT. One advantage of using these more specific measures of psychological inflexibility sub-processes is that they may highlight areas for revision to the web-based intervention. It may be that the program’s content targeting present moment awareness, cognitive fusion/defusion, and following one’s values was relatively weak. However, the fact that acceptance and obstructions in living values both improved and mediated outcomes, suggests the website did have an active impact on at least some aspects of psychological inflexibility which led to improvements on a range of outcomes.

The study also sought to evaluate the acceptability of an ACT web-based self-help program for college students, particularly using the cost-effective prototyping strategy with Qualtrics. Although a program completion rate of around 50% is less than ideal for a website, it does fall within the range typically found in non-guided self-help websites (e.g., Calear, Christensen, Mackinnon, Griffiths, & O’Kearney, 2009; Richards & Richardson, 2012). This in part represents a choice in implementation strategy between maximizing program retention/effect size versus maximizing cost-effectiveness and scalability. It is known that research including guided support from a paraprofessional or professional leads to higher engagement rates and treatment effects (Andersson, 2014; Richards & Richardson, 2012). However, this approach also significantly increases the complexity of large-scale implementation and the resource cost per end user, which is particularly a concern in universities given limited resources. Although the current program did not effectively engage every student, it does provide a very cost-effective alternative service option for those willing to complete it. The fact that more than half of the students finished the program suggests this is a treatment option that many (though certainly not all) would be willing to use and complete.

The program engagement rates and satisfaction ratings are also notable given the prototype development approach that was taken. The ACT website was developed through Qualtrics, which allowed for rapid development and refinement of the self-help sessions, without requiring grant funding or significant resources. Content management systems and website survey platforms such as Qualtrics provide a variety of readily available features that a behavioral scientist can then use to create a session without requiring programming experience. Some of these features can be quite costly and time-consuming if done outside of a pre-built solution such as database integration, skip and carry forward logic, cross-platform compatibility, database security and hosting, and so on. Although there are advantages to more custom built websites, they take much more time and resources. These costs become even more significant when initial research identifies problems and areas requiring revision to the program (which is likely to occur especially in early development and pilot research). This study demonstrates a more cost-effective and efficient method to start testing and refining web-based self-help prototypes much more quickly. Such an approach can ultimately inform development of a more sophisticated, custom website, but avoids the significant delays often encountered when trying to begin with a more costly and time-consuming development process.

Overall, the pattern of findings highlights some potential areas for revisions to the ACT website. For example, program content related to process of change measures that did not improve might be revised to see if these changes can lead to stronger effects (i.e., defusion, mindful awareness, and committed action in valued living). In addition, many participants requested that the intervention be shorter and less repetitive, which if revised may lead to greater program retention. These and other revisions can be completed quickly and with minimal resources when using a prototype approach with an existing web-based platform like Qualtrics. In contrast, such changes could be prohibitively costly without additional funding when testing a more sophisticated, custom website solution. Thus, this project highlights the benefits of using existing development platforms to rapidly create a web-based program, test it, and then revise the program, all prior to engaging in much more costly and time-consuming efforts to create a highly refined website.

Due to the pilot nature of this study, there were some notable limitations. First, although the sample reported at least some distress, most students reported participating for psychology course credits, rather than seeking psychological help per se. This might have led to lower engagement rates, satisfaction ratings, and outcome/process findings than would be expected in a more direct treatment sample. The limitations with the sample may have also reduced power to detect intervention effects due to more restricted variances in this sample on some of the outcome measures (e.g., alcohol problems).

These characteristics of students who chose to participate raise questions regarding how an ACT self-help website might be best delivered in a college setting and whether web-based self-help will ultimately be successful for improving student services. Arguably, the most likely explanation for recruiting a sample consisting of participants seeking extra credit is the recruitment methods used, which relied heavily on the Sona research platform and class announcements targeting students seeking course credit. This means that the current study has not yet determined whether college students would be willing to participate in a self-help ACT website for purely mental health reasons, without additional incentives. It is unclear how best to motivate and inform students to use web-based self-help. There are many promising directions due to the wealth of access points universities provide, including dorms, residential advisers, word of mouth from peers, health services, student service programs and events, instructors, counseling center outreach events, and so on. Future studies should ensure that a more “real world” implementation method is used that goes beyond undergraduate research pools to test whether such methods are adequate at engaging students and to identify the most effective outreach methods. This is also critical in testing whether findings generalize to students who specifically participate for the purposes of receiving psychological help. There are likely barriers to seeking self-help that will need to be overcome, but research to date is limited on what might interfere with students seeking web-based self-help. Preliminary research suggests that some barriers may be similar to those we know are present in seeking face-to-face care, such as self-stigma (Lannin, Vogel, Brenner, Abraham, & Heath, 2016). Further research is needed that identifies and attempts to overcome barriers to engaging in web-based self-help among college students in need of services. Until such research is conducted, it is reasonable to be cautious with regard to the degree to which students will truly use such self-help resources absent receiving incentives like course credits.

The study was limited with regard to testing the transdiagnostic impact of ACT. This was due in part to a low sample size, which minimized statistical power to study less frequent problems such as eating concerns. Furthermore, although seeing an impact on a range of CCAPS outcomes helps support the transdiagnostic model, it does not specifically test whether the program is helpful across a broad range of clinical populations. Research with larger samples is needed to examine moderation and subgroups to determine if the program is equally helpful for those who struggle with specific problems.

The pilot study was structured to rapidly test the potential efficacy, acceptability, and areas for revision in an ACT self-help website. This led to some limitations in evaluating efficacy, including the lack of follow up assessments and the use of a waitlist control condition. The use of a waitlist is a relatively low bar for determining efficacy as it does not control for demand characteristics, expectancy, and other effects from accessing any type of materials intended to help students. Web-based research is promising in part because it allows for much more feasible and well controlled RCTs using active comparison conditions. Further research is needed to test the impact of a transdiagnostic ACT self-help website relative to such an active comparison program and on long-term outcomes.

In summary, this study found that an ACT self-help website was adequately acceptable to college students and led to improvements in a range of, but not all, outcomes. Furthermore, results were supportive of targeting key ACT processes of change, with improvements in mindful acceptance and obstructions to valued living mediating outcomes. The study also found some areas requiring further revision, which are particularly feasible given the prototyping approach that was taken to develop the website. One particularly key challenge that was identified for future research is how to best motivate and engage students to participate in web-based self-help outside of receiving extra course credit. Overall, transdiagnostic ACT appears to be a promising approach for web-based self-help and providing mental health support to college students.

Footnotes

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Dr. Levin is a research associate with Contextual Change LLC, a small business that seeks to develop commercialized web-based programs for mental health issues.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported in part by an Alan E. Hall Innovation for Undergraduate Student Success Award (PI: Michael E. Levin).

Author Biographies

Michael E. Levin is an assistant professor at Utah State University. His research focuses on web/mobile app interventions and mechanisms of change in acceptance and mindfulness-based therapies.

Jack A. Haeger is a graduate student in the combined clinical, counseling, and school psychology PhD program at Utah State University. His research focuses on developing/testing technological implementations of acceptance and other mindfulness-based therapies, as well as exploring therapeutic mechanisms of change.

Benjamin G. Pierce is a graduate student in the combined PhD program at Utah State University. His research focuses on acceptance and mindfulness-based interventions and skills, statistical modeling and measurement, and social justice issues.

Michael P. Twohig is an associate professor at Utah State University. His research focuses on behavior analysis, translational research, ACT, and the treatment of anxiety disorders.

References

Andersson

(2014). The internet and CBT: A clinical guide. CRC Press. Boca: Raton, FL.

Association for University and College Counseling Center Directors. (2014). The association for university and college counseling center directors annual survey. Retrieved from http://www.aucccd.org/assets/documents/2014%20aucccd%20monograph%20-%20public%20pdf.pdf

A-Tjak

J. G.

Davis

M. L.

Morina

Powers

M. B.

Smits

J. A.

Emmelkamp

P. M.

(2015). A meta-analysis of the efficacy of acceptance and commitment therapy for clinically relevant mental and physical health problems. Psychotherapy and Psychosomatics, 84, 30-36.

Bangor

Kortum

P. T.

Miller

J. T.

(2008). An empirical evaluation of the System Usability Scale. International Journal of Human-Computer Interaction, 24, 574-594.

Black

A. C.

Harel

McCoach

D. B.

(2010). Missing data techniques for multilevel data: Implications of model misspecification. Journal of Applied Statistics, 38, 1845-1865.

Blanco

Okuda

Wright

Hasin

D. S.

Grant

B. F.

Liu

S.-M.

. . . Olfson

(2008). Mental health of college students and their non-college-attending peers: Results from the national epidemiologic study on alcohol related conditions. Archives of General Psychiatry, 65, 1429-1437.

Bluett

E. J.

Homan

K. J.

Morrison

K. L.

Levin

M. E.

Twohig

M. P.

(2014). Acceptance and commitment therapy for anxiety and OCD spectrum disorders: An empirical review. Journal of Anxiety Disorders, 6, 612-624.

Bohlmeijer

E. T.

Lamers

S. M. A.

Fledderus

(2015). Flourishing in people with depressive symptomatology increases with acceptance and commitment therapy. Post-hoc analyses of a randomized controlled trial. Behaviour Research and Therapy, 65, 101-106.

Bond

F. W.

Hayes

S. C.

Baer

R. A.

Carpenter

Orcutt

H. K.

Waltz

. . . Zettle

R. D.

(2011). Preliminary psychometric properties of the Acceptance and Action Questionnaire–II: A revised measure of psychological flexibility and acceptance. Behavior Therapy, 42, 676-688.

10.

Bricker

Wyszynski

Comstock

Heffner

J. L.

(2013). Pilot randomized controlled trial of web-based acceptance and commitment therapy for smoking cessation. Nicotine & Tobacco Research, 15, 1756-1764.

11.

Calear

A. L.

Christensen

Mackinnon

Griffiths

K. M.

O’Kearney

(2009). The YouthMood Project: A cluster randomized controlled trial of an online cognitive behavioral program with adolescents. Journal of Consulting and Clinical Psychology, 77, 1021-1032.

12.

Cardaciotto

Herbert

J. D.

Forman

E. M.

Moitra

Farrow

(2008). The assessment of present-moment awareness and acceptance: The Philadelphia Mindfulness Scale. Assessment, 15, 204-223.

13.

Center for Collegiate Mental Health. (2012). CCAPS 2012 technical manual. CCMH. University Park, PA.

14.

Chase

J. A.

Houmanfar

Hayes

S. C.

Ward

T. A.

Vilardaga

J. P.

Follette

(2013). Values are not just goals: Online ACT-based values training adds to goal setting in improving undergraduate college student performance. Journal of Contextual Behavioral Science, 2, 79-84.

15.

Enders

C. K.

(2012). Dealing with missing data in developmental research. Child Development Perspectives, 7, 27-31.

16.

Gallagher

(2014). National survey of counseling centers 2013. Section one: 4-Year directors. Pittsburgh, PA: University of Pittsburgh.

17.

Gillanders

D. T.

Bolderston

Bond

F. W.

Dempster

Flaxman

P. E.

Campbell

. . . Remington

(2014). The development and initial validation of the Cognitive Fusion Questionnaire. Behavior Therapy, 45, 83-101.

18.

Hayes

S. C.

Pistorello

Levin

M. E.

(2012). Acceptance and commitment therapy as a unified model of behavior change. The Counseling Psychologist, 40, 976-1002.

19.

Hayes

S. C.

Strosahl

Wilson

K. G.

(2012). Acceptance and commitment therapy: The process and practice of mindful change (2nd ed.). New York, NY: Guilford Press.

20.

Hooper

Larsson

(2015). The research journey of acceptance and commitment therapy (ACT). London, England: Palgrave Macmillan.

21.

Hunt

Eisenberg

(2010). Mental health problems and help-seeking behavior among college students. Journal of Adolescent Health, 46, 3-10.

22.

Keyes

C. L. M.

(2005). Mental illness and/or mental health? Investigating axioms of the complete state model of health. Journal of Consulting and Clinical Psychology, 73, 539-548.

23.

Lannin

D. G.

Vogel

D. L.

Brenner

R. E.

Abraham

W. T.

Heath

P. J.

(2016). Does self-stigma reduce the probability of seeking mental health information? Journal of Counseling Psychology, 63, 351-358.

24.

Lappalainen

Granlund

Siltanen

Ahonen

Vitikainen

Tolvanen

. . . Lappalainen

(2014). ACT Internet-based vs face-to-face? A randomized controlled trial of two ways to deliver acceptance and commitment therapy for depressive symptoms: An 18-month follow-up. Behaviour Research and Therapy, 61, 43-54.

25.

Lee

E. B.

Levin

M. E.

Twohig

M. P.

(2015). An initial meta-analysis of acceptance and commitment therapy for treating substance use disorders. Drug and Alcohol Dependence, 155, 1-7.

26.

Levin

M. E.

Pistorello

Hayes

S. C.

Seeley

(2014). Feasibility of a prototype web-based acceptance and commitment therapy prevention program for college students. Journal of American College Health, 62, 20-30.

27.

Levin

M. E.

Pistorello

Hayes

S. C.

Seeley

J. R.

Levin

(2015). Feasibility of an acceptance and commitment therapy adjunctive web-based program for counseling centers. Journal of Counseling Psychology, 62, 529-536.

28.

Muthén

L. K.

Muthén

B. O.

(2012). Mplus user’s guide (7th ed.). Los Angeles, CA: Author.

29.

Richards

Richardson

(2012). Computer-based psychological treatments for depression: A systematic review and meta-analysis. Clinical Psychology Review, 32, 329-342.

30.

Smout

M. F.

Davies

Burns

Christie

(2014). Development of the Valuing Questionnaire. Journal of Contextual Behavioral Science, 3, 164-172.

31.

Trompetter

H. R.

Bohlmeijer

E. T.

Veehof

M. M.

Schreurs

K. M. G.

(2015). Internet-based guided self-help intervention for chronic pain based on acceptance and commitment therapy: A randomized controlled trial. Journal of Behavioral Medicine, 38, 66-80.

32.

Tullis

Albert

(2008). Measuring the user experience. San Francisco, CA: Morgan Kaufmann.