Improving the Reach of Clinical Practice Guidelines: An Experimental Investigation of Message Framing on User Engagement

Abstract

Despite strong evidence for the efficacy of treatments for posttraumatic stress disorder (PTSD), most affected individuals are not receiving these treatments, in part because they may not know that evidence-based treatments exist. The American Psychological Association published a website to disseminate information about its Clinical Practice Guideline for treating PTSD. In Study 1, Google Optimize was used in a field study to examine whether altering the subheadings to three of the website pages would increase site visitor engagement. On the main page and on the page with treatment descriptions, no subheading alterations improved engagement. On the Patients and Families page, the subheading “say goodbye to symptoms” improved engagement on three outcome variables, including clicking a link to find a psychologist (although there were a small number of clicks). In a preregistered conceptual replication in a sample not actively seeking information about the PTSD guideline (N = 578), the results were not replicable. Results highlight challenges of disseminating information about evidence-based treatment.

Keywords

posttraumatic stress disorder PTSD dissemination clinical practice guideline user engagement replication preregistered

In August 2017, the American Psychological Association (APA) posted a page on its website to disseminate information about its new Clinical Practice Guideline for the Treatment of Posttraumatic Stress Disorder (PTSD; see http://www.apa.org/ptsd-guideline/). The guideline was based on a systematic review of the best available research on treatment efficacy, which led to strong recommendations for variations of cognitive behavior therapy (CBT), such as prolonged exposure therapy. The guideline not only takes into account the state of scientific evidence (e.g., from randomized controlled trials) but also puts the results in context of larger health care issues, such as patient values and preferences and potential benefits and risks of treatment. Although the guidelines were written with many audiences in mind, including clinicians and policymakers, one purpose is also to provide treatment information to patients and their families to allow them to make informed decisions (APA Advisory Steering Committee for Development of Clinical Practice Guidelines, 2017).

There are two substantial gaps in delivering evidence-based psychotherapy, both of which can be at least partially ameliorated by consumer education. First, in part because therapists often do not deliver evidence-based treatments, there is a research–practice gap (see Teachman et al., 2012). For example, in one investigation of the rate at which CBT (one of the guideline’s “strongly recommended” treatments) was offered to patients at an outpatient psychiatry hospital in Los Angeles, results revealed that only 15.8% of individuals with PTSD were offered CBT (with or without exposure) and that merely 1.7% were offered exposure elements specifically (Wolitzky-Taylor, Zimmermann, Arch, De Guzman, & Lagomasino, 2015). To address this gap, there is the potential for consumers to create a market “pull” for these treatments if they become more aware of evidence-based psychotherapies (Gallo, Comer, & Barlow, 2013). Research suggests that nonpsychologists have significant knowledge gaps about evidence-based treatments (Harik, Matteo, Hermann, & Hamblen, 2017; Schofield, Weis, Ponzini, & McHugh, 2017). Moreover, in a study of veterans who had completed at least part of an evidence-based treatment for PTSD, most of them reported learning about these treatments for the first time from their therapists (Hundt et al., 2015).

Taken together, these data suggest that public knowledge is not yet sufficient to create a market pull for evidence-based treatments. Although it may seem far-fetched to think that future consumers of psychotherapy may start demanding specific treatments from their therapists, consider other examples of treatment demands. An obvious comparison is with the world of pharmacological intervention. In 1997, the Food and Drug Administration in the United States changed its guidelines to allow pharmaceutical companies to advertise prescription drugs using direct-to-consumer advertising. Since this shift, there has been a well-documented increase in individuals seeking out more information about prescription drugs that they see advertised (Gellad & Lyles, 2007; Ventola, 2011; Wilkes, Bell, & Kravitz, 2000). More recently, shunts are being requested more frequently among individuals with dementia, given the increase in news stories about the success of shunts for normal-pressure hydrocephalus (see Favaro, St. Philip, & Slaughter, 2019; Levinson-King, 2019). These examples suggest that increased public awareness of specific interventions can lead individuals to seek out these treatments.

Second, the treatment gap is the difference between those who can benefit from psychotherapy and those who receive it (Kohn, Saxena, Levav, & Saraceno, 2004). Despite the extensive evidence for PTSD treatment efficacy, the majority of suffering individuals are not receiving these treatments. For example, in one investigation of beliefs about PTSD treatments among military men and women, 40% of participants who met diagnostic criteria for PTSD (but who had not sought treatment) were concerned about the anticipated treatment experience (e.g., “I don’t want medications” and “doctors can’t relate to me”), making it the most commonly cited barrier to seeking treatment (Stecker, Shiner, Watts, Jones, & Conner, 2013). Note that 26% of veterans in this study reported not wanting to take a medication for PTSD, which suggests that they believe that they would need to take medication to find relief from their symptoms. In fact, medications are not strongly recommended as a first-line intervention in the current APA guideline, so educating individuals about recommended psychotherapies could potentially increase interest in treatment. In addition, 13% reported not wanting group therapy as a thought that prevented them from seeking treatment. These individuals may be more likely to seek treatment if they learned that evidence-based (cognitive processing) individual therapy is more effective than group formats of the same treatment (Resick et al., 2017). In an online study of individuals who had a positive screening for PTSD, only 45% of participants reported knowing that exposure therapy is an evidence-based treatment, and it was less frequently selected than treatments that are not strongly recommended as evidence-based, such as therapy dogs and support groups (Harik et al., 2017). Disseminating information about treatments that work most effectively for PTSD (on average) may help address these concerns and thereby decrease the treatment gap.

Dissemination efforts for promoting evidence-based psychological treatments have largely been aimed at convincing mental health-care providers to adopt evidence-based therapies (McHugh & Barlow, 2010). However, pharmaceutical companies promote treatments to both providers and consumers, which greatly expands the reach of their marketing. Although direct-to-consumer advertising for evidence-based psychotherapies is clearly not the only answer to address the noted gaps in mental health treatment, it is important for the field to consider this strategy more seriously. Psychologists are actually uniquely qualified to study how messages and advertisements can be effectively worded for health-behavior promotion. There is much room for research in this area; in a recent study of U.S. adults in an online sample, only 20% could accurately define evidence-based mental health care (Mora Ringle et al., 2019). In a sample of parents concerned about their teens’ substance use, almost 70% reported wanting to hear about psychological treatments via the web (Becker, Helseth, Frank, Escobar, & Weeks, 2018), and most of these same parents also reported valuing evidence-based-practice principles (Becker, Weeks, et al., 2018). However, very little research has examined how to promote potential consumers’ engagement with information about evidence-based psychological treatments or how to best define it in terms that the public will appreciate. Important early work in this area suggests that how CBT is described (i.e., descriptions of mechanisms of change for CBT) does not alter the perceived credibility of the treatment (Schofield et al., 2017), so it remains critical to determine what factors do influence responses to information about evidence-based psychotherapies.

In the first of two studies, we examined how different subheadings on three APA Clinical Practice Guideline website pages might promote greater engagement on the site. Our goal was to examine real users’ behaviors to better understand how to disseminate information about evidence-based treatments for PTSD via the Internet. Thus, an experimental approach was used but in the context of a field study. Subheadings for the main landing page, the Treatments page (describing the empirical support for and content of each of the recommended treatments), and the For Patients & Families page (describing PTSD and how to find treatment) were created to experimentally test whether different subheadings affect user behavior in the real world. The second study was a conceptual replication and extension of the first study using a different website context, sample, and outcome measures.

We drew from five theoretical perspectives to create numerous subheadings for each of the pages. In the social psychology field, research has found that small wording differences in persuasive messages can lead to big differences in behaviors (e.g., indicating that previous hotel room guests reuse towels vs. a generic environmental message improved rates of reusing towels; Goldstein, Cialdini, & Griskevicius, 2008). Thus, the goal was to assess whether various subheadings would alter different indicators of engagement with page content: clicks to links on each page (suggestive of wanting to learn more about the information on each page), time spent on the page (suggestive of reading more information presented), and bounce rate (when a site visitor lands on a page and then closes it without clicking on site content, suggestive of not engaging with content). There were no a priori hypotheses about the relative success of each subheading, although it was hypothesized that the theoretically derived subheadings would result in significantly greater user engagement across all three pages compared with the original page versions without the subheadings. The results of this research can have direct implications for how to word messages about evidence-based treatments for psychological disorders. This could have considerable, scalable public-health impact given that small wording changes on a website (a free modification to existing sites) have the potential to change behavior on a large scale.

Study 1 Method

Participants and recruitment

On August 2, 2017, the APA posted a press release on the website (American Psychological Association, 2017) announcing the launch of a comprehensive online resource on treatment for PTSD. Following launch, APA staff did outreach to various groups to promote the site, and news organizations also picked up the story. Given that these were field experiments completed on a live site, participants are any individuals who viewed the website, so we do not have demographic information about the visitors. To give a crude estimate of sample size, the main landing page was viewed more than 20,000 times within the experimental window (note, however, that individuals could visit the site multiple times). The purpose of the site was to disseminate evidence-based treatment information to practitioners as well as consumers, and many resources on the site were written for patients and families. Thus, it seems likely that the majority of visitors to the site were potential treatment consumers, researchers, and clinicians.

Experimental design

Three experiments were conducted concurrently on the APA Clinical Practice Guideline website using Google Analytics Optimize. The pages with active experiments were the main landing page (Study 1a; http://www.apa.org/ptsd-guideline/index.aspx), the Treatments page (Study 1b; http://www.apa.org/ptsd-guideline/treatments/index.aspx), and the For Patients & Families page (Study 1c; http://www.apa.org/ptsd-guideline/patients-and-families/index.aspx). These were chosen because it was likely that individuals who were interested in seeking information about PTSD treatments for themselves or a loved one would arrive on the landing page and navigate to a subsequent page, would land directly on the page that describes different treatments (and their empirical support), or would land directly on the page written for consumers. Note, however, that the page written for consumers was the primary page of interest, given that it was the most direct test of our primary research question. On each page, eight website variants were tested against each other; Google’s platform randomly assigned visitors to one of eight versions of the page (one variant was always the version of the page without a theoretically derived subheading; the remaining seven variants had a different subheading).

Google Analytics Optimize requires one main objective (outcome) per experiment that is monitored and used for deciding when to conclude the experiment. The main objective for Study 1a was the number of clicks on the six images that linked to other website content pages of the guideline. This was chosen because clicking on boxes below the subheading would suggest that the user was interested in navigating to another page to learn more. The main objective for Study 1b was whether the participant clicked any of the links below the introductory text on the page (which showed more information about each supported treatment). This was chosen because it was predicted that more engaging or interesting subheadings would encourage individuals to learn more about the recommended treatments by clicking on page links. The main objective for Study 1c was the number of clicks of the “Find a Psychologist” button on the page. This was a particularly stringent test of the efficacy of the subheadings because it would directly reflect interest in seeking treatment after viewing information on the website.

Additional metrics that were assessed but not used as the deciding factor for the experiments included bounces (single-page session on a website in which a user does not engage in any other way on the page; Google, 2017a) and session duration (the amount of time spent on the site during a given session, which is a set of user actions within a given time frame; Google, 2017b). Note that bounces and clicks are highly negatively related in that if an individual clicks on a link on a page, someone has not bounced off of the page (exiting without clicking anywhere). These additional metrics allowed us to examine user behavior on the site to have a better understanding of engagement with material.

Materials

Subheading creation

Subheadings were created by drawing from five perspectives hypothesized to encourage site visitors to engage with the site content (for all subheadings, see Table 1). Subheadings were intended to leverage motivations or contradict attitudinal barriers; this approach was chosen because there is so little empirical evidence for what messages encourage the seeking of evidence-based mental health treatment, especially on this platform.

Table 1.

Subheadings, Frameworks, and Total Number of Sessions for Study 1

Study and subheading	Framework	Sessions
Study 1a
Original version, no subheading		2,307
“Find a treatment supported by the best available research”	Trust in scientific experts	2,299
“Find a treatment supported by science”	Trust in scientific experts	2,557
“Find a treatment endorsed by the APA”	Organizational support	2,457
“Find a treatment to help you take back your life”	Gain framework	2,812
“Find a treatment to help you say goodbye to symptoms”	Gain framework	2,419
“Join others with PTSD who have found relief in treatment”	Describing norms	2,386
“You don’t have to go through this alone; find a treatment that works”	Attitudinal barrier: wanting to handle problems on one’s own	2,494
Study 1b
Original version, no subheading		795
“Research shows strong support for four interventions for posttraumatic stress disorder, and conditional support for another four”	Trust in scientific experts	811
“Scientists strongly support four interventions for posttraumatic stress disorder, and conditionally support another four”	Trust in scientific experts	775
“Patients can take back their lives”	Gain framework	970
“Patients can say goodbye to symptoms”	Gain framework	784
“Patients don’t have to suffer with posttraumatic stress disorder”	Gain framework	703
“Patients are getting better. Find treatments that work to help you”	Describing norms	677
“Those with PTSD don’t have to suffer alone”	Attitudinal barrier: wanting to handle problems on one’s own	740
Study 1c
Original version, no subheading		618
“Treatment works: Feel like yourself again”	Gain framework	342
“Treatment works: Say goodbye to symptoms”	Gain framework	195
“Science says, treatment works”	Trust in scientific experts	259
“Research shows, treatment works”	Trust in scientific experts	312
“The APA supports these recommended treatments”	Organizational support	505
“Others are getting better, you can too”	Describing norms	288
“Treatment works: No one has to go through PTSD on their own”	Attitudinal barrier: wanting to handle problems on one’s own	324

Note: APA = American Psychological Association; PTSD = posttraumatic stress disorder.

Extensive research has focused on gain-framed messages (highlighting the positive consequences of performing or not performing a behavior) for motivating individuals to perform behaviors that are intended to prevent cancer (Rothman & Salovey, 1997). In the current study, we created subheadings that highlighted the positive consequences of seeking treatment (i.e., getting better).

Previous research has also found that describing norms (or what others are already doing) can change behaviors (Cialdini, Kallgren, & Reno, 1991). In this context, describing norms may be particularly influential for individuals who experience self-stigma (internalized stigma tied to an aspect of identity; Corrigan, 2004) regarding their mental health.

Less well studied is how framing messages around scientific support or organizational support changes behaviors. Critically, American adults generally believe that scientific research is beneficial and support public investment in research (American Academy of Arts & Sciences, 2018), which suggests that subheadings explicitly stating that the Clinical Practice Guideline is based on scientific research may be motivating for individuals seeking treatment information. Moreover, the website is hosted by the APA, so noting APA’s explicit support for guideline-recommended treatments in subheadings may influence treatment-seeking behavior.

We also created subheadings that explicitly countered a frequently cited attitudinal barrier to seeking mental health treatment: wanting to handle the problem on one’s own (Mojtabai et al., 2011). Specifically, among individuals who do not access mental health treatment but who recognize a need for treatment, the most common reason for not accessing treatment was wanting to handle the problem on their own (Andrade et al., 2014). Thus, it is possible that a subheading that counters that attitude may increase treatment-seeking behavior.

Website content

The website was written by APA staff, overseen by author L. Bufka. Content was organized around each of the recommended treatments, and descriptions of each treatment were derived from easily accessible published professional literature. Case examples and other resources were typically written by individuals closely associated with the development and evaluation of the treatments. Information for patients and families was derived from the guideline but written in a format that would be easily understood by members of the public. Material was reviewed by members of the guideline-development panel and the oversight committee for guideline development.

Procedure

Study 1a ran from August 18, 2017, to November 16, 2017. Studies 1b and 1c ran from August 23, 2017, to November 21, 2017. Google Analytics Optimize determined how long the experiments ran. Site visitors were randomly assigned to see one of the eight versions of the pages throughout the study’s time frame. This study was exempt from the A. Werntz’s and B. A. Teachman’s institutional review boards because the data were collected and analyzed by Google Analytics (as described below, no data were seen by the authors), so no informed consent was given to site visitors. A. Werntz and B. A. Teachman served as consultants to APA for the subheading creation and interpretation of results.

Study 1 Results

Google Analytics and Google Optimize were used to monitor the experiments as they ran and were used for all data analyses. The experiments used a multiarmed-bandit approach, estimated using Bayesian methods, to model and analyze the data (Scott, 2010, 2017). Twice per day, Google Analytics studied which arm subheading variant had the highest conversion rates (i.e., highest rate of visitors clicking on a target link) and then allocated users to the variants differentially on the basis of success the previous day. The more successful the variant, the more users were allocated to that variant.

“Winners” of experiments are determined when the “probability to beat baseline” reaches 95% (Google, 2018). Experiments on Google Optimize run for a minimum of 2 weeks and for a maximum of 3 months (Scott, 2017). Critically, Google Analytics will not declare a winner if there is a tie among variants. Two or more variants may perform similarly, so running the experiment until a single winner is chosen is not ideal given that finding a single winner may take a very large number of participants to identify. This multiarmed-bandit approach allows researchers to run an experiment until the optimal variants or variants are found (Scott, 2017). Note that Google provides only results as to how the variants performed against baseline; no post hoc comparisons of variants can be conducted with the information given. Google does provide credible intervals, which are used in Bayesian statistics to provide a range of values for which, given the measured data, “there is a 95% probability that the true value of [your parameter] lies within the credible region” (VanderPlas, 2014, p. 4).

Study 1a (landing page) results

Study 1a consisted of 19,731 sessions across seven variants and the original version of the page; the boxes (links) below the headline were clicked a total of 6,290 times. For all variants’ number of sessions, see Table 1. Critically, no single variant was found to be statistically significantly better performing than the rest in terms of the main objective for Study 1a using a 95% credible interval around modeled conversion rates. Although the full model parameters are not available from Google, modeled conversion rates consider “account time, user context, result consistency, and other factors” (Google, 2018). The models used are proprietary, and Google does not disclose all of the information it uses to measure conversion rates in experiments. Bounce rate had similar results; there was no statistically significant difference among variants. However, the subheading “Find a treatment to help you say goodbye to symptoms” had longer session duration compared with the original version of the page, with a 98% probability to beat baseline (for this subheading, 95% credible interval = [3:55 s, 4:47 s]; for the original, 95% credible interval = [3:22 s, 4:07 s]).

Study 1b (Treatments page) results

Study 1b consisted of 6,255 sessions, with 1,663 clicks to the links on the page (primary objective). None of the variants on the Treatments page were statistically significantly predicted to outperform the original. However, there was a trend for the subheading “Those with PTSD don’t have to suffer alone” to lead to a higher conversion rate than the original. This variant was rated as having a 94% probability of beating the original, which is conceptually similar to a p value of .06. The variant “Patients can take back their lives” was statistically significantly the worst in terms of conversion rate. This was predicted to outperform the original just 1% of the time. Bounce rate and session duration did not significantly vary depending on the variant.

Study 1c (For Patients & Families page) results

Study 1c consisted of 2,843 total sessions and only 34 clicks on the “Find a Psychologist” button on the For Patients & Families page (variants range = 2–6 clicks). Although one clear winner was not declared by Google Analytics for this study, two variants emerged as being more likely to lead to clicks on the “Find a Psychologist” button (primary objective) than the original. Both variants “Treatment works: Feel like yourself again” and “Treatment works: Say goodbye to symptoms” were statistically significantly predicted to outperform the original. A gain-framework subheading (“Feel like yourself again”) had a 96% chance of outperforming the original (similar to p = .04), and another gain-framework subheading (“Say goodbye to symptoms”) had a 97% chance of outperforming the original (similar to p = .03). The statistical model predicted that the “goodbye to symptoms” variant had a 42% chance of being the best variant of the eight options; for modeled conversion rates for all measured outcomes, see Figure 1. This variant also had the lowest bounce rate, with a 100% chance of outperforming the original version of the page. The variants “Science says, treatment works” and “Others are getting better, you can too” were also predicted to outperform the original 100% of the time. Of all of the variants, the “Say goodbye to symptoms” variant had a 60% chance to have the lowest bounce rate. The three variants that performed well in terms of bounce rate also performed well in terms of session duration. The gain-framework variant (“Say goodbye to symptoms”) was predicted to be the best variant at 45% (which was the highest predicted rate) and had a 99% chance of outperforming the original. Likewise, the “Science says . . . ” and “Others are getting better . . . ” variants also had a 99% chance of outperforming the original.

Fig. 1.

Results from Study 1c. The image at the top is the “For Patients & Families” page with one of the subheadings (highlighted for emphasis) used in the study. The graphs at bottom show modeled conversion rates (with 95% confidence intervals) of subheading variants compared with the original (i.e., no subheading) on the “For Patients & Families” page (Study 1). The image is from the American Psychological Association.

Interim Discussion

Across three field studies, changes to subheadings on the main landing page and treatment description page did not result in changes to users’ behavior; however, changes to subheadings on a page written for potential consumers of evidence-based PTSD treatments led to change in user behavior. When website users saw “Treatment works: Say goodbye to symptoms” on the For Patients & Families page of the APA clinical practice guideline website, they clicked more frequently on the “Find a Psychologist” link (although the total number of clicks on this link was small), stayed on the page longer, and left the page without clicking on content the fewest times. All three indicators suggest that site visitors who saw this headline were more engaged and interested in learning about evidence-based treatments for PTSD. These results could also suggest specificity of effects depending on the targeted audience; the headlines were written to target potential treatment consumers and their loved ones, and this page was the only one that showed differences according to changes in subheadings. Although speculative, this may also suggest that potential treatment consumers may have focused their attention to the page written expressly for that audience.

These results leave many unanswered questions, not the least of which is whether the results of this field study—using a novel (for our field) design—can be replicated. Although we attempted to conduct a direct replication on the live site, site traffic was too low to adequately power an experiment. So instead, we ran a conceptual replication to examine whether the effect of subheading on website user engagement could be replicated in a different sample and setting. In the second study, subheadings were randomly assigned to research volunteers in an online laboratory setting. Replicated results would suggest a very robust effect of “Treatment works: Say goodbye to symptoms” as an useful yet simple framework to promote evidence-based treatments in diverse samples.

Study 2

We opted to run a conceptual replication of the field study using an online research platform. Although the field study allowed for strong external validity of the potential effects of subheadings on website behavior, we were unable to characterize the sample, control the number of times individuals saw the content, or directly evaluate different aspects of participants’ reactions to the website material. In the replication, the experience mimicked what site visitors would experience on the “For Patients & Families” page in a relatively more controlled online setting.

Method

Participants

Participants for Study 2 were recruited from Project Implicit (https://implicit.harvard.edu/implicit/), which is a research website with a registered participant base. Volunteers participate in rotating studies that examine implicit attitudes. Participants qualified for this study if they were (a) at least 18 years old and (b) U.S. citizens or residents (to reduce the influence of cross-national differences tied to health care systems and treatment access in this initial investigation). We elected not to screen on the basis of mental health symptoms or treatment need given that messages directed to patients and their families would be relevant to a broad range of the general public and that this sampling approach also increased feasibility of recruiting the large sample required.

The study ran from August 16 to 24, 2019; 844 individuals were assigned to this study from the rotating pool of available studies. Of individuals assigned, 771 consented to the study. Following informed consent, participants were randomly assigned to one of five headline conditions (see Materials, below) to mirror the live APA PTSD web-page results; 578 individuals completed ratings for the subheadlines. Of those 578 individuals, 453 completed the study and saw the debriefing form. For the analyses presented here, the 578 individuals who completed the ratings were included in analyses, although note that some individuals declined to answer some of the ratings (“decline to answer” is always an option), and some individuals did not complete the questionnaires following the ratings, so ns do not always equal 578. Participants were 18 to 86 years old (M = 38.53, SD = 14.74); 429 identified as women, and 149 identified as men (the web platform did not include options to reflect gender diversity, hence the results are reported for men and women only). The sample consisted of 62 (10.7%) Black or African American, 414 (71.6%) White or European American, 22 (3.8%) Asian, 35 (9.8%) of more than one race, and 39 (6.7%) other or unknown. Sixty-four (11.1%) reported their ethnicity as Hispanic, 466 (80.6%) reported not Hispanic, and 48 (8.3%) either reported unknown or did not report. This was a well-educated sample: 219 (37.9) reported having an advanced degree, 198 (34.3%) had at least a bachelor’s degree, 126 (21.8%) had at least some college or an associate’s degree, 26 (4.5%) had at least a high school degree, and 6 (1.0%) had less than a high school degree.

Twenty-six (4.5%) reported that they were currently struggling with PTSD; 29 (5.0%) reported struggling with PTSD in the past. Although the majority of individuals (257, 44.5%) reported not knowing anyone with PTSD, 127 (22.0%) reported knowing someone with the disorder; the remaining individuals reported not knowing someone with PTSD, were unsure of whether they knew someone, or did not answer the question.

Materials

Demographic variables

When registering for Project Implicit, participants are asked to report their demographic information.

Subheading rating task

In this task, participants were instructed,

In this task, we are interested in your opinion on a real webpage. [On the next page] you will see a webpage from the American Psychological Association. Take as little or as much time as you would like examining the page and its content. When you are finished . . . you will be prompted with questions about your opinion of the webpage. There are no right or wrong answers. We are simply interested in your thoughts about the page and its contents.

Participants then saw the “For Patients & Families” page of the guideline with one of five randomly assigned subheadings: (a) no subheading; (b) “Treatment works: Say goodbye to symptoms”; (c) “Treatment works: Feel like yourself again”; (d) “Others are getting better, you can too”; and (e) “Studies show treatment works.” Subheadings a and b were selected because these were the two subheadings that increased clicking on the “Find a Psychologist” button statistically significantly over no subheadline on the live web page in Study 1. Subheadings d and e were selected on the basis of focus groups we held with 11 adults with anxiety to identify effective language for increasing interest in evidence-based treatments among potential consumers. Time spent on the page was measured.

On the following page, participants were presented with questions about the version of the “For Patients & Families” page that they saw. Questions were presented one at a time in a random order: “How engaging was the content of the webpage you just saw?” (1 = not at all engaging, 5 = extremely engaging); “How helpful was the content of the webpage you just saw?” (1 = not at all helpful, 5 = extremely helpful); “How trustworthy was the content of the webpage you just saw?” (1 = not at all trustworthy, 5 = extremely trustworthy); “If you were struggling with posttraumatic stress disorder (PTSD), how likely would you be to click the ‘Find a Psychologist’ button on the righthand side of the page?” (1 = not at all likely, 5 = extremely likely, or I did not see that button); “If a family member or friend were struggling with PTSD, how likely would you be to recommend the webpage you just saw?” (1 = not at all likely, 5 = extremely likely). The final question was always this: “If you would like to receive an email from the study investigator with more information about recommended treatments for PTSD and a link to the live webpage, please enter your email address below.” Because of the format, participants could not click on the “Find a Psychologist” button during the study, so there was no direct parallel in the replication study. Asking for e-mail addresses served as a proxy for a behavior that would be seen as high investment or engagement (submitting personally identifiable information for more information on the treatments).

Mental health history and proximity to PTSD

Participants were asked to indicate whether they are currently struggling or have previously struggled with PTSD. They were also asked to indicate whether they know someone with PTSD (a family member, a close friend, someone else close, or all of the above).

Procedure

Following informed consent, participants were randomly assigned to one of the five subheading conditions and were then asked to rate their version of the “For Patients & Families” page. Participants then completed the mental health history items. Finally, participants were given the opportunity to read more about the research in the debriefing form. (Additional measures were collected during the study that are not central to the current questions, including a brief Implicit Association Test that assessed self and traumatized implicit association strength; Sriram & Greenwald, 2009.)

Results

Hypotheses were preregistered on August 26, 2019, before data analyses (see https://osf.io/rha53). As outlined in the preregistration, six one-way analyses of variance were conducted to examine whether the five subheading variants differed in the six ratings. Post hoc multiple comparisons using a Bonferroni correction were planned for significant omnibus tests. A χ² test was used to examine whether seeing a specific subheading was more or less likely to encourage a participant to enter an e-mail address for more information. To correct for multiple comparisons, α was set to .007 (.05 divided by seven tests). All ratings (how engaging, helpful, and trustworthy; whether someone would click “Find a Psychologist”; and likelihood of referring a friend) were made on a Likert-type scale and were normally distributed. Four extreme outliers on the time spent on the “For Patients & Families” page were removed (range = 34.40 min–26.17 hr); the rest of the participants viewed the page for 1.21 s to 9.87 min. However, the viewing-time variable was positively skewed, so a log transformation was used to normalize this variable.

Between 129 and 149 individuals were randomly assigned to see each subheading variant; a χ² analysis revealed that participants did not leave the study during this task at different rates depending on the subheading, χ²(4) = 1.53, Cramér’s V = .05, p = .822 (17 to 25 individuals dropped out per subheading). Across all dependent variables, no differences across subheadings emerged (in contrast to results for the field study on the APA website). A one-way analysis of variance that tested whether individuals spent varying amounts of time on the site by subheading revealed no significant differences, F(4, 569) = 1.66, η_p² = .01, p = .157. Subheadings did not receive reliably different ratings for engagement, F(4, 562) = 0.73, η_p² = .01, p = .550; helpfulness, F(4, 560) = .65, η_p² = .01, p = .626; or trustworthiness, F(4, 547) = 1.30, η_p² = .01, p = .269. Furthermore, individuals did not vary in their self-reported likelihood of clicking the “Find a Psychologist” button, F(4, 505) = 0.51, η_p² = .00, p = .726, or referring friends or family to the site, F(4, 561) = 0.58, η_p² = .00, p = .678. A χ² test revealed that participants were not more or less likely to enter an e-mail address for more information depending on subheading, χ²(4) = 1.83, Cramér’s V = .06, p = .766 (26 gave an e-mail address).

Discussion

Taken together, the results of the field study and conceptual replication were mixed. On the one hand, promising results from the field study suggest that minimal changes to text on a website can potentially affect visitors’ interaction with a web page. In particular, visitors to the APA Clinical Practice Guideline for the treatment of PTSD “For Patients & Families” web page did vary their behavior on the basis of subheading. Although low in number across all subheadings, site visitors did statistically more frequently click on the “Find a Psychologist” button and were more likely to engage with site content when presented with the subheading “Treatment works: Say goodbye to symptoms” relative to no subheading. This suggests that a no-cost “intervention” (changing text on a web page) may help engage individuals in seeking more information about evidence-based treatments and perhaps even influence treatment-seeking behaviors. However, the field-study results could not be replicated in a controlled experiment with the Project Implicit sample.

Focusing on direct-to-consumer marketing

The current studies do not provide a clear answer for how to frame evidence-based treatments for PTSD to potential consumers, and so it remains critical to determine the dissemination strategies (including message framing) that will reliably influence engagement and motivation to seek treatment for different subpopulations given the treatment–practice gaps. In their discussion of direct-to-consumer marketing of evidence-based anxiety treatments, Gallo et al. (2013) argued that providers underuse evidence-based treatments and that consumers do not have knowledge about the effectiveness of these treatments. Recent research shows that there is a lack of knowledge regarding evidence-based psychotherapies among U.S. adults (Becker, Weeks, et al., 2018; Mora Ringle et al., 2019). Gallo and colleagues noted that with increased knowledge about treatment, consumers could create a market pull for treatments that work, ultimately pushing providers to provide these treatments. With increased exposure to information about the treatments on the Clinical Practice Guideline site, individuals may feel more empowered to request the treatments that have the strongest research support, although this remains to be tested. In an online study of adults who screened positively for PTSD, participants were asked how much control over PTSD treatment decisions they would prefer: 23.6% reported wanting to make the decision themselves, and 44.2% reported wanting to make the final decision after taking the doctor’s opinion into account (Harik, Hundt, Bernardy, Norman, & Hamblen, 2016). If more than 60% of individuals with PTSD report wanting to ultimately choose their treatment, it seems that treatment providers should want to help them at least make informed treatment decisions. Moreover, in a study of veterans who had elected to complete an evidence-based treatment for PTSD, veterans reported that they most frequently heard about these treatments from therapists, psychiatrists, and other veterans (Hundt et al., 2015)—suggesting there is room for improvement in promoting evidence-based treatments in more easily accessible formats, such as through websites, social media, and advertising.

Today, direct-to-consumer marketing for pharmaceuticals is commonplace, and it causes some individuals to have conversations about drugs with their physicians and request specific drugs (Aikin, Swasy, & Braman, 2004). Despite valid criticisms associated with this trend, direct-to-consumer marketing of pharmaceuticals does to a certain extent inform patients about treatment options and increase medication compliance, and it may reduce stigma associated with having the disorders (Ventola, 2011). However, psychologists face a practical barrier associated with promoting psychological treatments; unlike pharmaceuticals, the majority of evidence-based therapies are not proprietary and do not have the financial backing of large corporations. Although this is certainly a challenge for future work, steps in creating motivating messages about psychotherapy may be an important step.

We hope that researchers will continue to examine the influence of framing on treatment messages using experimental approaches. In the current study, the five approaches used to create subheadings were not chosen from one particular field but instead were selected to allow for a wide approach to testing various frameworks. Future research that examines more nuanced messaging would likely benefit from a more systematic approach to creating subheadings. However, current results suggest positive results for the “Say goodbye to symptoms” condition in one of the field studies, which may follow from its emphasis on the loss of a negative symptom. In behavioral-economics research, this is considered a gain (or positive) frame because it highlights the potential positive outcomes of engaging in treatment. Thus, an individual with PTSD symptoms may be willing to seek treatment despite potential costs (e.g., the costs of increased feelings of vulnerability by talking about the trauma in treatment or loss of time or money) if clear positive outcomes are expected. Symptoms of PTSD are aversive, so the potential for reducing these symptoms is attractive. However, future work needs to investigate why this message specifically would be appealing to individuals suffering from PTSD. These results are the first to our knowledge that suggest that the gain-framework message conveying that individuals suffering from PTSD can say goodbye to symptoms after evidence-based treatment may promote engagement with treatments among consumers, but obviously, the results require replication and should be interpreted with caution, especially given the null findings in Study 2.

Failure of replication or meaningful differences across studies

The promising results of Study 1c (on the “For Patients & Families” page) were not replicated across web pages (Studies 1a and 1b) or in Study 2. One interpretation is that Study 1c results were spurious; another interpretation is that the results show specificity of the subheading intervention. Unfortunately, there is no clear way to tease apart these possibilities at this time. With respect to the lack of consistent results across web pages, it is notable that the subheadings were written with potential consumers of treatment (and their families) as the primary target (i.e., focused on addressing barriers to them seeking treatment) rather than clinicians, researchers, or other interested stakeholders as the main target. An implicit assumption was that potential consumers would view multiple pages and that potential consumers would alter their behavior across pages. However, if potential consumers were only a small proportion of site visitors on the landing and “Treatments” pages, there may not have been enough power to detect results on those pages. This may partly explain the different results across web pages in that the “For Patients & Families” page is likely the only web page that we can reasonably assume that the main visitors are potential consumers. Providing messaging about saying goodbye to symptoms or not having to suffer alone is unlikely to be as relevant to providers or other stakeholders as the messages would be to potential consumers. This may be especially relevant to explain the null results on the “Treatments” page (Study 1b), which explicitly states, “The information below about the recommended interventions is intended to provide clinicians with a basic understanding of the specific treatment approach.” Therefore, subheadings written for potential consumers (or their family members) may not have been relevant to that page’s target audience.

With respect to the lack of a conceptual replication in Study 2, it is noteworthy that these participants were adults who were registered volunteers for psychology research at Project Implicit. There were no restrictions for the sample except that the participants had to be U.S. citizens (to reduce the influence of cross-national differences in this initial set of studies). Thus, this was a difficult replication test in that the majority of the Study 2 sample was unlikely to have the same motivations (seeking treatment information) as the sample from Study 1c. It is plausible that participants’ motivations likely differentiate the two samples in very important ways, although this requires making inferences because motivation was not directly measured. It may be that simple changes to messages about evidence-based-treatment information matter more when the audience is motivated to learn about the treatments and thus perhaps pays more attention to details of the message framing. Clearly, however, given the results of these studies, there is not yet a consistent way of using subheadings to reliably change individuals’ behaviors while browsing information regarding evidence-based treatments for PTSD. However, the current work highlights a low-cost, efficient method for future research in this area.

Limitations

There were limitations to the field-study methodology that need to be considered. First, all of the variants are confounded with organizational support given that the information is listed on the APA website. So, although not explicitly listed in each subheading, organizational support of the treatment guideline is to an extent present in each condition. Second, it is notable that only 34 of 2,843 visitors clicked on the “Find a Psychologist” button. Although many site visitors may have been simply interested in APA’s first attempt at publishing clinical-practice guidelines, it is likely that more than 34 site visitors were suffering from PTSD. Thus, although our subheadings did statistically improve the original variant of the site, there is room for improvement on this outcome. (However, it may also be the case that some site visitors already have therapists, and so learning information about evidence-based treatments may promote a conversation with the current therapists. This would not be reflected by our measures.) In addition, subheadings varied in length and complexity, which potentially would confound the results.

Critically, Google Analytics gives limited information about the statistics behind the results. We are able to compare the variants only with baseline (which had no subheading), and we are unable to make post hoc comparisons across variants. This is because Google does not provide complete parameters for the Bayesian models and does not provide values that would allow for additional tests (e.g., values that would allow for comparisons of credible intervals). Moreover, these results may be specific to these subheadings, this disorder, or these particular treatments. We were also unable to examine moderators of the effect in the field study because information about the sample was not available.

The more controlled Study 2 involving Project Implicit also had limitations, including that most of the sample was probably not currently seeking information about PTSD treatments, which limits the study’s ecological validity. In addition, this replication study was not designed for conducting subgroup analyses, so we were unable to consider audience segmentation, that is, whether certain messages appeal more to one subgroup than another (e.g., on the basis of age, gender, race, symptom severity). This is an important direction for future research.

Conclusions

To our knowledge, this is the first study of its kind to use Google Optimize to answer a basic science question about dissemination of psychological treatments that is directly translatable to promotion of evidence-based care. Note that these results were not replicable in a more traditional, online laboratory setting. After the fact, it is easy to generate reasons why the positive results on the “For Patients & Families” page may not have been replicable, especially tied to the difference in target audiences and their motivations (i.e., whether participants were potential consumers seeking information about PTSD treatments), but the bottom line is that we do not yet know whether and when these effects can replicated.

To the extent that changes to subheadings can change website-user behavior tied to selecting evidence-based mental health care, websites can use a much more data-driven approach when creating messages about evidence-based treatments. Moreover, future work in this area can take advantage of recent advances in technology and predictive algorithms; Google Analytics is free and automatically applies advanced statistical modeling techniques to answer questions that can have real-world impact. If adding just a few words on a web page could mean that more individuals seek evidence-based treatments, this methodology holds promise to help identify highly scalable and cost-effective strategies to promote care.

Supplemental Material

Werntz_CPSOpenPracticesDisclosure – Supplemental material for Improving the Reach of Clinical Practice Guidelines: An Experimental Investigation of Message Framing on User Engagement

Supplemental material, Werntz_CPSOpenPracticesDisclosure for Improving the Reach of Clinical Practice Guidelines: An Experimental Investigation of Message Framing on User Engagement by Alexandra Werntz, Lynn Bufka, Brian E. Adams and Bethany A. Teachman in Clinical Psychological Science

Footnotes

Transparency

Action Editor: Scott O. Lilienfeld

Editor: Scott O. Lilienfeld

Author Contributions

A. Werntz, B. A. Teachman, and L. Bufka developed the study concept. The clinical practice guideline was developed by L. Bufka and a team at the American Psychological Association. All authors contributed to study design. Data collection for Study 1 was performed by B. E. Adams and for Study 2 was performed by A. Werntz. A. Werntz interpreted results under the supervision of B. A. Teachman. A. Werntz drafted the manuscript. B. A. Teachman and L. Bufka made critical revisions. All of the authors approved the final manuscript for submission.

ORCID iD

Alexandra Werntz

References

Aikin

K. J.

Swasy

J. L.

Braman

A. C.

(2004). Patient and physician attitudes and behaviors associated with DTC promotion of prescription drugs: Summary of FDA survey research results. U.S. Department of Health and Human Services, Food and Drug Administration, Center for Drug Evaluation and Research. Retrieved from https://www.fda.gov/downloads/AboutFDA/CentersOffices/OfficeofMedicalProductsandTobacco/CDER/UCM600276.pdf

American Academy of Arts & Sciences. (2018). Perceptions of science in America. Retrieved from https://www.amacad.org/publication/perceptions-science-america

American Psychological Association. (2017, August 1). APA launches comprehensive online resource on treatment for PTSD. Retrieved from http://www.apa.org/news/press/releases/2017/08/ptsd-treatment.aspx

Andrade

L. H.

Alonso

Mneimneh

Wells

J. E.

Al-Hamzawi

Borges

. . . Kessler

R. C.

(2014). Barriers to mental health treatment: Results from the WHO World Mental Health (WMH) surveys. Psychological Medicine, 44, 1303–1317. doi:10.1017/S0033291713001943

APA Advisory Steering Committee for Development of Clinical Practice Guidelines. (2017, July). Placing clinical practice guidelines in context. Retrieved from https://www.apa.org/about/offices/directorates/guidelines/context

Becker

S. J.

Helseth

S. A.

Frank

H. E.

Escobar

Weeks

(2018). Parent preferences and experiences with psychological treatment: Results from a direct-to-consumer survey using the marketing mix framework. Professional Psychology, Research and Practice, 49, 167–176. doi:10.1037/pro0000186

Becker

S. J.

Weeks

B. J.

Escobar

K. I.

Moreno

DeMarco

C. R.

Gresko

S. A.

(2018). Impressions of “evidence-based practice”: A direct-to-consumer survey of caregivers concerned about adolescent substance use. Evidence-Based Practice in Child and Adolescent Mental Health, 3, 70–80. doi:10.1080/23794925.2018.1429228

Cialdini

R. B.

Kallgren

C. A.

Reno

R. R.

(1991). A focus theory of normative conduct: A theoretical refinement and reevaluation of the role of norms in human behavior. In Zanna

M. P.

(Ed.), Advances in experimental social psychology (Vol. 24, pp. 201–234). San Diego, CA: Academic Press.

Corrigan

P. W.

(2004). How stigma interferes with mental health care. American Psychologist, 59, 614–625.

10.

Favaro

St. Philip

Slaughter

(2019, July 28). New clinic treats patients with reversible condition often mistaken for dementia. CTV. Retrieved from https://www.ctvnews.ca/health/new-clinic-treats-patients-with-reversible-condition-often-mistaken-for-dementia-1.4525349

11.

Gallo

K. P.

Comer

J. S.

Barlow

D. H.

(2013). Direct-to-consumer marketing of psychological treatments for anxiety disorders. Journal of Anxiety Disorders, 27, 793–801. doi:10.1016/j.janxdis.2013.03.005

12.

Gellad

Z. F.

Lyles

K. W.

(2007). Direct-to-consumer advertising of pharmaceuticals. The American Journal of Medicine, 120, 475–480. doi:10.1016/j.amjmed.2006.09.030

13.

Goldstein

N. J.

Cialdini

R. B.

Griskevicius

(2008). A room with a viewpoint: Using social norms to motivate environmental conservation in hotels. Journal of Consumer Research, 35, 472–482. doi:10.1086/586910

14.

Google. (2017a). Bounce rate. Retrieved from https://support.google.com/analytics/answer/1009409?hl=en

15.

Google. (2017b). Session duration, avg. Retrieved from https://support.google.com/analytics/answer/1006253?hl=en

16.

Google. (2018). Experiment results. Retrieved from https://support.google.com/360suite/optimize/answer/7405044?hl=en&ref_topic=7404525

17.

Harik

J. M.

Hundt

N. E.

Bernardy

N. C.

Norman

S. B.

Hamblen

J. L.

(2016). Desired involvement in treatment decisions among adults with PTSD symptoms. Journal of Traumatic Stress, 29, 221–228. doi:10.1002/jts.22102

18.

Harik

J. M.

Matteo

R. A.

Hermann

B. A.

Hamblen

J. L.

(2017). What people with PTSD symptoms do (and do not) know about PTSD: A national survey. Depression and Anxiety, 34, 374–382. doi:10.1002/da.22558

19.

Hundt

N. E.

Mott

J. M.

Miles

S. R.

Arney

Cully

J. A.

Stanley

M. A.

(2015). Veterans’ perspectives on initiating evidence-based psychotherapy for posttraumatic stress disorder. Psychological Trauma, 7, 539–546. doi:10.1037/tra0000035

20.

Kohn

Saxena

Levav

Saraceno

(2004). The treatment gap in mental health care. Bulletin of the World Health Organization, 82, 858–866.

21.

Levinson-King

(2019, August 25). ‘There was no hope’: Treatable disease often mistaken for Alzheimer’s. BBC. Retrieved from https://www.bbc.com/news/world-us-canada-49375308

22.

McHugh

R. K.

Barlow

D. H.

(2010). The dissemination and implementation of evidence-based psychological treatments: A review of current efforts. American Psychologist, 65, 73–84. doi:10.1037/a0018121

23.

Mojtabai

Olfson

Sampson

N. A.

Jin

Druss

Wang

P. S.

. . . Kessler

R. C.

(2011). Barriers to mental health treatment: Results from the national comorbidity survey replication. Psychological Medicine, 41, 1751–1761. doi:10.1017/S0033291710002291

24.

Mora Ringle

V. A.

Walsh

L. M.

Maxwell

C. A.

Smith

A. M.

Grossman

R. A.

Becker

S. J.

Jensen-Doss

(2019). Understanding of evidence-based mental health care and the perceived importance of scientific information in a sample of U.S. adults. Journal of Clinical Psychology. doi:10.1002/jclp.22856

25.

Resick

P. A.

Wachen

J. S.

Dondanville

K. A.

Pruiksma

K. E.

Yarvis

J. S.

Peterson

A. L.

, . . . STRONG STAR Consortium. (2017). Effect of group vs individual cognitive processing therapy in active-duty military seeking treatment for posttraumatic stress disorder: A randomized clinical trialgroup vs individual cognitive processing therapy in PTSD group vs individual cognitive processing therapy in PTSD. JAMA Psychiatry, 74, 28–36. doi:10.1001/jamapsychiatry.2016.2729

26.

Rothman

A. J.

Salovey

(1997). Shaping perceptions to motivate healthy behavior: The role of message framing. Psychological Bulletin, 121, 3–19.

27.

Schofield

C. A.

Weis

Ponzini

McHugh

R. K.

(2017). Improving perceptions of cognitive-behavioral therapy with brief website exposure: Does neuroscientific allure impact attitudes? Journal of Behavior Therapy and Experimental Psychiatry, 57, 53–61. doi:10.1016/j.jbtep.2017.03.003

28.

Scott

S. L.

(2010). A modern Bayesian look at the multi-armed bandit. Applied Stochastic Models in Business and Industry, 26, 369–658. doi:10.1002/asmb.874

29.

Scott

S. L.

(2017). Multi-armed bandit experiments. Retrieved from https://support.google.com/analytics/answer/2844870?hl=en&ref_topic=1745207

30.

Sriram

Greenwald

A. G.

(2009). The brief implicit association test. Experimental Psychology, 56, 283–294. doi:10.1027/1618-3169.56.4.283

31.

Stecker

Shiner

Watts

B. V.

Jones

Conner

K. R.

(2013). Treatment-seeking barriers for veterans of the Iraq and Afghanistan conflicts who screen positive for PTSD. Psychiatric Services, 64, 280–283. doi:10.1176/appi.ps.001372012

32.

Teachman

B. A.

Drabick

D. A. G.

Hershenberg

Vivian

Wolfe

B. E.

Goldfried

M. R.

(2012). Bridging the gap between clinical research and clinical practice: Introduction to the special section. Psychotherapy, 49, 97–100. doi:10.1037/a0027346

33.

VanderPlas

(2014). Frequentism and Bayesianism: A python-driven primer. In van der Walt

Bergstra

(Eds.), Proceedings of the 13th Python in Science Conference (pp. 85–93). Retrieved from https://conference.scipy.org/proceedings/scipy2014/pdfs/proceedings.pdf

34.

Ventola

C. L.

(2011). Direct-to-consumer pharmaceutical advertising: Therapeutic or toxic? Pharmacy and Therapeutics, 36, 669–684.

35.

Wilkes

M. S.

Bell

R. A.

Kravitz

R. L.

(2000). Direct-to-consumer prescription drug advertising: Trends, impact, and implications. Health Affairs, 19, 110–128. doi:10.1377/hlthaff.19.2.110

36.

Wolitzky-Taylor

Zimmermann

Arch

J. J.

De Guzman

Lagomasino

(2015). Has evidence-based psychosocial treatment for anxiety disorders permeated usual care in community mental health settings? Behaviour Research and Therapy, 72, 9–17. doi:10.1016/j.brat.2015.06.010

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.58 MB