A Pilot Randomized Controlled Trial to Evaluate a Cognitive Behavioral Videogame Intervention: empowerED

Abstract

Objective:

To design a cognitive behavioral videogame intervention and to evaluate its preliminary effect on improving youth’s perceptions about themselves, their future, and their ability by incorporating skill development in cognitive reappraisal, an emotion regulation strategy to change perceptions.

Materials and Methods:

We conducted a pilot RCT to evaluate the preliminary efficacy of a videogame intervention, empowerED, in enhancing cognitive reappraisal skills. We also assessed beliefs/attitudes, self-efficacy, and emotional self-efficacy as well as user experience and feasibility. Youth (N = 100) aged 14–19 years from one school were enrolled in a pilot RCT assigned either to play empowerED or a control condition.

Results:

Improvements were noted in cognitive reappraisal in the empowerED group compared to the control group (LSM difference = 1.33, P = 0.01). There were no significant differences observed between treatment groups for beliefs/attitudes, self-efficacy, and emotional self-efficacy; however, the empowerED group reported improvements in beliefs/attitudes from pretest to posttest survey responses (M = 1.33, P = 0.01).

Conclusion:

Overall, the intervention was deemed easy to use and beneficial among youth, and feasibly delivered in a high school setting. Given the growing youth mental health needs in schools and the importance of school climate on healthy development, empowerED may offer an effective and innovative student-level approach to improve cognitive reappraisal and later empower youth to enact change in their school climate. Clinical Trials.gov Identifier: NCT04025294.

Impact statement

Videogames offer an engaging and easy to use approach to deliver a cognitive behavioral intervention and are feasibly delivered in schools. Through a brief single-session intervention, we showed promising results in delivering an emotion regulation strategy, and improving cognitive reappraisal, to alter one's perceptions. Further research is needed to explore if altering perceptions will lead to enacting change in one's school climate.

Introduction

School climate is a comprehensive, multidimensional construct¹ that includes five essential areas: safety, relationships, teaching and learning, institutional environment, and school improvement process.² According to the National School Climate Center, school climate is the overall experience of school life that captures the norms, goals, values, interpersonal relationships, teaching and learning process, and organization structures.³ Prior research has linked a positive school climate to improvements in youth self-esteem,⁴ academic achievement,^5,6 absenteeism,⁷ aggression and violence,^8,9 and alcohol and illicit drug use.¹⁰ Therefore, establishing a sustainable and positive school climate is essential for youth development and learning,³ and engaging youth in this process is fundamental to the overall approach.

Promoting youth empowerment and agency is an approach to engage youth in decision-making processes.¹¹ Youth have the potential to take action to enact change in their lives, including their school climate.¹² However, evidence suggests that instead of taking on a more active role in enacting change, youth often choose to carry out a more passive and thus less effective approach to addressing their concerns (e.g., spectating).¹² Therefore, it is essential for youth to develop agency to advocate and feel empowered to enact change as they navigate their school climate.¹³

Youth who develop agency are better able to utilize their assets and aspirations to make their own decisions about their lives and set their own goals to achieve desired outcomes.¹⁴ Given their unique perspectives, youth must be given a platform and feel empowered to actively shape and advocate for their advancements.¹³ According to the Positive Youth Development Framework,¹⁴ positive youth perceptions around their identity, their ability, and their future are critical in developing agency.

To improve youth perceptions, cognitive reappraisal serves as an emotion regulation strategy to reframe negative thoughts/beliefs into positive ones and is considered an effective strategy.¹⁵ A recent study reported cognitive reappraisal fully mediated the link between agency and the changes in one's perceptions of self, social relationships, and life priorities as a result of experiencing a challenging event.¹⁶ Therefore, to support the development of youth agency, the videogame intervention, empowerED: Think It Through Digital Stories, models the process of restructuring negative automatic thoughts through the application of an adapted thought record in a digital format. A thought record is an empirically supported psychosocial intervention¹⁷ that serves as a strategy in cognitive behavioral therapy to challenge negative beliefs and thought patterns, and practice skill development in cognitive reappraisal (i.e., reframing negative thoughts).

The goal of a thought record is to identify the situation when feeling discomfort, describe and rate the intensity of the emotions felt, and describe the thoughts that went through one's mind¹⁸ to reduce the intensity of negative moods by using rational thoughts to settle the mind.¹⁹ The central aim of this study was to assess the preliminary efficacy of an evidence-informed, interactive narrative-based videogame intervention in enhancing the skill cognitive reappraisal for youth to later develop agency and enact change in their school environment.

Further, we evaluated beliefs/attitudes and skills (e.g., self-efficacy and emotional self-efficacy) as well as usability and acceptability (e.g., user experience) among youth and feasibility (e.g., protocols and procedures in conducting the study) in delivering a videogame intervention embedded into school climate assessment tool in a high school setting. Future research will explore agency as the driving force to empower high school-aged youth to enact change in their school climate as well as evaluate the actual responses of the school climate assessment tool in the context of gameplay.

Materials and Methods

empowerED intervention

empowerED is a videogame intervention where the player engages with branching decision-based mini-stories (or scenarios) focused on school climate. The empowerED videogame intervention is embedded within a web-based application and follows the completion of a digital school climate assessment tool. The school climate tool asked a series of questions related to five dimensions of one's school climate: relationships, support for learning, social media, safety, and environment. Research on the school climate assessment tool was expanded and discussed in another study on web-based educational assessment tools for students.²⁰ The mechanics for both the assessment tool and the empowerED intervention were based on a popular strategy game Reigns®, where the player swipes among limited options to accept or reject suggestions to progress.

To create a fun and engaging strategy-based game, each mini-story in empowerED included six short, relatable narratives where each player must successfully model the 7-step “Think It Through” process on how to restructure negative automatic thoughts in different situations and navigate challenging situations that take place in school settings. The “Think It Through” process, which is a thought record, prompts players to reflect on the story and observe how the character reframes their thinking by swiping on appropriate options to improve the situation (Table 1). The thought record was adapted to include wise interventions²¹ to alter the meaning of players' thoughts. A wise intervention is a theory- and research-based activity that helps people change how they interpret themselves, others, or social situations.²¹

Table 1.

Think It Through Steps: Based on an Adapted Thought Record¹⁸

Step:	Name of step	Description
Step 1	Feel It!	Identify physical symptoms of distress.
Step 2	Name It!	Label the emotion.
Step 3	Sync It!	Connect the emotion to the thought.
Step 4	Know It!	Make meaning out of what one is telling oneself.²¹
Step 5	Flip it!	Reframe the negative thought into a more balanced one.
Step 6	Master It!	Provide examples to support the more balanced thought.
Step 7	Share It!	Share what they learned from the mini-game.

One example of a wise intervention is a self-affirmation. Based on the motives that guide their meaning, players then engaged in a step which served as a “call to action” to encourage youth to take one specific action to improve their school environment outside of the digital experience. When players do not appropriately restructure negative automatic thoughts, they are provided with an evidence-based reason why that choice was incorrect and given an opportunity to go back to do it over. An example of a story and summary of the flow for each story are shown in Supplementary Appendix S1.

The six stories are based on an introduction and the five school climate dimensions. Examples of stories range from a student posting and feeling underappreciated because no one liked a post on social media (Social Media dimension) to a student experiencing test anxiety and speaking to the counselor (Support for Learning dimension) to a student being pushed while walking to class and feeling frustrated about lack of control in hallways (Safety dimension).

Participants

Participants were youth who were students from one high school in Connecticut. Before recruitment, the research team met with school administration to provide an overview of the study and obtain approval. Inclusion/exclusion criteria included (1) youth aged 14–19 years; (2) English-speaking; (3) youth assent; (4) parent/guardian consent (if younger than 18 years of age); and (5) willingness to engage with an iPad for about 1–2 hours after school (on school grounds). Youth were informed that they would stay afterschool one time for approximately an hour and a half. Those who participated received a $20 Visa gift card for their participation in completing assessments.

Randomization

After obtaining written or electronic youth assent and parent/guardian legal consent, eligible youth were assigned to the empowerED group or control group in a 1:1 ratio using a computerized single randomization scheme. Randomization was stratified by grade (9/10 and 11/12) and sex at birth. Enrolled youth were randomized in an unmasked manner to the empowerED group or to the control group. A randomization scheme was generated in OnCore, a comprehensive clinical trial management system managed by Yale University School of Medicine.

Procedure

All procedures were approved by the Human Investigation Committee (HIC) at Yale University. The research team used three checklists to ensure their readiness for visit, participants' completion of tasks, and the team's appropriate next steps with data: (1) Visit Checklist; (2) Gameplay Checklist; and (3) Assessment Checklist. Pretest/posttest surveys were collected through a secured, online data collection website (Qualtrics Data Collection Software) before and after youth's 60–90-minute gameplay experience as instructed by the research team. The series of questions related to school climate and the empowerED intervention (only experimental group) were accessed through a private, password-protected website. Data were collected between October 2019 and December 2019, and data were analyzed in early 2020.

empowerED group

The empowerED group completed the following steps: (1) pretest surveys; (2) a series of questions about school climate, (3) the empowerED intervention, (4) a review of a public website inspirED Online Resource Center;²² (5) posttest surveys; and (6) focus groups.

Control group

The control group completed the following steps: (1) pretest surveys; (2) a series of questions about school climate; (3) a review of a public website inspirED Online Resource Center;²² and (4) posttest surveys.

Assessments

To explore the preliminary efficacy of this intervention, our research team assessed cognitive reappraisal as our primary outcome. Secondary outcomes include: (1) beliefs/attitudes; (2) self-efficacy; (3) emotional self-efficacy; (4) user experience (postintervention survey and focus group only); and (5) feasibility. Feasibility was measured through observation. We also collected number of minutes spent on the three stages of game play: (1) a series of questions about school climate; (2) the intervention (if experimental); and (3) inspirED Online Resource Center.

Pretest/posttest surveys

Cognitive reappraisal

Cognitive Reappraisal, the primary outcome of this study, was assessed with a six-item measure (5-point scale) of The Emotion Regulation Questionnaire for Children and Adolescents (ERQ-CA),²³ which is designed to assess two emotion regulation strategies: cognitive reappraisal and expressive suppression.

Beliefs/attitudes

Eight questions related to beliefs and attitudes asked about one's opinion on the process of changing the way one thinks. Beliefs and attitudes were measured using questions developed by our research team and were not previously validated. Questions assessed if participants believed it was helpful to change how they think, if the way they think affects how they feel, if they can change how to think about difficult situations, about setting and accomplishing goals was easy, and how they felt about enacting change in their schools. Responses ranged from strongly disagree to strongly agree (5-point scale).

Self-efficacy

Self-efficacy was measured using 10 items (5-point scale) from a subdomain of the NIH Toolbox core domains (e.g., emotion domain)²⁴ to assess the belief in one's competence to respond to stressful situations.

Emotional self-efficacy

Emotional Self-Efficacy was measured using 8 items (5-point scale) that are part of a subscale for Emotional Self-Efficacy from the Self-Efficacy Questionnaire for Children (SEQ-C)²⁵ to assess the perceived capability of coping with negative emotions.

User Experience (empowerED group only)

The Shortened User Experience Questionnaire (UEQ-S)²⁶ is a widely-used evaluation tool for interactive products to assess the usability (e.g., pragmatic quality) and acceptability (e.g., hedonic quality) of a digital experience with benchmarks using eight-items. It was measured using 8 items (7-point scale with two subscales: pragmatic and hedonic quality; 4-items each). For the purpose of this study, we attributed pragmatic quality to “usability,” or whether it is easy and clear to use, and hedonic to “acceptability,” or whether it is fun and engaging to use.

Focus Groups (empowerED group only)

For the empowerED group, each gameplay session was followed by a brief 25–45-minute (M = 21.3) focus group to collect input from youth about their digital experience for a total of 10 focus groups (n = 49). Youth provided additional insight about their user experience and suggested next steps to build out the intervention. One author (C.S.F.F.) guided the discussion using a semistructured interview guide while another author (C.K.T.) observed behaviors and collected detailed field notes on observations. To explore usability and acceptability, questions were asked to gather insight about how youth perceived the intervention, including but not limited to their perceptions on whether it was easy and clear as well as engaging and relatable.

Feasibility

Feasibility was measured by observation of actual practices to address the following: Can it work? Does it work? Will it work?²⁷ Given that the last two questions are discussed in our other results regarding the delivery of a school climate assessment tool embedded into a videogame intervention (e.g., focus group findings and preliminary efficacy), feasibility also explored Can it work? through piloting procedures and processes (e.g., recruitment, randomization, measures, data collection, etc.) to obtain information required to design a full-scale randomized controlled trial (RCT).²⁸ It was assessed through observation of completion rates and amendments needed to the Yale University HIC, if any, to the protocol to successfully execute the study with plans to design a full-scale RCT in the future.

Data analyses

Descriptive statistics were generated for participant baseline characteristics (e.g., age, grade, sex at birth, ethnicity, and race) and compared between treatment group using chi-square or Fisher's exact test for categorical variables and Student's t test for continuous variables. Descriptive statistics were also generated for overall user experience and feasibility based on youth completion of the UEQ-S. All focus groups were transcribed using a third party. Given the quick-paced, iterative, and real-time nature of game development, our approach was guided by Braun and Clarke's thematic analysis²⁹ and techniques from rapid qualitative research and analysis.³⁰ Rapid qualitative analysis is useful to inform near real-time intervention development and supports the collection of applied qualitative data while ensuring rigor.³¹ Two authors completed systematic debriefing³² after each focus group and then created an outline of salient themes from them.

Two authors then independently reviewed transcriptions using thematic coding to identify main responses that aligned with the outline. Discrepancies were discussed until a consensus was reached. This study's sample size is based on the literature providing rationale for sample size for pilot studies,³³ given the focus on preliminary efficacy, user experience, and feasibility. General Linear Model (GLM) was used to examine whether change in outcomes (including cognitive reappraisal, beliefs/attitudes, self-efficacy and emotional self-efficacy) were different between treatment group. Changes in paired pre- and postvalues of the outcomes were presented for each treatment group with 95% confidence interval and tested using paired t test.

To further explore factors associated with improvement on outcomes within treatment group, GLM univariate analyses were also used. Missing data accounted for 1.77% of total data and were determined to be randomly missing. SPSS software (IBM Corp. Released 2019. IBM SPSS Statistics for Macintosh, Version 26.0.) was used for all analyses, with P-value <0.05 as the significance level.

Results

Participants

Participants were youth recruited from a large high school that serves a diverse student body of 1177 with 3% of youth identifying as Asian, 11% identifying as Black, 48% identifying as Hispanic/Latinx, 35% identifying as White, and 2% identifying as two or more races. In total, the setting includes 65% total racial/ethnic minority enrollment. Across the school, 48% of youth identify as female and 52% of youth identify as male. Sixty-five percent of youth are also part of the free or reduced lunch program.

A total of 100 youth was randomized into one of the two groups, and 98 of them (40 females; 58 males) in grades 9 through 12 between the ages of 14 and 19 (M = 15.42, SD: 1.31) completed the pretest survey, gameplay session, and posttest survey (Table 2). Two youth withdrew from the study due to change in interest and did not attend a session. See Figure 1 for Consort Flow Diagram.

FIG. 1.

Consort Flow Diagram.

Table 2.

Baseline Demographic Characteristics of Study Participants

Characteristic	empowerED, n = 49	Control, n = 49	Total, n = 98	P-value
Age (years), mean (SD)	15.45 (1.40)	15.39 (1.22)	15.42 (1.31)	0.82
Grade, n (%)				0.21
Ninth	15 (30.6%)	13 (26.5%)	28 (28.6%)
Tenth	16 (32.7%)	16 (32.7%)	32 (32.7%)
Eleventh	14 (28.6%)	9 (18.4%)	23 (23.5%)
Twelfth	4 (8.2%)	11 (22.4%)	15 (15.3%)
Female, n (%)	21 (42.9%)	19 (38.8%)	40 (40.8%)	0.76
Hispanic/Latinx, n (%)	28 (57.1%)	29 (59.2%)	57 (58.2%)	0.84
Race, n (%)
White	16 (32.7%)	14 (28.6%)	30 (30.6%)
Multi-Racial	11 (22.4%)	8 (16.3%)	19 (19.4%)
Black/African American	3 (6.1%)	8 (16.3%)	11 (11.2%)	0.62
Asian/Native American/Other	16 (32.7%)	18 (36.7%)	34 (34.7%)
Prefer not to say	3 (6.1%)	1 (2.0%)	4 (4.1%)

Outcomes empowerED versus control group

We observed improvement in cognitive reappraisal (M = 1.67, P < 0.001) and beliefs/attitudes (M = 1.33, P = 0.01) in the empowerED group. There were no significant changes observed for self-efficacy (M = −0.20, P = 0.74) or emotional-self efficacy (M = 0.47, P = 0.48) in the empowerED group. There were no significant changes observed for any outcomes for the control group (Table 3 and Fig. 2). Average time to complete the empowerED intervention was 13.04 minutes.

FIG. 2.

Paired pre- and post-mean change in outcomes for each treatment group.

Table 3.

Paired Pre- and Post-Mean Change in Outcomes for Each Treatment Group and Comparison of Overall Mean Change in Outcomes Between Treatment Groups

	empowerED, (n = 49)		Control, (n = 49)		Comparing groups, (N = 98)
Construct	M [CI]	P-value	M [CI]	P-value	LSM Diff [CI]	P-value
Cognitive reappraisal	1.67 [1.03 to 2.32]	<0.0001^***	0.15 [−0.58 to 0.87]	0.69	1.33 [0.38 to 2.27]	<0.01^*
Beliefs/attitudes	1.33 [0.30 to 2.35]	0.01^*	0.57 [−0.27 to 1.42]	0.18	0.76 [−0.56 to 2.07]	0.26
Self-efficacy	−0.20 [−1.41 to 1.00]	0.74	0.75 [−0.24 to 1.74]	0.13	−0.95 [−2.5 to 0.59]	0.22
Emotional self-efficacy	0.47 [−0.86 to 1.80]	0.48	−0.19 [−1.10 to 0.73]	0.68	0.66 [−0.94 to 2.26]	0.42

P value less than 0.05.

***

P value less than 0.0001.

For cognitive reappraisal, the empowerED group demonstrated greater improvement compared to the control group (least squares means [LSM] difference 1.33, 95% CI, 0.38–2.27, P < 0.01) (Table 3). There were no significant differences observed between treatment groups for beliefs/attitudes, self-efficacy, or emotional self-efficacy.

Impact of grade, sex, and ethnicity on outcomes in empowerED group

In a subanalysis, we observed that ninth graders exhibited greater improvement in cognitive reappraisal compared to 10th graders (LSM Difference = 1.63, P = < 0.01) or 12th graders (LSM Difference = 1.73, P = 0.02), and 11th graders exhibited greater improvement in cognitive reappraisal than 10th graders (LSM Difference = 1.33, P = 0.04) (Supplementary Table S1).

In a subanalysis, we observed that female Hispanic/Latinx youth exhibited greater improvement in beliefs/attitudes compared to male Hispanic/Latinx youth (M_F = 1.89, M_M = −0.24, P = 0.03).

User experience (empowerED group only)

User experience was measured on a range from −3 to 3, with mean scores above 1.5 interpreted as highly favorable, scores below −1.5 interpreted as highly unfavorable, and scores near 0 interpreted as neutral. Youth described empowerED as more supportive than obstructive (M = 1.1), much more easy than complicated (M = 1.6), more efficient than inefficient (M = 1.1), and more clear than confusing (M = 1.7), which demonstrate highly favorable usability. Ninth graders reported less favorable usability scores overall compared to older youth.

Youth described empowerED as only slightly more exciting than boring (M = 0.1), slightly more interesting than not interesting (M = 0.6), slightly more inventive than conventional (M = 0.8), and slightly more leading edge than usual (M = 0.6), which demonstrate neutral acceptability (Supplementary Table S2). Salient themes emerged from focus group data that align with themes related to usability and acceptability and may provide additional context to data collected in the UEQ-S.

Overall, youth noted that the purpose of the intervention was clear and helpful. One youth said: “…the Think it Through process helps you actually know that you can actually do something about your problem…it can help you to take more steps to like make the actual like right decision.” However, they also shared that the text was too long and that youth do not like to read long texts. One youth shared: “…when I was reading the stories… I'm pretty sure most of us just skimmed through and skipped… it's a lot to read, and not a lot of teenagers love reading these days unless it's like an Instagram post or something.”

Several youths also noted that the digital stories were relatable. One youth shared: “Specifically, the stories were basically something that if people like us who go to this high school, we could experience that at the school, so they were something that really could happen during the school day.” Another one noted: “I could actually connect myself to them [the stories].” See additional themes in Table 4 that emerged from focus groups.

Table 4.

Focus Group Themes Related to User Experience

Construct of user experience	Theme	Quote from students
Usability	Clear understanding of purpose	“…the Think it Through process helps you actually know that you can actually do something about your problem…it can help you to take more steps to like make the actual like right decision.”“Yeah, I learned how to control my thoughts and change my perspective.”“I think that the Think it Through process is … think about something before you do it. You think about the possible outcomes and the consequences, and what you could have done better, or what you can do to prevent a certain situation. Think before you do, like that's pretty much what is was.”“I thought the stories were just a little quick lesson to just rethink like the situation you were in. Take a second. Stop. And rethink and some scenarios when somebody had anxiety from taking a test, stop. Think about it. Think about reality, but there's tons of other people that have anxiety towards, before taking a test. And it's just a natural thing. It's something people face in life. And it's nothing severe.”“The first step was when the character encountered the problem, they were usually not feeling so good, and you had to interpret how they were feeling inside, and it was usually more exaggerated than what it actually was. Then the next step was them trying to confront the problem in their own way, and then how we would expect them to, and the next was how they finally overcame it, and we had to choose how they would overcome it, and after that was the effects of them overcoming it.”“…think about it before you react. I think something that I already knew, but I didn't actually do it.”“…the point of the stories was to make you realize that you could take action and stuff, and that you're not always helpless.”
Usability	Text is long	“…when I was reading the stories… I'm pretty sure most of us just skimmed through and skipped… it's a lot to read, and not a lot of teenagers love reading these days unless it's like an Instagram post or something.”“There was too much reading.”“It's a lot of reading which I understand that some people don't like the reading.”
Acceptability	Relatability	“I could actually connect myself to them [the stories].”“I just felt like when I was reading… I can relate to this and it really does help me. I feel that it could help other people if they actually try.”“…I like the technology better, because it gave me like a visual with all the pictures, so I could like imagine myself in that situation, and what I would think.”“… since they're like relatable like from real life stuff, then you can imagine that had happened before, it makes it like more credible for the ones that are taking the survey they say. And it makes, like by credible I mean the connection it makes between the students.”“Specifically, the stories were basically something that if people like us who go to this high school, we could experience that at the school, so they were something that really could happen during the school day.”“I really enjoyed how it was like interactive with the person playing. So swiping it kind of made you more focused and paying attention to what was going on, and having the pictures, so I liked that. And also, going off of the bathrooms, like the other story lines, I feel like I related to all of them, like the anxiety with the tests, the peer pressure in the bathrooms, smoking, the locked bathrooms obviously. What other ones? I felt all of them I definitely related to.”“I feel as if we could bring it to real life situations and scenarios, because honestly, they are real life scenarios, even though we may not see them, it happens. It goes on, so for us to be able to think of a process, even if we can't think of it right away, kind of like give yourself time to think about a decision. So, I feel like it definitely does relate to life.”“I thought that even if some of them weren't relatable to us, I think it helped us through the thought process of any more like hypothetical situations.”
Acceptability	Benefits	“I mean, honestly, I was not having the best day before I came in here …Thinking pretty negatively, and then after I took the game or test or whatever, I was feeling more like accomplished or like oh yeah, I can do this. I definitely answered the questions more positively towards the end.”“Because I feel like it…I don't really want to say opened my eyes, but like I do, because I feel like it's just not really something that you typically think about on like a day to day basis, and when it's just in front of you, it's kind of like…Like the anxiety thing. It's like oh, I'm not the only one, like a lot of people think like that.”“… when you're dealing with emotions, sometimes your response isn't always what you mean. It's out of your emotional response. I have to learn about that. If you take that extra second or two to think. You'll sort of be coming from your heart rather than how you feel in that moment in time. A lot of people they'll act straight out of anger and then they'll look back on it and regret it. But even in these things like that extra two seconds to think about the choice kind of brings out how you would really feel about it. Because reading it is one thing but then understanding is another.”“I think obviously you have to identify the issue. But thinking this through comes like a flowchart almost. Like one of these the next leads to the next. So like, if you identify the issue, you can identify solutions. You think about the impacts of these solutions and whichever solution you take, you have to think about the outcome of it.”“…a lot of people struggle with a thought process of being able to calm themselves down, or just kinda’ thinking about a situation and saying, it's not as bad as it seems, and if we can start learning how to do that, it can really improve a lot of people's lives.”
Acceptability	Swiping mechanics	“I thought it was interesting…You had to hold it to left, right just to see the answer. And I thought it was kind of cool.”“So swiping it kind of made you more focused and paying attention to what was going on.”“… I kind of thought that was a little bit restricting for your thought process.”“I think it was better to just like swipe it. You know, like it was easier.”“I feel like, again, you should have more choices, but also the two choices that it was, maybe you should carry on with what the story was, and if you did the bad choice, maybe have something that would bring it up to a better choice, or something. Because if you pick, like what [participant name redacted] said. S/he said that it was, the negative choice, it would tell you to try again. Well, maybe don't tell us to try again. Maybe try to answer it and then try to make it better.”

Feasibility

The study protocol was found to be feasible and followed with minimal HIC protocol amendments (e.g., collecting parent/guardian consent electronically). On the basis of completion rates and minimal amendments, it was concluded that the study was feasible for a large-scale RCT and the intervention may be considered appropriate for a school setting. In addition, the school climate assessment tool was well-received when embedded into the intervention and the intervention was successfully delivered to high school youth as a single-session intervention in digital format.

Discussion

School climate is an elusive concept that has been assessed in numerous ways. While it is elusive, student perceptions often mediate actual school climate effects.^34–37 This is the first study, to date, to focus on reframing student perceptions around what students are able to control with the hypothesis that improved skill development in cognitive reappraisal may increase agency for youth to shift their school climate in a future study. This pilot RCT examined the preliminary efficacy, user experience (e.g., usability and acceptability), and feasibility of a cognitive behavioral intervention, empowerED: Think It Through Digital Stories. The main finding of this study suggested that youth who played empowerED reported significant improvement in cognitive reappraisal, which is critical given that cognitive reappraisal may serve as an important protective individual factor in the context of high stress.³⁸

While the field of mental health digital interventions is burgeoning and in high demand, many digital interventions are not rigorously evaluated.^39,40 Specifically, digital games that may have an impact on mental health concerns and self-regulation strategies hold promise^41,42 and many similarly incorporate aspects of CBT; however, few exist that focus on a universal prevention and youth empowerment with a focus on the school environment. For example, SPARX is a CBT game-based intervention to address depressive symptoms. It is delivered in the form of seven modules over the course of 4–7 weeks in clinical service settings⁴³ and was later evaluated by another study in a middle school setting.⁴⁴

Similar to empowerED, SPARX utilizes CBT techniques to reframe negative automatic thoughts; however, empowerED is a brief single-session intervention delivered in a high school setting while SPARX requires a longer duration of time delivered in a clinical service setting and later middle school setting—a challenge to deliver given the time commitment. Another study evaluated Reach Out Central where the player navigates real-life scenarios and uses role-play to practice CBT techniques and apply coping strategies to overcome barriers to seeking mental health support among youth and young adults who are 16–25 years of age;⁴⁵ however, the study sample included only those aged 18–25 years due to complications in obtaining informed consent from minors when their unique perspectives are warranted.

Overall, the findings from the current study reported preliminary evidence that empowerED may support youth in developing a strategy to reframe negative automatic thoughts with the goal of improving beliefs about themselves, their ability, and their future to increase their agency. In addition, findings provided insight into user experience among youth as well as perceived success in feasibly piloting procedures and protocols of the digital intervention in a high school setting.

Furthermore, subanalyses offer insight into potential explorations for increased impact. For example, ninth graders exhibited significantly greater improvements in cognitive reappraisal compared to 10th graders, and 11th graders exhibited greater improvement in cognitive reappraisal than 10th graders. It is possible that complementing school support services with empowerED may offer a more substantial impact. Often times, youth receive additional support services in ninth grade—because of the pivotal transition from middle school to high school⁴⁶—and 11th grade—because of the focus on college and career readiness during their junior year. Moreover, female Hispanic/Latinx youth exhibited greater improvement in belief/attitudes compared to male Hispanic/Latinx youth. This finding is particularly of interest because female Hispanic/Latinx youth report alarming rates of lifetime and current alcohol/drug use compared to others.⁴⁷ Literature supports approaches that utilize empowerment frameworks to prevent the risk of substance use and improve health/developmental outcomes among female Hispanic/Latinx youth.⁴⁸

Our findings were corroborated by a systematic review¹¹ that found no significant effects for self-efficacy outcomes in youth empowerment programs. We hypothesize that this may be due, in part, to the diverse cultural representation of our study sample. Adolescents from collectivist cultures tend to rate their self-efficacy lower than those from individualistic cultures despite outperforming individualistic counterparts;⁴⁹ therefore, cultural differences may account for the absence of effect of the intervention on self-efficacy.

Limitations of this study must be noted. While the findings are promising, they must be interpreted in the context of a small sample size⁵⁰ and that the study was conducted in one school setting. Given that the study was solely conducted in one school setting, potential threats to validity include social contamination between groups, unmasked randomization, and that the study may not be generalizable to more individuals other than those in our study sample. Despite these limitations, strengths of the current study include the pilot RCT design evaluating a novel digital intervention. In addition, the rationale behind the intervention has been clearly supported by research, it was offered as a single-session intervention, and it was developed in videogame form—a popular approach for engaging youth.⁵¹ Additional strengths include the use of well-validated measures, a diverse sample, and a high completion rate.

In the future, conducting a full-scale RCT that allows for a larger sample size and long-term outcomes, including an evaluation on actual behaviors to improve school climate, will help to better assess the efficacy of this intervention. Future research would also benefit from exploring alternatives to dense texts to support the various forms of engagement for diverse learners. Next steps include a second iteration of empowerED where we will incorporate salient themes collected from youth feedback. As part of our next iteration, youth will continue to review and provide feedback on design and development to ensure a meaningful experience with regard to usability and acceptability.

In conclusion, our study demonstrated that by playing through empowerED: Think It Through Digital Stories, youth developed critical skills to restructure negative automatic thoughts, which has notable implications in improving their perceptions about themselves, their ability, and their future as well as their overall mental health. Further research is needed to evaluate if altering these perceptions will lead to enacting change in one's school climate. Overall, empowerED was considered easy to use and beneficial among youth, and is feasible to deliver in a school setting. Given the growing mental health needs of our youth in schools and the importance of youth involvement in improving their school climate, empowerED: Think It Through Digital Stories offers the potential of an effective and innovative approach that does not overburden schools but addresses this critical need.

Footnotes

Acknowledgment

The authors thank the students, educators, and school leaders who supported this work and who are deeply committed to our youth. The authors also thank the team at Schell Games for their partnership in designing the game experience.

Authors' Contribution

C.S.F.F. conceived the presented intervention, led the design and development of the intervention, as well as devised the plan for data collection. She also drafted the article. Twitter: @DrClaudiaSantiF

Y.D. provided statistical expertise in the analysis, supported the writing, and provided a critical review of this article.

A.H.T. supported the development of the intervention. He also supported the writing of this article and provided a critical review of it. Twitter: @iamalvintran

K.D.H. provided feedback on the conceptualization of the intervention as well as provided a critical review of article. Twitter: @khieftje

T.M.P.B. provided input on the data collection stage, supported the analysis and interpretation of focus group data, as well as provided a critical review of article.

C.K.T. supported the stage of data collection as well as performed data analysis/interpretation with the guidance of Mrs. Deng from the Yale Center for Analytical Sciences. She also supported the writing of the article.

L.E.F. oversaw the study design. She provided a critical review of the article as well as final approval of the version of this article to be published. Twitter: @LFiellin

Funding Information

Funded, in part, by the Silicon Valley Community Foundation Grant Number 2017-178768. CSFF is supported by Clinical and Translational Science Awards (CTSA) Program Grant Number KL2 TR001862 from the National Center for Advancing Translational Science (NCATS)/National Institutes of Health (NIH). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the National Center for Advancing Translational Sciences or the National Institutes of Health.

Supplementary Material

References

Maxwell

, Reynolds

, Lee

, et al. The impact of school climate and school identification on academic achievement: Multilevel modeling with student and teacher data. Front Psychol, 2017; 8:2069.

Thapa

, Cohen

, Guffey

, et al. A review of school climate research. Rev Educ Res, 2013; 83(3):357–385.

What is School Climate? 2007; Available from: https://www.schoolclimate.org/about/our-approach/what-is-school-climate [Last accessed: August 28, 2020].

Way

, Reddy

, Rhodes

. Students' perceptions of school climate during the middle school years: Associations with trajectories of psychological and behavioral adjustment. Am J Community Psychol, 2007; 40(3–4):194–213.

Dimitrova

, Ferrer-Wreder

, Ahlen

. School climate, academic achievement and educational aspirations in roma minority and bulgarian majority adolescents. Child Youth Care Forum, 2018; 47(5):645–658.

Brand

, Felner

, Seitsinger

, et al. A large scale study of the assessment of the social environment of middle and secondary schools: The validity and utility of teachers' ratings of school climate, cultural pluralism, and safety problems for understanding school effects and school improvement. J Sch Psychol, 2008; 46(5):507–535.

Van Eck

, Johnson

, Bettencourt

, et al. How school climate relates to chronic absence: A multi–level latent profile analysis. J Sch Psychol, 2017; 61:89–102.

Eliot

, Cornell

, Gregory

, et al. Supportive school climate and student willingness to seek help for bullying and threats of violence. J Sch Psychol, 2010; 48(6):533–553.

Gage

, Prykanowski

, Larson

. School climate and bullying victimization: A latent class growth model analysis. Sch Psychol Q, 2014; 29(3):256–271.

10.

Brand

, Felner

, Shim

, et al. Middle school improvement and reform: Development and validation of a school-level assessment of climate, cultural pluralism, and school safety. J Educ Psychol, 2003; 95(3):570–588.

11.

Morton

, Montgomery

. Youth empowerment programs for improving adolescents' self-efficacy and self-esteem: A systematic review. Res Soc Work Pract, 2013; 23(1):22–33.

12.

Noguera

, Cannella

. Beyond Resistance! Youth Activism and Community Change: New Democratic Possibilities for Practice and Policy for America's Youth. Routledge: London & New York; 2006.

13.

Cook-Sather

. Sound, Presence, and Power: “Student Voice” in Educational Research and Reform. Curriculum Inquiry, 2006; 36(4):359–390.

14.

Positive Youth Development (PYD) Framework. Available from: https://www.youthpower.org/positive-youth-development-pyd-framework. [Last accessed: August 28, 2020].

15.

Troy

, Shallcross

, Brunner

, et al. Cognitive reappraisal and acceptance: Effects on emotion, physiology, and perceived cognitive costs. Emotion, 2018; 18(1):58.

16.

Laslo-Roth

, George-Levi

, Margalit

. Social participation and posttraumatic growth: The serial mediation of hope, social support, and reappraisal. J Community Psychol, 2022; 50(1):47–63.

17.

Beck

, Emery

, Greenberg

. Anxiety Disorders and Phobias: A Cognitive Perspective. Basic Books: New York; 2005.

18.

Greenberger

, Padesky

. Mind Over Mood: A Cognitive Therapy Treatment Manual for Clients. Guilford Press: New York; 1995.

19.

Dubord

. Part 9. Thought records. Can Fam Physician, 2011; 57(8):913–914.

20.

Hoffmann

, Baumsteiger

, Seibyl

, et al. Building useful, web-based educational assessment tools for students, with students: A demonstration with the school climate walkthrough. Assess Educ Principles Policy Pract, 2022; 29(1):95–120.

21.

Walton

, Wilson

. Wise interventions: Psychological remedies for social and personal problems. Psychol Rev, 2018; 125(5):617.

22.

inspirED. Available from: https://inspiredstudents.org [Last accessed: August 28, 2020].

23.

Gross

, Thompson

. Emotion Regulation: Conceptual Foundations. Guilford Press: New York; 2007.

24.

Gershon

, Wagster

, Hendrie

, et al. NIH toolbox for assessment of neurological and behavioral function. Neurology, 2013; 80(11 Suppl 3):S2–S6.

25.

Muris

. A brief questionnaire for measuring self-efficacy in youths. J Psychopathol Behav Assess, 2001; 23(3):145–149.

26.

Schrepp

, Hinderks

, Thomaschewski

. Design and evaluation of a short version of the user experience questionnaire (UEQ-S). IJIMAI, 2017; 4(6):103–108.

27.

Bowen

, Kreuter

, Spring

, et al. How we design feasibility studies. Am J Prev Med, 2009; 36(5):452–457.

28.

Eldridge

, Lancaster

, Campbell

, et al. Defining feasibility and pilot studies in preparation for randomised controlled trials: Development of a conceptual framework. PLoS One, 2016; 11(3):e0150205.

29.

Braun

, Clarke

. Using thematic analysis in psychology. Qual Res Psychol, 2006; 3(2):77–101.

30.

Vindrola-Padros

, Johnson

. Rapid techniques in qualitative research: A critical review of the literature. Qual Health Res, 2020; 30(10):1596–1604.

31.

Lewinski

, Crowley

, Miller

, et al. Applied rapid qualitative analysis to develop a contextually appropriate intervention and increase the likelihood of uptake. Med Care, 2021; 59(6 Suppl 3):S242.

32.

McMahon

, Winch

. Systematic debriefing after qualitative encounters: An essential analysis step in applied qualitative research. BMJ Global Health, 2018; 3(5):e000837.

33.

Whitehead

, Julious

, Cooper

, et al. Estimating the sample size for a pilot randomised trial to minimise the overall trial sample size for the external pilot and main trial for a continuous outcome variable. Stat Methods Med Res, 2016; 25(3):1057–1073.

34.

Loukas

, Robinson

. Examining the moderating role of perceived school climate in early adolescent adjustment. J Res Adolesc, 2004; 14(2):209–233.

35.

Kuperminc

, Leadbeater

, Blatt

. School social climate and individual differences in vulnerability to psychopathology among middle school students. J Sch Psychol, 2001; 39(2):141–159.

36.

Roeser

, Eccles

, Sameroff

. School as a context of early adolescents' academic and social-emotional development: A summary of research findings. Elementary Sch J, 2000; 100(5):443–471.

37.

Roeser

, Eccles

. Adolescents' perceptions of middle school: Relation to longitudinal changes in academic and psychological adjustment. J Res Adolesc, 1998; 8(1):123–158.

38.

Troy

, Wilhelm

, Shallcross

, et al. Seeing the silver lining: Cognitive reappraisal ability moderates the relationship between stress and depressive symptoms. Emotion, 2010; 10(6):783.

39.

Huguet

, Rao

, McGrath

, et al. A systematic review of cognitive behavioral therapy and behavioral activation apps for depression. PLoS One, 2016; 11(5):e0154248.

40.

Firth

, Torous

, Nicholas

, et al. The efficacy of smartphone-based mental health interventions for depressive symptoms: A meta-analysis of randomized controlled trials. World Psychiatry, 2017; 16(3):287–298.

41.

Villani

, Carissoli

, Triberti

, et al. Videogames for emotion regulation: A systematic review. Games Health J, 2018; 7(2):85–99.

42.

Shah

, Kraemer

, Won

, et al. Developing digital intervention games for mental disorders: A review. Games Health J, 2018; 7(4):213–224.

43.

Merry

, Stasiak

, Shepherd

, et al. The effectiveness of SPARX, a computerised self help intervention for adolescents seeking help for depression: Randomised controlled non-inferiority trial. BMJ (Clinical research ed.), 2012; 344:e2598.

44.

Poppelaars

, Tak

, Lichtwarck-Aschoff

, et al. A randomized controlled trial comparing two cognitive-behavioral programs for adolescent girls with subclinical depression: A school-based program (Op Volle Kracht) and a computerized program (SPARX). Behav Res Ther, 2016; 80:33–42.

45.

Shandley

, Austin

, Klein

, et al. An evaluation of ‘Reach Out Central’: An online gaming program for supporting the mental health of young people. Health Educ Res, 2010; 25(4):563–574.

46.

Roderick

, Kelley-Kemple

, Johnson

, et al Preventable Failure: Improvements in Long-Term Outcomes When High Schools Focused on the Ninth Grade Year. Research Summary. ERIC: Chicago, IL; 2014.

47.

Kann

, McManus

, Harris

, et al. Youth risk behavior surveillance—United States, 2017. MMWR Surveill Summ, 2018; 67(8):1.

48.

Opara

, Lardier Jr

, Fernandez

, et al. Intrapersonal psychological empowerment profiles on ethnic identity, social support, and lifetime drug use among Hispanic adolescent girls. J Ethn Subst Abuse, 2022; 21(3):886–913.

49.

Ahn

, Usher

, Butz

, et al. Cultural differences in the understanding of modelling and feedback as sources of self-efficacy information. Br J Educ Psychol, 2016; 86(1):112–136.

50.

Leon

, Davis

, Kraemer

. The role and interpretation of pilot studies in clinical research. J Psychiatr Res, 2011; 45(5):626–629.

51.

Lenhart

, Kahne

, Middaugh

, et al. Teens' gaming experiences are diverse and include significant social interaction and civic engagement. Pew Internet & AMerican Life Project: Washington, DC; 2008.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.01 MB

0.02 MB

11.99 MB