Iterative Design and Testing for the Development of a Game-Based Chlamydia Awareness Intervention: A Pilot Study

Abstract

Objectives:

Herein we describe a methodology for developing a game-based intervention to raise awareness of Chlamydia and other sexually transmitted infections among youth in Boston's underserved communities.

Materials and Methods:

We engaged in three design-based experiments. These utilized mixed methods, including playtesting and assessment methods, to examine the overall effectiveness of the game. In this case, effectiveness is defined as (1) engaging the target group, (2) increasing knowledge about Chlamydia, and (3) changing attitudes toward Chlamydia testing. These three experiments were performed using participants from different communities and with slightly different versions of the game, as we iterated through the design/feedback process.

Results:

Overall, participants who played the game showed a significant increase in participants' knowledge of Chlamydia compared with those in the control group (P = 0.0002). The version of the game, including elements specifically targeting systemic thinking, showed significant improvement in participants' intent to get tested compared with the version of the game without such elements (Stage 2: P > 0.05; Stage 3: P = 0.0045). Furthermore, during both Stage 2 and Stage 3, participants showed high levels of enjoyment, mood, and participation and moderate levels of game engagement and social engagement. During Stage 3, however, participants' game engagement (P = 0.0003), social engagement (P = 0.0003), and participation (P = 0.0003) were significantly higher compared with those of Stage 2. Thus, we believe that motivation improvements from Stage 2 to 3 were also effective. Finally, participants' overall learning effectiveness was correlated with their prepositive affect (r = 0.52) and their postproblem hierarchy (r = −0.54).

Conclusion:

The game improved considerably from its initial conception through three stages of iterative design and feedback. Our assessment methods for each stage targeted and integrated learning, health, and engagement outcomes. Lessons learned through this iterative design process are a great contribution to the games for health community, especially in targeting the development of health and learning goals through game design.

Introduction

Researchers in the field of Games for Health have particularly focused upon designing gamified systems for health behavior change, specifically focusing on the design, development, and evaluation of such approaches. Examples of this include games that encourage physical activity,^1–4 healthy eating,^1,5,6 and self-regulation of emotions.^7,8 Several evaluation methods for these games have been proposed, including (but not limited to) Pham et al., who utilized a randomized controlled trial (RCT) approach to evaluate the feasibility and clinical efficacy of a mobile health game.⁸ Previous research has shown that game engagement and sustained use are essential for health behavior change⁹ and learning.¹⁰ Despite this, evaluation of health games has rarely focused on engagement as a modifiable factor.^9,10

Additionally, while there has been a great deal of work designing games for health purposes, relatively little has addressed sexually transmitted infections (STIs) in youth. In 2008 alone, 110 million people in the United States were affected by at least one of the top eight STIs.¹¹ Among Boston residents in 2014, Chlamydia was both the most commonly reported STI and the most communicable disease overall.¹² To reduce rates of Chlamydia and racial, ethnic, and social determinants of health associated with Chlamydia, the Boston Public Health Commission (BPHC) hosted several workshops in predominantly high-risk areas. In the interest of providing better outreach and avenues for the successful dissemination of sexual health education, however, BPHC and our group collaborated to develop and evaluate a game for educating youth about Chlamydia and encouraging STI testing. There are several reasons that games, the media of choice for today's youth, may be a successful alternative in this endeavor. Namely, games provide a safe and private channel for education regarding sensitive issues, such as safer sex and STI testing and treatment.^13–15 Furthermore, games may provide an alternative/simulated environment in which students can be immersed in realistic contexts while mitigating and resolving real-life dilemmas.

While there have been games developed for STI education, the majority of these are either prototypes or have not been thoroughly evaluated.^16–18 For example, even those that have undergone systematic evaluation (e.g., Miller et al.^19,20) failed to include in-depth assessments of players' experience (e.g., engagement) to determine whether participants would choose to play these games without experimental incentives. As discussed above, previous research has shown that engagement and retention of subjects using a game environment are critical factors²¹ that affect learning¹⁰ and health behavior change.⁹ Finally, despite the fact that youth (especially young women) have been shown to be most at risk for acquiring STIs,²² no previous research specifically targeted Chlamydia education and prevention among youth.

In this article, we address this lack of Chlamydia education by presenting an innovative game designed to increase knowledge of Chlamydia among at-risk youth, motivate them to get tested, and, if they test positive, encourage them to seek treatment early to stop the spread within the population. In addition to the game, we developed a methodology that will help to address the lack of evaluation methods in the field by combining engagement metrics with knowledge, health, and learning outcomes. We incorporated an iterative design and testing method to calibrate the game and balance its serious purpose with players' motivation and engagement. The method included formative and summative evaluation processes,²³ similar to Microsoft's Rapid Iterative Testing and Evaluation,²⁴ but adjusted for education content by emphasizing evaluation techniques that take into account both engagement and educational outcomes. Incorporating both of these factors into our formative and summative evaluations affected the instruments we chose. Engagement is an essential component in games that has a direct impact upon motivation and thus learning; however, engagement alone cannot improve learning outcomes unless the game design also emphasizes educational elements.²⁵ In this study, we present the iterative testing method and design of a Chlamydia education game called “Neighborhood.”

Game-based health interventions for STIs

Examples of STI interventions include partner notification, screening programs, and school programs.¹¹ Of these examples, school health programs may be the most cost-effective prevention program to reduce the risk of STIs.²⁶ Curricula that disseminate knowledge through small-group discussions, videos, interactive exercises, and skill-building activities have been shown to be effective STI prevention interventions.²⁷ Games may be a reasonable alternative or complementary intervention as students are already familiar with the medium and the barrier to engagement is low.

Several games have previously been developed for STI interventions, utilizing the game environment to educate students by presenting them with scenarios similar to real life. Downs et al.²⁸ developed an interactive video intervention and conducted an RCT with 300 adolescent girls. Participants showed (1) increased academic knowledge about STIs and (2) reduced reported risky behavior and STI acquisition. Miller et al.^14,20 designed a 3D storytelling game and conducted a 6-month RCT with 876 participants.¹⁹ Participants revealed shame reduction and cognitive variable increase (intention, self-efficacy, and consideration of future consequences, all had P values <0.01) after playing the game.¹⁹ Chu et al.¹⁸ developed a game and found moderate improvement in 788 participants' safe sex knowledge after play. Guana et al.¹⁷ created a game, which rather than promoting behavioral change or awareness, showed STIs as bacteria and taught players about the diseases at a biological and chemical level. However, Guana et al.'s¹⁷ work lacked in-depth evaluation of the effectiveness of their game.

When evaluating an intervention intended to change behavior, self-efficacy has been shown to be a key prerequisite.²⁹ Miller et al.^15,20 explored the idea of increasing self-efficacy by incorporating observational and procedural learning into SOLVE. Their work argued that decisions made in a virtual environment with appropriate feedback can alter players' risky decision-making patterns.²⁰ Such an approach can be translated directly from HIV in SOLVE to Chlamydia in “Neighborhood.” Both deal with similar hidden and shame-filled aspects, as well as participants' lack of general knowledge about the specific ways that the disease spreads. Furthermore, both support similar testing of participants' knowledge development, self-efficacy, and motivation to take action regarding testing and treatment.

Furthermore, while Downs et al., Chu et al., and Miller et al. have each shown the efficacy of a game-based intervention in a laboratory setting, it is unclear whether such interventions would be effective at engaging or retaining players in the general public as engagement and retention were not studied as part of their evaluations. None of these projects focused on engagement as part of the design or development process, as we advocate herein. Tennyson and Jorczak's interactive cognitive complexity model proposed that a participant's affective (i.e., motivation and attitude) and cognitive (i.e., memory, knowledge base, and executive control) processes interact with sensory information from educational content to enhance knowledge.³⁰ Malone emphasized the importance of intrinsic motivation and designing different play activities to promote deep learning and behavioral change.³¹ These and others have shown that measuring engagement is critical to evaluation of a game-based educational tool due to its impact upon learning. Thus, the “Neighborhood” design and evaluation extended previous methods for designing health-based games by incorporating engagement into the formative and summative evaluation at each iterative step.

Materials and Methods

The “Neighborhood” game

We developed a game called “Neighborhood” (Fig. 1). “Neighborhood” serves as an alternate reality, social simulation board game, which mirrors students' life during summer break. It offers a chance to engage in social situations, safer sex practices (use of a condom), and disease testing activities; between rounds of play, participants' knowledge about Chlamydia is tested through trivia questions. Table 1 describes the characteristics of the game in greater detail. A video of the gameplay is available at https://www.youtube.com/watch?v=wKrpvd2MVY4 During gameplay, players may be infected with Chlamydia; as in real life, these individuals may not be aware of their infection. Thus, to reduce personal (and partner) risk, we advise that participants routinely get tested, which is an option available to participants throughout the game.

FIG. 1.

Physical game space (left), game's main menu (center), and one of the four stations while the game is played (right).

Table 1.

Characteristics of “Neighborhood” Game

Characteristic	Description
General game characteristics
Health topics	Knowledge and attitude toward STIs, especially Chlamydia
Target age group	13–24 years old
Other targeted group characteristics	Underserved and high-risk youth in Boston
Short description of game idea	The “Neighborhood” game is a mixed (mixes the physical and virtual) alternate reality social board game. It has a digital scoring board; its mechanics and scoring system were designed to teach students how Chlamydia can be prevented when all members work together to raise awareness, ensure engagement in condom use and other forms of safe sex practices, get tested, and (when necessary) receive proper treatment. The game is played with a group of 7–24 people led by 1–2 facilitators (educator). Engaging in the gameplay process allows students to learn the effect of their actions on themselves and the community.
Target players:	Small group
Guiding knowledge or behavior change theory(ies), models, or conceptual framework(s):	Tennyson and Jorczak's interactive cognitive complexity mode and Theory of Planned Behavior—The game focuses on changing knowledge, beliefs, and attitudes around Chlamydia and to eventually influence intentions to get tested for Chlamydia if necessary.
Indented health behavior changes:	Raise awareness and motivate early testing of STIs, specially Chlamydia, among youth
Knowledge elements to be learned:	Basic Chlamydia-related knowledge and consequence of good and bad sex-related decisions.
Behavior change procedure:	Experiential learning: the players take the role of a student who will get Chlamydia if he/she is not careful enough; they will experience the situation to make sex-related decision, clinic decision, and the consequence of their decisions to themselves even to the community.
Clinical or parental support needed? (please specify):	No
Data shared with parents or clinician:	□Yes No
Type of game:	□Active □Action □Adventure Role-playing Sports Casual Educational □Other:
Story(if any)
Synopsis (including story arc):	Players each play the role of a student on a summer break. In the game, they could (1) go to a party to make relationship with others; (2) compete with peers in a park to do sports challenges; (3) go to the Community Center to learn Chlamydia knowledge; (4) go for a test even treatment at “Clinic Center”; and (5) set up a relationship with others.
How the story relates to targeted behavior change:	All decisions the student makes
Game components
Player's game goal/objectives:	Players of “Neighborhood” game should earn as much points as possible to win the game.
Rules	Points in the game are distributed based on partaking in specific health behaviors, such as participating in activities at the park station, making friends in the party station, and answering questions correctly at the community center station and during the trivia phase of each round. However, points are lost when a participant acquires Chlamydia or infects others. It is important to note that participants have to make decisions about where to spend their time during each round; for example, the clinic allows them to get tested, but they miss out on the collection of points from other stations. Participants will be punished (get negative points) if they transmitted the disease or left it untreated. By the end of this game, a facilitator calculates participants' points; points are announced as well as the winner of the game.
Game mechanics:
Procedure to generalize or transfer what is learned in the game to outside of the game:	Procedural rhetoric or the idea that the player of the game is part of the message of the game.
Virtual environment
Setting(describe)	NA
Avatar
Characteristics:	NA
Game platforms need to play the game:	Smartphone Tablet Computer (The game is a social card game with an interface for scoring used by the facilitator in each round.)
Sensors used	NA
Estimated play time:	35 minutes

NA, Not Applicable; STI, sexually transmitted infection.

The game is played in several turns, each of which includes three different phases (Fig. 2). During each of these turns, participants can choose to perform activities in one of several stations. The goal is to show participants how communicable diseases can spread within a population and from person to person.

FIG. 2.

One round of “Neighborhood” game.

At the start of the game, participants receive a randomly assigned, undisclosed Chlamydia status (infected or not). After each round, participants' Chlamydia status (and knowledge of their status) may change based on several factors: (1) getting tested and/or treated, (2) partner interactions, represented in the game as a high five, and (3) condom use (or nonuse) during such interactions. If infected participants do not choose to go to the clinic or get tested, they may infect others unknowingly since Chlamydia status is hidden until testing.

Points in the game are distributed based on specific health behaviors undertaken at each station, such as engaging in activities at the park station, making friends in the party station, or answering questions correctly at the community center station. Participants can also earn points by answering questions correctly during the trivia phase of each round. However, points are lost (sometimes without the player's knowledge) when a participant acquires Chlamydia, goes without treatment, or infects others. It is important to note that participants must decide where to spend their time during each round to maximize their points; for example, the clinic allows them to get tested, but they miss out on the collection of points from other stations. At the end of the game, a facilitator calculates and announces participants' points; the winner is the player who earned the most net points (most positive and least negative).

Study design

Prototypes and playtesting have been noted as critical components to the design of good games.³² To develop and balance a game that fulfilled our educational goals, we formulated a study that comprised three stages. We began with a prototype, and then used an iterative testing, evaluation, and refinement process to improve the game before the final playtesting stage. In each of these stages, we used several validated measures to improve the game. We targeted both engagement and learning as integrated pieces of the game evaluation.

Stage 1

The primary purpose of the first stage was to test the game's usability and engagement³³ (basic measures used in commercial game design). After this first stage, the requirements could be refined. For these early-stage tests, we combined several approaches. To test usability, we employed qualitative observations of play sessions to identify confusion or points where participants asked procedural questions about the game. To understand engagement, we utilized the Intrinsic Motivation Inventory^34,35 developed through Self-Determination Theory.³⁶ This theory ties game design to players' motivation; its application in educational and health-related games has shown the significant effect of player engagement on knowledge retention and positive outcomes.⁹ We used other measures as well, including the Game Expertise Questionnaire, which involved the game playing habits of the participants, and the Time Use Survey,³⁷ which allowed participants to report how they used their time in general. Thus, in summary, we used the following measures:

• Observation and play notes to measure usability

• Self-Determination Theory's Intrinsic Motivation Inventory to measure motivation

• Game Expertise Questionnaire to measure game habits and interests

• Time Use Survey to measure lifestyle and routines during a typical day

Information about participants' motivations, interests, and habits allowed us to adjust the game to target these qualities. Observation of gameplay helped us to understand and address usability issues with the current prototype.

Stage 2

The goal of the second stage was to refine the game to achieve its educational objectives within the population. This involved more rigorous testing of both learning and engagement of participants. For this stage, we employed several measurements, shown in Figure 3, and enumerated as follows:

• For learning:

○ Knowledge and Attitude Assessment (KAA)^12,38

• For engagement and affect:

○ Positive Affect, Negative Affect Scale (PANAS)³⁹

○ Player Experience Questionnaire⁴⁰

○ Usability observation and facilitator notes

FIG. 3.

The procedure of Stage 2.

For this stage, we used a between-subjects experimental design in which participants either experienced the game-based intervention or a control lecture, which covered the same material as the game. Participants were randomly selected from the total group to participate in either the control or the intervention; however, to have the best possible control, it was recommended to have a small class setting for the lecture. Thus, only 7 of the 38 participants were randomly selected into the control group. We measured learning, engagement, and affective outcomes for each condition. Specifically, we administered the PANAS and KAA surveys before and after each session to compare the two interventions. Our goal was to assess the educational utility of the game relative to a lecture (a standard method used by BPHC, our funder, to educate and raise awareness about STIs in classrooms and small-group settings), as well as affect and engagement measures for each.

We developed the KAA survey by combining (1) knowledge-testing questions based on the materials disseminated through the BPHC website⁹ and (2) attitude-testing questions modified from the Monitoring Outcomes of HIV Intervention questionnaire published by University of Texas Southwestern Medical Center.³⁸ The KAA survey contained 24 questions in total and was reviewed by the BPHC before use in our experiment.

The PANAS survey³⁹ is a validated instrument to measure affect. It consists of several words describing different feelings (positive and negative) that participants are asked to rate based on their current emotional state. The survey generates two scores: one for Positive Affect (PA) and one for Negative Affect (NA).

To measure participants' engagement, we used Downs et al.'s player experience scale.⁴⁰ Compared with other game experience questionnaires (e.g., Brockmyer et al.⁴¹), this survey has the versatility to capture player experience during a physical multiplayer game, such as “Neighborhood”. It contains questions about players' emotion during play (enjoyment and mood) as well as active engagement levels (game engagement, social engagement, and participation).

We also collected qualitative notes on usability during play, as we did for Stage 1.

Stage 3

At the final stage, we conducted a summative evaluation (rather than the formative evaluation done in earlier stages). The summative evaluation assessment goals were to investigate whether (1) the game disseminated knowledge about Chlamydia effectively, (2) participants exhibited greater willingness to get tested for Chlamydia, use condoms, and pay more attention to personal risks and environmental factors associated with Chlamydia after playing the game than before, (3) participants' affect influenced learning outcomes, (4) participants were engaged in the game, and (5) participants' engagement in the game affected the learning outcomes.

Therefore, in this stage, we again made use of the evaluation measures described in Stage 2. In addition to the KAA, which represents a relatively objective aspect of participants' learning, we also introduced a Self-reported Learning questionnaire extracted from Fu's E-Gameflow Scale⁴² in this stage. This survey gauged players' subjective opinion of their knowledge improvement through the game. A high score in the Self-reported Learning subcomponents would indicate that the participant believes he or she is learning while engaged in the intervention.

Participants

Our study procedures were approved by the Institutional Review Board (IRB) at Northeastern University. We recruited a total of 88 participants over all stages of the study. In Stages 1 and 2, we recruited 32 (12 females) and 38 (19 females) participants, respectively, from Whittier Street Community Center, Boston Glass, EMK Academy and Grove Hall Community Center. In Stage 2, 7 participants (3 females) were randomly assigned to the control group from the participant list and the final 31 participants (16 females) were assigned to the intervention (game) group. In the final stage, we recruited 18 (13 females) participants through Whittier Street Community Center (Boston). A signed consent was obtained, in person, from participants recruited for this study. Participants came to the abovementioned health centers where they participated in the experiment for a total of 70 minutes; 35 were devoted to the play session, 15 to pretest, and 20 to the post-test assessment.

Data collection and variable extraction

In Stage 2, data from two participants in the intervention group were incomplete and were thus deleted before analysis, which made a total of n = 29 for the intervention group. In Stage 3, 1 participant's data were deleted for the same reason and thus n = 17.

We extracted 22 and 23 variables and from the instruments used in Stage 2 and Stage 3, respectively; we examined each in depth. Table 2 describes how each variable was calculated. Note that variables marked with * have both a pre- and postintervention component (e.g., preknowledge and postknowledge).

Table 2.

Variable Calculation

Instrument	Score	Variable	Stage
Knowledge and Attitude Assessment	Sum of scores on knowledge questions	Knowledge^*	2 and 3
	Sum of questions involving participant's appraisal of personal risk of Chlamydia	Appraisal of personal risk^*	2 and 3
	Single question involving participant's problem hierarchy for Chlamydia	Problem hierarchy^*	2 and 3
	Sum of scores on questions involving intent to get tested for Chlamydia	Testing intent^*	2 and 3
	Sum of scores on condom use questions	Condom attitude^*	2 and 3
	Sum of scores on familiarity of environment facilitator questions	Environmental facilitator^*	2 and 3
PANAS	Sum of scores on 10 positive affects	Positive affect^*	2 and 3
PANAS	Sum of score on 10 negative affects	Negative affect^*	2 and 3
Self-reported Learning	Sum of score on participant's subjective opinion on game's educational function	Self-reported learning	3
Game Score	Rank the participants based on their game score	Game rank	2 and 3
Player Experience Questionnaire	Sum of scores on game enjoyment questions	Game enjoyment	2 and 3
	Sum of scores on 2 good mood subscale questions minus sum of scores on 3 bad mood subscale questions.	Game mood	2 and 3
	Sum of scores on game engagement questions	Game engagement	2 and 3
	Sum of scores on social engagement questions	Social engagement	2 and 3
	Sum of scores on Game Participation questions	Participation	2 and 3

Variables marked with ^* have both a pre- and postintervention component (e.g., preknowledge and postknowledge).

PANAS, Positive Affect, Negative Affect Scale.

Data analysis

To compare pre- and postgame scores for each participant, we used the nonparametric paired Wilcoxon signed rank test. We also used Spearman's ρ to calculate the correlation between variables to determine the relationship between these variables and the transfer of knowledge intended by the intervention. Note that we used multiple comparisons during the analysis of these results and, as such, used a Bonferroni correction to determine the significance of differences.

Results

Stage 1 results

Usability testing in Stage 1 indicated that (1) the trivia phase was the most valuable part of this game for engaging participants and fostering learning; (2) timing in the game was unbalanced, for example, the activity in the clinic station took more time than activities in other stations, leaving some participants waiting and inviting distraction; and (3) the goal of the game was unclear to players. The surveys regarding time use, intrinsic motivation, and game preference helped us understand our participants. Figure 4a shows that 58% of participants reported being motivated by competence—that is, feeling efficient, effective, and masterful in one's environment.³⁷ Thus, activities/games providing a strong sense of competence may be more attractive to players. Furthermore, Figure 4b shows their five favorite games.

FIG. 4.

Target group's (a) intrinsic motivation, (b) game preference.

The Time Use Survey³⁷ revealed that five participants spent more than 10 hours playing games per week. Furthermore, we found that boys among our participants enjoyed sports after school, while girls engaged in social activities. This provided guidance for the redesign of activities at the park station and party station.

These results helped us refine the game in preparation for further testing in Stage 2. These refinements were focused upon four distinct areas: (1) maximizing the educational value of trivia phase by adding more questions; (2) reducing the participants' waiting/distraction time by streamlining activity at the clinic station and during the high five phase; (3) adding activities during the station phase to satisfy intrinsic competence needs; and (4) clarifying the goal of the game for participants.

Stage 2 results

Analyses of data collected in Stage 2 showed that participants' overall knowledge about Chlamydia significantly increased in the game intervention group from pre to post (P = 0.0002). Participants in the control group did not show a significant increase (P > 0.05) (Fig. 5a). Moreover, there was only a numerical (not a statistically significant) improvement in participants' intention to get tested for both groups (Fig. 5b).

FIG. 5.

Comparison between intervention and control groups' (a) knowledge of Chlamydia, (b) intention for test of Stage 2. ***p < 0.01.

The PANAS showed that the intervention group increased in PA from pre to post (P = 0.02), while the control group did not (P = 0.26).

To test our assumption that game engagement would positively affect knowledge gained, we performed correlation tests between participants' game experience (taken from Downs et al.'s player experience scale⁴⁰) and knowledge acquisition. We found no significant correlations. Descriptive statistics from the game experience questionnaire are shown in Table 3.

Table 3.

Game Experience in Stage 2

Game experience	Mean	SD	95% confidence Interval
Enjoyment	4.98	1.42	±0.53
Mood	5.61	1.09	±0.40
Game engagement	3.42	1.78	±0.66
Social engagement	4.09	1.64	±0.61
Participation	5.00	2.00	±0.74

We also gathered usability notes during play. Facilitators noted that streamlining the clinic station and high five phase reduced the time balance problem considerably. However, another time balance problem emerged during this stage, centering on point distribution: tracking the undisclosed-to-participants Chlamydia status and awarding points at each phase once again caused a bottleneck in gameplay. Further notes revealed that participants engaged in two distinct processes: learning and exploring. The game failed to properly incentivize learning over other behaviors. Above all, we have found that the game did not properly target system learning.

Stage 3 results

In Stage 3, we removed the lecture component in favor of greater statistical power to examine the pre/posteffects of the game intervention. Figure 6a shows that participants' postgame knowledge of Chlamydia is significantly greater than their pregame knowledge (P = 0.003). This is consistent with results from Stage 2.

FIG. 6.

Pre- and post- (a) knowledge and (b) intention for test of Chlamydia in Stage 3. ***p < 0.01.

Additionally, we collected players' self-reported feelings about their own learning during play in this stage. As a group, the participants in this phase indicated the following from the questionnaire (on a scale 0–5, with 0 indicating strong disagreement and 5 indicating strong agreement):

• “The game has increased my knowledge” (mean = 4.72, SD = 0.46)

• “I catch the basic ideas” (mean = 4.61, SD = 0.78)

• “I tried to apply the knowledge in the game” (mean = 4.28, SD = 1.02)

• “The game motivates me to integrate the knowledge taught” (mean = 4.5, SD = 0.62)

• “I want to know more about what was taught” (mean = 3.83, SD = 1.04)

Participants' intention to be tested for Chlamydia increased significantly from pre to post (P = 0.004), as shown in Figure 6b.

Participants' affect, as measured by the PANAS, did not significantly change from pre- to post-gameplay (P > 0.05 for each). However, there was a significant increase in the interested mood of participants (P = 0.02) when we examined each mood component separately.

Descriptive statistics of participants' overall game experience in this stage is shown in Table 4. When compared with Stage 2 results, participants demonstrated higher levels of Game Engagement, Social Engagement, and Participation. However, this group of participants reported lower levels of overall mood compared with those of Stage 2's game group.

Table 4.

Game Experience in Stage 3

Game experience	Mean (max = 7)	SD	95% confidence interval
Enjoyment	4.76	1.28	±0.61
Mood^**	5.29	0.97	±0.46
Game engagement^***	3.99	1.19	±0.57
Social engagement^***	4.15	1.04	±0.49
Participation^***	5.24	1.30	±0.62

Statistically significant compared with variables in Stage 2: ^**P < 0.05, ^***P < 0.01.

We also correlated learning effectiveness with other variables to investigate which other factors may be related to the educational component of the game. As Table 5 shows, two of these correlations were significant: pre-PA (positively) and postproblem hierarchy (negatively). See the Discussion and Limitations section for further details.

Table 5.

Correlations Between Learning Effectiveness, Self-Reported (SR) Learning, and Other Variables in Stage 3

	Prepositive affect	Postproblem hierarchy	SR learning	Game rank
Learning effectiveness	0.52^**	−0.54^**	−0.1	−0.16

	Enjoyment	Mood	Game engagement	Social engagement	Participation
SR learning	0.58^***	0.5^**	0.44^*	0.58^**	0.52^**

P < 0.1, ^**P < 0.05, ^***P < 0.01.

We also examined the relationship between other variables and the Self-reported Learning score—that is, what factors may be related to participants feeling that they learned something during the game, regardless of the outcome. Table 5 shows variables that significantly positively correlated with Self-reported Learning score. These are all engagement variables such as enjoyment, mood, game engagement, social engagement, and participation (all from Downs et al.'s player experience scale⁴⁰).

Discussion and Limitations

Main contribution of this work: an iterative design approach based on engagement and learning

The final version of the intervention was significantly improved from the initial prototype. Through iterative design and testing, we targeted specific aspects of the game that influence players' learning and game engagement. The evaluation performed in Stage 1 helped us to identify one major time balance problem as well as the elements which provided the greatest opportunity for educational improvement. The formative experiment in Stage 2 demonstrated our game's advantages over a standard (lecture) approach to disseminating the information. It should be noted, however, that we did not compare a full class-based approach with the game-based approach because we feel that this game can be incorporated as part of a class for ideal engagement and transfer of information.

One of the most important findings from Stage 2 was that even though knowledge gain was more significant for the game than the lecture, the difference in terms of effect size was very small (as shown in the figure above). This challenged us to explore ways in which game-based learning could be more superior and to recommend the incorporation of system learning into this intervention, as discussed in the games for learning literature.⁴³ This key contribution of the study led us to make several useful changes to the game design. Thus, the final version targeted systems learning with balanced rules and a more challenging activity.

Knowledge and attitude change

From the significant positive change in overall knowledge across participants both in Stage 2's intervention group and Stage 3 (Figs. 5a and 6a), we can conclude that our game was an effective intervention with regard to increasing participants' knowledge of Chlamydia. As this is the primary underlying purpose of the trivia questions and overall game, we are not surprised to find this result. Furthermore, as the control group in Stage 2 showed no significant knowledge gain, we can conclude that our game intervention was more effective in educating students about Chlamydia than a traditional lecture method.

Moreover, we were surprised to find that the final version of the game promoted a detectable attitude change in such a small group. Figure 6b shows this to be the case, with a significantly higher intent to test for Chlamydia in participants after the game than before. However, as shown in Figure 5b, neither the intervention group nor the control in Stage 2 showed a significant attitude change. This suggests that the improvements made in the third iteration of the game engaged participants' attitudes. This is notable as the game was designed to target systems thinking, emphasizing the importance of testing and fostered learning.

In both Stages 2 and 3, several attitude measures showed no significant change during play. These included attitudes toward the use of condoms, appraisal of personal risk, problem hierarchy, and environmental barriers. This lack of change in some attitude measures may simply be a reflection of the small sample size of this study or an indication of future opportunities for game improvement.

Game experience and mood

In Stage 2, our analyses showed that participants in the intervention group had significantly higher positive mood after their participation, compared with the control group. In Stage 3, by contrast, only participants' interest showed a significant increase. This may be due to the small sample size or to a ceiling effect of the other subcategories of positive mood.

Regardless, participants clearly enjoyed the game, rating their game enjoyment and mood during play high in both Stages 2 and 3. Thus, we can conclude that the game successfully encouraged participants to engage in most game activities by implementing proper game design strategies leading to enjoyment. In Stage 3, results show that participants' 3 most highly rated game experiences were Game Engagement, Social Engagement, and Participation. These findings guided the final changes we made to tailor the game to our target population.

Self-reported learning

In the final stage, participants reported feeling that they learned a great deal. This is positive as participants who feel that they learned a lot often have a more enjoyable educational gaming experience. Indeed, our group showed positive correlations between self-reported learning and game enjoyment, good mood, social engagement, and participation (Table 5). This suggests that participants who enjoyed the game (1) engaged more with other participants and (2) participated more in the game activity than those who did not enjoy the game.

However, it is important to note that we found no significant correlation between participants' self-reported learning and their actual knowledge attainment. This suggests that the participants who feel that they are learning the most are not necessarily correct. This may also be partially due to a ceiling effect (i.e., participants' self-reported learning could not change much in the positive direction), however, as the overall Self-reported Learning score is quite high (and variance is relatively low) compared with knowledge score.

Learning effectiveness mediated by affect

Results of Stage 3 indicated that the learning effectiveness of the game is highly positively correlated with positive mood before play, as noted in Table 5. Game facilitators during the study also reported this relationship; “the participants who seem enthusiastic/interested/excited perform with more concentration in the trivia phase.” This suggests that participants who are more upbeat before the activity are more willing to take in and retain information during the activity.

The negative correlation between learning effectiveness and postgame problem hierarchy was surprising; however, one possible explanation is that participants were less afraid of Chlamydia when they understood the knowledge being conveyed. The game provided a relaxing and safe environment to learn about the nonfatal disease, Chlamydia. This may simply indicate that our game successfully conveyed the idea that Chlamydia can be treated by one dose of antibiotics.

Limitations of this study

The data presented herein represent a small sample (n = 88) of a very specific population (underserved and high-risk youth in Boston). This small sample size limits the generalizability of results from the research and may lead to some population bias especially when we conduct an RCT. While we believe that these results are useful and relevant to the population studied, we have no way of knowing whether they would continue to hold true for other populations. Thus, more work is needed to replicate these results. Furthermore, our quantitative data were obtained from surveys, which were self-reported, and thus are limited. Finally, in our discussion, we presented some possible causes for observed correlations between our data. However, we did not have the power to create statistically significant predictive models indicating causal relationships. This is a goal for future research.

Conclusion

In conclusion, through an iterative design and testing method that combined learning and engagement with formative and summative evaluation methods, we have successfully developed an engaging game that disseminates knowledge about Chlamydia and encourages participants to get tested. Our game involves social interactions that urge players to work as a group to control the spread of disease. The repercussions of individual actions during the game, designed to simulate the spread of the disease, taught players to think carefully about the sexual health-related choices they will be making in later game turns and in life. Even though the intervention is nondigital, the dissemination of knowledge was facilitated due to the game's self-contained and easy-to-replicate nature (a simulation-driven card game).

Although the study is limited due to the small sample of (n = 88) participants distributed across multiple stages and game versions, we were able to show some significant quantitative results and introduce methods that contribute to the growing body of literature in games for health. We specifically argued for, and showed the importance of, utilizing an iterative design method to produce better results (i.e., better engagement and educational outcomes). The game was refined according to intermediate assessments and finally evaluated in its third version. Our results also showed that the final game is self-motivating and participants enjoy joining in the game's activities (answering questions, competing at the park station, giving a partner a high five, etc.). The correlations we found through these studies offer guidance for our future research and for other researchers that intend to develop educational games in this or other areas.

The next step of our study includes the transfer of “Neighborhood” to a digital version that can be disseminated within the Boston public health and community centers. In the development of the digital version, we intend to continue our methodological focus on engagement, affect, and motivation, as well as learning. We also intend to strengthen our methodology through a more fully developed qualitative component to measure engagement and connections made during learning as these have proven to be critical contributions to our present study.

Footnotes

Acknowledgments

The authors are grateful to the participants and Whittier Street Community Center, Boston Glass, EMK Academy, and Grove Hall Community Center for assisting them with this research project. They would also like to acknowledge the BPHC for providing financial support for this project and Stork Burnt Down, LLC, for facilitation of the design and fine-tuning of the game.

Author Disclosure Statement

No competing financial interests exist.

References

Elnasr

, Andres

, Lavender

, et al. IgnitePlay: Encouraging and sustaining healthy living through social games. IEEE International Games Innovation Conference. IEEE. 2011:23–25.

Lin

, Mamykina

, Lindtner

, et al. Fish'n’Steps: Encouraging Physical Activity with an Interactive Computer Game. UBICOMP 2006: Ubiquitous Computing, International Conference, UBICOMP 2006, Orange County, CA, September. DBLP. 2006:261–278.

Moller

, Majewski

, Standish

, et al. Active fantasy sports: Rationale and feasibility of leveraging online fantasy sports to promote physical activity. JMIR Serious Games, 2014; 2:e13.

Wang

. Building personal wellness communities: Meaningful play in the everyday life of a network society. Dissertations & Theses—Gradworks, 2010.

Kaufman

, Flanagan

, Seidman

, et al. “RePlay Health”: An experiential role-playing sport for modeling healthcare decisions, policies, and outcomes. Games Health J, 2015; 4:295–304.

Peng

. Design and evaluation of a computer game to promote a healthy diet for young adults. Health Commun, 2009; 24:115–127.

Morris

, Kathawala

, Leen

, et al. Mobile therapy: Case study evaluations of a cell phone application for emotional self-awareness. J Med Internet Res, 2010; 12:398–400.

Pham

, Khatib

, Stansfeld

, et al. Feasibility and efficacy of an mHealth game for managing anxiety: “Flowy” randomized controlled pilot trial and design evaluation. Games Health J, 2015; 5:50.

El-Nasr

, Durga

, Shiyko

, Sceppa

. Unpacking adherence and engagement in persuasive health games. Foundations of Digital Games, 2015.

10.

Garris

, Ahlers

, Driskell

. Games, motivation, and learning: A research and practice model. Simul Gaming, 2002; 33:441–467.

11.

Peterman

, Carter

. Effective interventions to reduce sexually transmitted disease: Introduction to the special issue. Sex Transm Dis, 2016; 43(2 Suppl 1):S1.

12.

States, United, By Gender, By Race, and Boston Epidemiology. STIs in Boston: Chlamydia Brief 2014; 2014:0–2.

13.

Chib

. Network influences in health initiatives: Multimedia games for youth in Peru. Dissertations & Theses—Gradworks, 2007.

14.

Miller

, Christensen

, Godoy

, et al. Reducing risky sexual decision-making in the virtual and in the real-world: Serious games, intelligent agents, and a solve approach. In: Serious Games: Mechanisms and Effects. 2008:34–37.

15.

Miller

. SOLVE-IT: Socially optimized learning in virtual environments: A web-delivered HIV prevention 3D game intervention for young at-risk MSM. J Mob Technol Med, 2012; 1(4S):10–10.

16.

Gabarron

, Schopf

, Serrano

, et al. Gamification strategy on prevention of STDs for youth. Stud Health Technol Inform, 2013; 192:1066.

17.

Guana

, Xiang

, Zhang

, et al. UnderControl an educational serious-game for reproductive health. ACM Sigchi Symposium on Computer-Human Interaction in Play—Chiplay. ACM. 2014:339–342.

18.

Chu

, Kwan

, Reynolds

, et al. Promoting sex education among teenagers through an interactive game: Reasons for success and implications. Games Health J, 2015; 4:168–174.

19.

Christensen

, Miller

, Appleby

, et al. Reducing shame in a game that predicts HIV risk reduction for young adult MSM: A randomized trial delivered nationally over the web. J Int Aids Soc, 2013; 16(Suppl 2):18716.

20.

Miller

, Marsella

, Dey

, et al. Socially optimized learning in virtual environments (SOLVE). Lecture Notes Comput Sci, 2011; 7069:182–192.

21.

Milam

. Game design framework and guidelines based on a theory of visual attention. Diss. Communication, Art & Technology. School of Interactive Arts and Technology, 2013.

22.

Division of STD Prevention.. Tracking the hidden epidemics; trends in STDs in the United States 2000. Epidemiology, 2013.

23.

Seif El-Nasr

, Desurvire

, Nacke

, et al. Game user research. CHI ‘12 Extended Abstracts on Human Factors in Computing Systems. ACM. 2012:2679–2682.

24.

Desurvire

, El-Nasr

. Methods for game user research: Studying player behavior to enhance game design. IEEE Comput Graph Appl, 2013; 33:82.

25.

Egenfeldt-Nielsen

. Beyond edutainment exploring the educational potential of computer games. Lulu.Com, 2005.

26.

Centers for Disease Control and Prevention.. Effective HIV and STD prevention programs for youth: A summary of scientific evidence. Atlanta: US Department of Health and Human Services. http://www.cdc.gov/healthyyouth/sexualbehaviors/pdf/effective_hiv pdf (accessed May 13, 2007).

27.

Villarruel

, Iii

JBJ

, Jemmott

. A randomized controlled trial testing an HIV prevention intervention for Latino youth. Arch Pediatr Adolesc Med, 2006; 160:772–777.

28.

Downs

, Murray

, De Bruin

, et al. Interactive video behavioral intervention to reduce adolescent females' STD risk: A randomized controlled trial. Soc Sci Med, 2004; 59:1561–1572.

29.

Gaskill

, Hoy

. Self-efficacy and self-regulated learning: The dynamic duo in school performance. In: Aronson (Ed.) Improving 176 Shaughnessy Education: Classic and Contemporary Lessons From. Academic Press, 2002:185–208.

30.

Tennyson

, Jorczak

. A conceptual framework for the empirical study of instructional games. In: O'Neil

, Perez

(Eds.), Computer Games and Team and Individual Learning, 2008:39–54.

31.

Malone

. What makes things fun to learn? Heuristics for designing instructional computer games. SIGSMALL ‘80, 1980:162–169.

32.

Boller

. Play Testing Games-An Essential Step. http://www.bottomlineperformance.com/play-testing-games-an-essential-step/,2011-11-10/2016-10-26 (accessed May 13, 2017).

33.

Capra

. Usability problem description and the evaluator effect in usability testing. Doctoral dissertation; Industrial and Systems Engineering, Virginia Polytechnical Institute and State University; 2006.

34.

Ryan

, Mims

, Koestner

. Relation of reward contingency and interpersonal context to intrinsic motivation: A review and test using cognitive evaluation theory. J Pers Soc Psychol, 1983; 45:736–750.

35.

Rigby

, Ryan

. The Player Experience of Need Satisfaction (PENS). White Paper 2007. http://immersyve.com/white-paper-the-player-experience-of-need-satisfaction-pens-2007/ (accessed May 13, 2017).

36.

Deci

, Ryan

. Self-determination theory: A macrotheory of human motivation, development, and health. Can Psychol, 2008; 49:182–185.

37.

Harvey

. Guidelines for time use data collection. Soc Indicators Res, 1993; 30:197–228.

38.

University of Texas Southwestern Medical Center. Available at: http://www.utsouthwestern.edu/media/files/300/205983Question-Bank—English.pdf (accessed May 3, 2017).

39.

Watson

, Clark

, Tellegen

. Development and validation of brief measures of positive and negative affect: The PANAS scales. J Pers Soc Psychol, 1988; 54:1063–1070.

40.

Downs

, Vetere

, Howard

, et al. Measuring audience experience in social videogaming. Australian Computer-Human Interaction Conference: Augmentation, Application, Innovation, Collaboration, 2013:217–220.

41.

Brockmyer

, Fox

, Curtiss

, et al. The development of the Game Engagement Questionnaire: A measure of engagement in video game-playing. J Exp Soc Psychol, 2009; 45:624–634.

42.

, Su

, Yu

. EGameFlow: A scale to measure learners' enjoyment of e-learning games. Comput Educ, 2009; 52:101–112.

43.

Zhou

. The Princess in the Castle: Challenging Serious Game Play for Integrated Policy Analysis and Planning. Gilderprint Drukkerijen:Enschede, The Netherlands, 2014.