Getting Research on Games for Health Funded

Abstract

Background

Getting funding to conduct research, in general, is a major challenge that requires many skills and single-minded persistence. Getting funding to conduct research on games for health (G4H) may pose even more challenges. A number of scientists (the authors) who have applied for grant funding to support research on G4H (most of whom have been successful, several more than once, but most of whom have also received grant application rejections) have shared their insights in regard to the funding challenges specific to G4H research and suggestions for successfully obtaining funding. Comments that elucidate points made are quoted.

The authors have applied for research funding for G4H from across the United States or several countries in Europe, and one has successfully competed for grants for G4H in both continents. Most of the authors also have served as reviewers of grant applications to fund G4H or related projects. Separate articles were considered for funding from the United States versus Europe, but several thought the differences in review considerations were minimal, and to the extent they existed, the contrast could be informative. The European context has a reputation for funding more serious game projects in general compared to the United States, which funds diverse research including both scientific and technical aspects of G4H. Alternatively, “Grant applications for serious games submitted to these agencies will be evaluated by the same criteria in US and European Union (EU).”

This article could not possibly address all the extensive and complex issues in successful grantsmanship. Readers are guided to books, workshops, articles, and/or guidelines from academics or government agencies for that (e.g., www.grantcentral.com/workbooks/national-institutes-of-health). In this study, we emphasize issues specific to G4H research applications.

The Funding Problems We Face

A couple of our coauthors (who must be way off the curve on intelligence, or luck) report never having had a problem getting multiple grants funded on game research: “In fact, we have not encountered problems with these grant proposals nor have we received ‘silly comments.’ The whole process went rather smoothly in all 4 cases.” “In our experience using the term ‘game’ is in fact a positive label. Somehow, at least in Europe, there are these fashions that funders have for health promotion, which slowly change over time. Nowadays the two most effective terms for funding are ‘Nudging’ and ‘Games’ (or gamification or serious games). I looked up the reviews for that last grant and all three reviewers were positive about the idea of using a serious game as intervention.”

The rest of us mortals, however, report encountering many challenging critiques from grant application reviewers, which ostensibly reflect four categories: (1) research on games or aspects of games not being novel: “I received the comment ‘Gaming interventions are not particularly novel’ from multiple reviewers across multiple study sections.”; (2) the reviewers do not see the value of a game approach: “… the problem I have encountered the most is that many reviewers simply devalue videogames per se.” Thus many harsh comments are based upon the inappropriateness of using exergaming in the intervention or criticism of using videogames to promote health.”; (3) a lack of credibility that specific audiences would find games useful: “The reviewers didn't believe that older adult women would use mobile devices or like zombies.”; and (4) games may be subject to biased manipulation not reflecting real behavior: “Trackers such as Fitbit and Jawbone can be manipulated to produce high steps without actually walking.” Each of these review issues will be addressed under the appropriate grant writing sections below.

Some of the comments may be appropriate for any application, while others may be unique to G4H researchers. Some comments may reflect a reviewer's lack of technical expertise: “It is not clear what the measurable objectives are for the software coding tools, or why they are needed when a playable prototype already exists.” Getting into too much detail on technical aspects is usually out-of-scope for most reviewers' expertise, but some detail would be necessary in this case. The technology that undergirds the development and functioning of G4H is rapidly evolving with new opportunities and capabilities being presented and existing dependable technology being phased out as game developers want to take advantage of the latest possibilities.¹ Given the technological basis of games and rapid progression in the enabling hardware and software: “Reviewers said the findings may not generalize to other games or consoles, and it wasn't clear how the program of research will be affected if the games and gaming consoles are discontinued.” “Two review comments I received: ‘a substantial amount of effort will have gone into identifying exergames, building of platforms, however, in a year, the technology landscape may have changed markedly’ and ‘the potential impact of the research as planned is relatively low because the innovation and significance to health outcomes are likely to be minimal by the time the work is completed.’”

Getting to Know the Funding Agency

Intimately knowing the agency is critical to writing a grant application they will want to fund. Diverse sources of funding for G4H research include the following: government agencies (e.g., U.S. National Institutes of Health [NIH], U.S. National Science Foundation [NSF], EU Horizon 2020, likely the sources with the most funds), industry (e.g., pharmaceutical and/or health insurance companies), foundations (e.g., the Wellcome Trust in the United Kingdom, Robert Wood Johnson Foundation, and HopeLab in the United States), voluntary organizations (e.g., American Heart Association, American Cancer Society, and Dutch Aids Fund), and individual hospitals or healthcare organizations. Each source has its own research interests, levels of funding offered, and sets of requirements that must be met to obtain funding. It is critical to identify the funding sources most relevant to an applicant's interests and strengths, become thoroughly familiar with the source's offerings, forms, requirements, and guidelines, and even ingratiate a staff person who may provide insights not otherwise available. Find the person in the agency who is the advocate for, or most receptive to, research on games, and get their perspective on the rules for that agency's successful submissions. For example, some funding sources are interested in randomized clinical trials (RCTs) of outcomes from game-play (e.g., NIH), while others are interested in clinical relevance and meeting the interests of their enrollees (e.g., health insurance companies). Some funding agencies are more interested in the innovation and technical advancement of the game, as well as educational elements of students working in the area of health games (e.g., NSF), while others are at least equally interested in the sophistication of the research design (e.g., NIH).

Each funding source often has solicitations for specific research issues. Responding to these (rather than to nonspecific research solicitations) is particularly valuable since those applications do not have to provide justification for the importance of the research topic to the same extent as needed for a general solicitation. They also vary in their levels and types of funding (at NIH: R01, R21, Small Business Initiative Research [SBIR]) and grant application forms. Sometimes game research applications are submitted to calls for diverse research, for example, methods to reduce posttraumatic stress disorder, interventions for disease regime adherence, or new measures of behavioral disorders. Funding agencies often change the specific types of projects they are willing to fund and/or the criteria an applicant must meet to obtain funding. So there is a premium on staying on top of the informational releases from an agency. A call to the right person working at the agency could also reveal changes in any of these that are pending.

The U.S. Congress directed the NIH to set aside 2.5% of its funds to be spent on small businesses (under 50 employees). One way the NIH implemented this directive was to develop the SBIR program. Only small businesses could be the primary applicant for SBIR funding. This has become a major source for small gaming companies to obtain game development funds. SBIR grant applications have two phases. The first phase provides a small amount of money (about $150,000) to produce a proof-of-concept prototype. The prototype is then reviewed by NIH through an application for a second phase, which, if approved, provides a larger amount of money (about $1 million) to develop and evaluate a full game. Many of the characteristics of an SBIR application are similar to those for a regular research grant, with the distinction that the SBIR applicant offers a marketing plan that holds out promise of the applicant making a profit from selling the game. “If applying for an SBIR grant, we have the additional hurdle of proving innovation in the commercial marketplace. The reviewers want to see that the game will move the literature/field forward PLUS be a viable commercial product.” The reviewers for SBIR-type grants tend to include game developers and others with substantial digital technical and business expertise, thereby requiring more attention to these issues in the application.

Some funding agencies like the NIH use topic-specific committees of reviewers and allow an applicant to select a specific review committee to review their application. Selecting the right committee (preferably with at least one member who has published research on games) could maximize the chances of a favorably disposed committee (at least with a likely game research advocate listening and responding to the discussion about G4H research). A grant applicant can benefit from having served on a committee of reviewers to obtain insights about the process of review and the issues to which reviewers most carefully attend (so volunteer to serve in this capacity!).

Research Strategy

Before writing a research grant application, the writer needs to have a clear idea of the hypotheses and methods to be incorporated into the application. Thinking and writing in terms of hypotheses-to-be-tested enhance the scientific rigor of the application and the receptiveness of science-oriented reviewers. Knowing the relevant literature (e.g., strengths, weaknesses, inconsistencies, and gaps) and how the proposed game and research build on, extend, or fill a gap will enhance the perception of rigor.^2,3 Furthermore, the game must be placed within the context of promoting, and factors influencing, health, not as an end in itself.

Successful grant applications iteratively leverage smaller research studies into next larger studies. In regard to developing a game, an initial small grant may be submitted to conduct one or more studies to collect formative data on issues in the design of the grant, for example, nature of the story/narrative to be used,⁴ and types of feedback desired by participants.^5,6 This formative data can be incorporated into a next larger grant application that proposes to build a prototype responsive to the formative data and conduct alpha tests (i.e., assessments of game mechanics and processes while the game can still be modified), maybe a beta test (identifying and eliminating bugs in programming), and a pilot/feasibility study (e.g., do the players like the prototype? will they play the full game [to maximize the intervention dose]? are tasks within the game developmentally appropriate?). “The funding in these smaller grants is sometimes barely enough to cover developers' work, before even accounting for subject matter experts. It's critical to scope the promises made for the early grants so that we can build something playable, but know it will not be more than a ‘vertical slice’ of the final experience.”

Another grant could expand the prototype into a full game, including another pilot/feasibility study. Feasibility studies are valuable to work out the bugs in an intervention trial. Feasibility studies, however, are underpowered and cannot provide statistical tests of hypotheses. A growing literature has identified the strengths and limits of feasibility studies.^7–12 New guidelines have appeared for feasibility studies, which should be carefully followed.^13,14 A fully powered RCT might then be proposed to assess the efficacy of the game under optimal circumstances (e.g., ensuring all participants play all episodes, and high quality measures are employed to assess outcomes). An effectiveness study might be proposed to test the effects in real-world circumstances (e.g., having the game offered by clinicians in their offices with the amount of the game played at the discretion of patients, using measures available in the electronic health record). Finally, if shown to be effective, a dissemination study could be proposed.

Thus, each ensuing grant application builds on what was learned from the previous grant-funded research and proposes to do only so much as can be thoroughly justified to that point (thereby likely increasing the probability of funding success). Some have been concerned that this slow pace of advancement, especially in light of the rapid evolution of the technology, runs the risk of making researches irrelevant.¹ A funding mechanism that allows combining these steps without further grant applications would be ideal. However, jumping too far ahead in the work plan without doing each step increases the likelihood of failure. While the process seems tortuous (it is!), not following the steps may be a reason for the limited evidence of effectiveness from G4H.¹⁵ This process also conservatively minimizes the funding agency's risk of failure, that is, the funding can be terminated at any point that evidence reveals the game does not work as desired. A number of steps have been proposed (e.g., early involvement of stakeholders, use of innovative smaller scale design) to minimize the time delays.¹⁶

Convening a team (that meets regularly) with expertise in all aspects of the proposed research (e.g., game development/programming, behavior change, the health issue of concern, statistics, etc.) will increase the likelihood that the application will offer sophisticated methods in all aspects of the proposed game and development. Asking a colleague who successfully competed for grant funding from the targeted funding source to see a copy of their application could provide key pointers, and even a template, for the grant application to be written. Reading successful application abstracts may provide an idea of the trend in the funding agency's preference for type of research of interest.

Frequently, the investigators submitting the research grant application are not the game developers. While some game developers prioritize working as part of a multidisciplinary team, others often work from intuitive notions of how a game might function and are not responsive (may not even read) detailed game design plans, which incorporate specifications for behavior change procedures based on the latest research. “The game designers told us they wouldn't read long design comments. We should have believed them.” In this light, when necessary, the behavioral science researcher should attempt to stay on top of game development and raise concerns when game mechanics may not be consistent with desired behavior change procedures. Conflicts will develop. All teams face severe challenges in “… balancing the content with the fun, which needs to be seamless during gameplay.” Thus, the behavioral scientist needs to identify any issues for which the game developers would like a more detailed understanding of how to design some aspect of gameplay (although often not in text) and either provide an understanding based on the published literature regarding this issue or design studies to be incorporated into the grant application to obtain that understanding.

Grant applicants vary in style. Some put off writing until the last possible minute, while others prefer generating early drafts, which get progressively refined to increase chances of success. (We endorse the latter approach.) In either case, a grant application should not be rushed to submission if the authors have any reservations that it is not in the best possible shape. Inadequately developed applications quickly become obvious to reviewers and usually receive a sound and justified thrashing.

Writing the Application

Many research funding agencies require the applicant to address similar sets of issues, and/or have reviewers grade these applications on similar criteria. We approach this task by addressing the grant sections and corresponding review criteria.

Specific aims/rationale

NIH grant applications require a single first page, called Specific Aims, which requires an overview of the Rationale for conducting the research, leading to the research's numbered Specific Aims, and ending with a brief paragraph on Significance (a busy first page). The grant application needs to reflect an intimate knowledge through specific citations of the relevant literature, starting with broader issues, for example, why addressing a particular health problem is important (e.g., national prevalence, severe disability, and earlier mortality), progressively narrowing to focus on the behavioral theory guiding the research, and ending with the issue addressed, for example, increasing fruit and vegetable consumption or reducing stress response to surgery. The rationale for the use of games as the method of intervention in the proposed research needs to be specifically addressed. This entails citing the literature on other games having successfully addressed the same or similar problems and the added benefit(s) of using a game to do this (e.g., more effectively? increased participant engagement and enjoyment? easier, less expensive dissemination?). Citing the successful use of previous games and their added benefits addresses the second common comment received on game research grant applications that reviewers do not see the value of games.

The Specific Aims must be positioned as filling a gap in the existing understanding of a problem or resolving a conflict pointed out in the literature. Some grant writers divide the Specific Aims into addressing specific hypotheses to be tested, while others identify the sequence of tasks that need to be completed (e.g., recruiting participants, formative research, game development, and alpha and beta testing). There is probably no right or wrong way of doing this, but all applications should state hypotheses to be tested. “On each hypothesis, ask yourself if you think a pretty good journal in your field would be interested in publishing the findings regardless of the result. If not, do something else.” “The Specific Aims give reviewers their first and most important impression of the proposed research. It is vital to refine the Specific Aims over and over. The non-primary review panel members may just read the Specific Aims before casting their score.”

The Specific Aims page should also briefly discuss the project's significance to specifically inform the reviewer how or why the proposed research will resolve the important issue regarding the health problem(s) addressed (summarizing points made earlier on this Specific Aims page). This ties what you propose to do to why it is important.

Comments from reviewers on this section include the following: “The reviews are often centered on the need to define the behavioral theory, which I have included by linking specific game features to types of motivational support and theoretical constructs.” “Clarify how the research project builds on current knowledge and/or available game products (most research does not start from scratch). Programmatic research will receive better response than isolated projects.”

Significance/scientific premise

The primary purpose of the second section, labeled Significance, should provide the Scientific Premise for the proposed research. For NIH grants, the words Scientific Premise need to appear in the application so reviewers do not miss it.

Addressing Scientific Premise requires providing sufficient justification for doing the research by citing appropriate work (more detailed with more references than under Specific Aims) and preliminary data (formative, pilot or feasibility studies conducted for, or positioned as conducted for, the proposed research). This section also presents the theoretical framework organizing the game and its evaluation, identifies the strengths and weaknesses in prior work in the field, and positions the proposed research as filling a significant gap or resolving an inconsistency.

Research that is a part of an overall program of research is likely to be more well-received since it indicates the investigator's commitment to the research for which she/he can marshal extensive experience to anticipate and minimize problems. “Be clear about where the proposal fits into your research trajectory and why that trajectory is super important.”

Innovation

In the last 5–10 years or so, a premium has been placed on innovativeness of the proposed research. Sometimes the emphasis on innovativeness may be misplaced (e.g., when previous work has not been done particularly well and needs to be replicated), but it avoids the trap of proposing the 20th study testing a relationship of dubious additional scientific value, for example, the possible intensity of physical activity from exergames, even if done very well. Lack of innovation of using games is a comment commonly encountered: “This is a big problem for me lately, even though everything I'm proposing has literally never been done before. Reviewers seem to feel that what I've been proposing won't move the literature forward enough … though they might have genuinely meant that game interventions are by definition not innovative but how would anything ever be innovative by that standard?” In a sense, this comment would be similar to saying that behavioral interventions are not innovative. Behavioral interventionists parry this comment by emphasizing the innovativeness of components of their behavioral interventions, often by basing this on an innovative behavioral theory, innovative operationalization of one or more behavioral constructs, or innovative theoretical framework that combines multiple theories. Game researchers need to do the same, especially by emphasizing how their proposed game mechanics and procedures operationalize behavioral theory constructs in new ways: “I think applications that propose innovative, significant research that are carefully and clearly argued have a good chance, particularly those that take a unique perspective toward a seemingly intractable problem.” Thus, positioning games to test theoretical constructs and thereby generalizing the discussions of theory and mechanisms of change can avoid the concern about the rapid changes in technology and specific game research no longer being relevant as the new technology emerges. Being scientists, reviewers will be influenced by the extent to which behavioral theory undergirds the game mechanisms and procedures.

Games may also offer innovative solutions to existing problems. For example, narratives/stories can encourage immersion in game play, thereby focusing the player's attention on the health messages deliberately inserted in gameplay¹⁷ and perhaps increase message impact. The anticipated “fun” of gameplay¹⁸ can enhance player engagement more than usual cognitive training procedures.¹⁹ However, “Many games with educational or behavior-transformation goals fall into the ‘chocolate covered brocolli’ problem, i.e., the content is coated with game-based components that are created and integrated without the expertise of a game designer. Creating a G4H is far more complex than creating an entertainment-only game.”

A proposal can also be perceived as too innovative: “There's an ongoing tension between proposing something with sufficient pilot data that ‘works’ vs. something riskier and more cutting edge.” Linking the game mechanics and procedures to behavioral theory is key to minimize this perception: “I am still struggling to get NIH funding for game-based research. The main issue I have with NIH funding is that a game is a very complicated system and it is very hard to disentangle the unique contribution of each critical component in the game. The NIH reviewers would like to see clear mapping of theoretical concepts to game components as well as study design that is able to test the effect of each of the game components.” This can be done by including a graphic conceptual model that links game mechanisms to change in psychosocial variables to behavioral and health outcomes, to help reviewers see how the game is supposed to work. Adequate pilot data can be convincing: “I think clear pilot data that demonstrate feasibility is also important. Translation research (i.e., converting a program successfully delivered through more traditional channels to a game format) may also be of interest.”

Other types of innovation are also possible: “My new direction is to use games as a tool to address theoretical questions that could impact health behaviors. I think it'll be a lot more successful than ‘I made this cool game that does xyz, let's see if it works.’ No matter how cool the game is, I think it's less exciting to reviewers (and seems less innovative) than if the research question is more specific. They are asking for very specific innovations related to new scientific knowledge or methods.” See, for example, Squires Quest!2.²⁰

Approach/research methods/scientific rigor and transparency

This is the research methods section, which allows an investigator to demonstrate the rigor of their science. This section will often have separate sections on participant identification/delineation, design, intervention description, recruitment, measures, reducing biases, process evaluation, research process, data management and analyses, sample size (statistical power or informational/response saturation), and Scientific Rigor. The words Scientific Rigor should appear toward the end of this section, summarizing how it was operationalized in the application.

Participant identification/delineation, also called inclusionary and exclusionary criteria, identifies the characteristics of participants who will be recruited, screened, and invited to participate or not. This target group should reflect the populations experiencing the health problems justifying the application, not just simply to contact participants (e.g., undergraduates at an institution), but who may not provide a good test of the game's effects.

The design selected must be appropriate to the research questions raised, for example, using a tightly controlled RCT (which is often considered the highest quality design) to obtain qualitative responses to the appreciation of gameplay would likely not be well-received, nor would open-ended statements of whether participants changed their behavior after completing a game that are subjectively, rather than objectively, measured. Some government agencies are enamored with innovative designs that allow for adapting an intervention midway in its delivery^21,22 or testing combinations of intervention procedures.^22,23 While these designs offer interesting methodological advances, they usually require larger numbers of participants to test the original hypothesis with the same statistical power after allowing for intervention modification,²⁴ which may be prohibitive for the practical concerns of producing an already expensive game.

Sometimes the game is the lone intervention and sometimes a component of a more comprehensive intervention. In either case, the intervention must be thoroughly described both in terms of theory (i.e., how it operationalizes behavior change theory constructs) and gameplay (to provide an intuitive sense of whether players will enjoy it and it is likely to have the intended effects). Often a conceptual diagram is valuable here interrelating relevant theoretical constructs and gameplay.²⁵ A single matrixed page identifying components or episodes of the game (as rows) and description of these aspects of the game²⁶ is valuable.

Recruitment is often the biggest problem faced when actually conducting the research. Programming a game for only certain kinds of hardware (e.g., iOS versus Android), or hardware requiring expensive high-level operating systems, may place limits on who can be recruited, and thereby limits on the available sample. Research with certain kinds of patients, for example, breast cancer survivors, may also experience limits on patients available from an institution's hospital. Possible problems/limitations in participant recruitment must be anticipated, clearly specified in the application, and credible solutions offered. Since participants may reflect only a small percentage of those available, access to larger populations (often from more than one institution or through more than one mechanism) must be clearly specified. Evidence of prior success (e.g., percent who participated) using the proposed recruitment mechanisms can be convincing; otherwise letters of agreement to participate are needed from institutions offering their enrollees. Participant confidentiality must be addressed and carefully protected.

Some game researchers are tempted to use intuitively obvious single question indicators of whether their game had the desired effect(s). Reviewers will be concerned that these types of measures give inaccurate (i.e., not valid) responses. Validated multi-item scales of the game-related (e.g., immersion,²⁷ sense of challenge²⁸) and behavioral theory (e.g., self efficacy,^29,30 intrinsic motivation^31,32) mediating variables are critical. Objective (i.e., nonself reported) measures of the behaviors are de rigueur (e.g., accelerometer measures of physical activity, sedentary behavior, and sleep³³). Measures of all the variables in the pathway from game play to behavioral and health outcomes should be measured both to ensure the games are having the desired gaming effects (e.g., immersion) and the hypothesized pathways of effects account for the outcomes.³⁴

Guidelines for conducting systematic reviews of the literature are usually seriously concerned with assessing the qualities of the included studies, which have been labeled “study bias.” Specific lists of biases have been generated for each type of research.³⁵ Grant writers should select the most appropriate bias/quality of research check list and state how their proposed research minimizes each source of bias. Not doing this runs the risk of minimizing the contribution of the proposed research to systematic reviews and meta-analyses. What's the point of doing research if future research synthesizers don't take the findings seriously?

Process evaluation includes assessment of dosage and fidelity of the intervention delivered.³⁶ Since games provide a programmed intervention, fidelity of the intervention to that prespecified will not likely be a concern, but needs to be specifically pointed out. Dosage, alternatively, is important and usually assessed by keeping a digital record of each player's selections in game play. Key indicator variables will need to be generated from this detailed digital record.

Research process concerns the sequence of participant activities from baseline assessment through reception of intervention, gameplay, intervention activities between gameplay episodes, and through postassessment. Providing this information assures reviewers that there will be minimal burden and minimal difficulty managing participation: “Implementation seems to be big, such as specific details about how exactly things are going to happen. I find I often unconsciously think something is so obvious that it doesn't occur to me to explain it, but of course someone unfamiliar with games would not have that background knowledge. Getting non-game folks to read the proposal before submission is really important.”

At the very end of methods should be a brief paragraph with paragraph head labeled Significance. This should remind the reviewer of the importance of conducing the proposed research.

Writing style

Reviewers often have to review 10 or more applications in a relatively short period of time (e.g., 2–4 weeks). They like simple, easy to read applications that clearly present all the information they need to judge the importance of a grant application. Respect the reviewers' time and attention span: use active voice; use no more words than absolutely necessary. Involve a professional writer if your institution offers one. “Communicate rigor using tables, bullets, charts, and figures that show important parts of study flow. Be super systematic. Put a table of behavior change techniques, theoretical constructs they target, and specific strategies for implementation in all approach sections.”

Self-assessment

Before submitting a grant application, be very critical of what you have just produced. Remember that reviewers are looking for reasons not to fund you. “As a reviewer I've mostly dinged game and gamification studies for study design, such as incomplete power analyses, not enough information on how feasibility and acceptability will be measured, and not enough information on theoretical mediators and how they would be measured.” If a key issue is not addressed or not stated in an intelligible way, the reviewers will likely assume that you do not know it and not give you the benefit of the doubt. Some of us review our own applications 10 or more times, each review attempting to find and address newly identified limitations: “... think from the reviewers' standpoint, and have some type of mock reviews before submission.” Finding a colleague who does research in your area and requesting they provide a thorough critique of your application is an important step (you may have to give them a financial incentive to enable them to give this their attention that it deserves). Doing this requires producing drafts of the application well in advance of submission deadlines.

Institution(s)

Some grant funding agencies request a reviewer's assessment of the institution from which the grant was submitted, especially the resources (e.g., other faculty, research and grant management support staff, physical facilities, hardware, and software) available at the institution to support the proposed research. This section will need to include the institutional resources for game development and testing. If an applicant is working with a game development and/or testing company outside the institution, then the relevant resources of this company need to be presented. Sufficiently detailed resource information needs to instill confidence in the reviewer that all aspects of the proposed research can be accomplished at that and collaborating institutions. For example, “I've gotten some pushback about mentors/collaborators not being on the same campus with me, which will be a problem since games for health researchers are typically not surrounded by a ton of other games for health researchers. Showing previous publications and projects together seems to have helped with that, though.”

Human subjects protection

Even though the research we generally do is low risk of harm to the participants, there usually are possible small risks of violation of privacy, health problems from games (e.g., epilepsy from TV screens), exergames (e.g., injury), and other health problems from other kinds of games. This section needs to thoroughly review the possible risks and provide realistic solutions to minimize the risks. Sometimes it helps to get assistance to write this section from relevant institutional staff: “I got help from our clinical and translational science award ethics group, and actually had a reviewer comment glowingly about my human subjects section.”

Summary

So there's the nuts and bolts of how we suggest games research grant application should be designed and written. We may have missed a couple of key points, but not many. We all have something to learn from one another. Hopefully, “This discussion might help promote more international collaborations which could be mutually beneficial.”

Sometimes the usual research funding sources may not be the best for research on games: “Personally, my most successful games and highest quality research has come from non-traditional sources (not NIH and not EU funds). I like to partner with organizations that value quality research and have the ambition and means to put their game out in the world to make a difference.” If you happen upon such institutions, take advantage of them!

“However, as a reviewer, I can tell you, without reservation, that reviewers put a lot of time and effort into reviewing applications and try to be fair in their review of these types of applications. They are looking for the best research that is attempting to move the field forward. So, I challenge us, let's be innovative, cutting edge, and focus on the potential scientific significance of the research we propose. How can we build on what's been done, expand on it, address the gaps, and potentially make a difference in the lives of the groups we are trying to reach? Let's propose the best, most cutting edge research that builds on, extends, and addresses the gaps in the current serious games research. Do not give up, but strive to propose stronger, better, cutting edge research that has the potential for making a difference in people's lives.” Amen!

References

Patrick

, Hekler

, Estrin

, et al. The pace of technologic change: Implications for digital health behavior intervention research. Am J Prev Med, 2016; 51:816–824.

Kato

. Evaluating efficacy and validating games for health. Games Health J, 2012; 1:74–76.

Kato

. The role of the researcher in making serious games for health. In: Information Resources Management Association, ed. Human Rights and Ethics: Concepts, Methodologies, Tools, and Applications. Hershey, PA: Information Science Reference; 2014, pp. 1329–;1347.

Brand

, Beltran

, Buday

, et al. Prose fiction as a narrative companion for a vegetable parenting videogame. Games Health J, 2015; 4:305–311.

Brand

, Beltran

, Buday

, et al. Training vegetable parenting practices through a mobile game: Iterative qualitative alpha test. JMIR Serious Games, 2015; 3:e6.

Baranowski

, Beltran

, Chen

, et al. Structure of corrective feedback for selection of ineffective vegetable parenting practices for use in a simulation videogame. Games Health J, 2013; 2:29–33.

Leon

, Davis

, Kraemer

. The role and interpretation of pilot studies in clinical research. J Psychiatr Res, 2011; 45:626–629.

Arain

, Campbell

, Cooper

, et al. What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Med Res Methodol, 2010; 10:67.

Kraemer

, Mintz

, Noda

, et al. Caution regarding the use of pilot studies to guide power calculations for study proposals. Arch Gen Psychiatry, 2006; 63:484–489.

10.

Bowen

, Kreuter

, Spring

, et al. How we design feasibility studies. Am J Prev Med, 2009; 36:452–457.

11.

Stevens

, Taber

, Murray

, et al. Advances and controversies in the design of obesity prevention trials. Obesity (Silver Spring), 2007; 15:2163–2170.

12.

Jago

, Sebire

. Publishing pilot and feasibility evaluations of behavioural interventions: Implications for preventive medicine. Prev Med, 2012; 55:548–549.

13.

Eldridge

, Lancaster

, Campbell

, et al. Defining feasibility and pilot studies in preparation for randomised controlled trials: Development of a conceptual framework. PLoS One, 2016; 11:e0150205.

14.

Eldridge

, Chan

, Campbell

, et al. CONSORT 2010 statement: Extension to randomised pilot and feasibility trials. BMJ, 2016; 355:i5239.

15.

Lazem

, Webster

, Holmes

, et al. Games and diabetes: A review investigating theoretical frameworks, evaluation methodologies, and opportunities for design grounded in learning theories. J Diabetes Sci Technol, 2015; 10:447–452.

16.

Riley

, Glasgow

, Etheredge

, et al. Rapid, responsive, relevant (R3) research: A call for a rapid learning health research enterprise. Clin Transl Med, 2013; 2:10.

17.

, Baranowski

, Thompson

, et al. Story immersion of video games for youth health promotion: A review of literature. Games Health J, 2012; 1:199–204.

18.

Mellecker

, Lyons

, Baranowski

. Disentangling fun and enjoyment in exergames using a expanded design, play, experience framework: A narrative review. Games Health J, 2013; 2:142–149.

19.

Boendermaker

, Peeters

, Prins

PJM

, et al. Using serious games to (re)train cognition in adolescents. In: Ma

, Oikonomou

, Jain

, eds. Serious Games and Edutainment Applications: Volume II. United Kingdom: Springer-Verlay; 2017 [In Press].

20.

Thompson

, Bhatt

, Vazquez

, et al. Creating action plans in a serious video game increases and maintains child fruit-vegetable intake: A randomized controlled trial. Int J Behav Nutr Phys Act, 2015; 12:39.

21.

Collins

, Murphy

, Bierman

. A conceptual framework for adaptive preventive interventions. Prev Sci, 2004; 5:185–196.

22.

Collins

, Murphy

, Strecher

. The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): New methods for more potent eHealth interventions. Am J Prev Med, 2007; 32(5 Suppl):S112–S118.

23.

Collins

, Murphy

, Nair

, et al. A strategy for optimizing and evaluating behavioral interventions. Ann Behav Med, 2005; 30:65–73.

24.

Pellegrini

, Hoffman

, Collins

, et al. Optimization of remotely delivered intensive lifestyle treatment for obesity using the Multiphase Optimization Strategy: Opt-IN study protocol. Contemp Clin Trials, 2014; 38:251–259.

25.

Baranowski

, Baranowski

, Thompson

, et al. Behavioral science in video games for children's diet and physical activity change: Key research needs. J Diabetes Sci Technol, 2011; 5:229–233.

26.

Baranowski

, Baranowski

, Thompson

, et al. Video game play, child diet, and physical activity behavior change: A randomized clinical trial. Am J Prev Med, 2011; 40:33–38.

27.

, Thompson

, Baranowski

, et al. Story immersion in a health video game for child obesity prevention. Games Health J, 2012; 1:37–44.

28.

Staiano

, Abraham

, Calvert

. Motivating effects of cooperative exergame play for overweight and obese adolescents. J Diabetes Sci Technol, 2012; 6:812–819.

29.

Staiano

, Abraham

, Calvert

. Adolescent exergame play for weight loss and psychosocial improvement: A controlled physical activity intervention. Obesity (Silver Spring), 2013; 21:598–601.

30.

Gao

, Zhang

, Stodden

. Children's physical activity levels and psychological correlates in interactive dance versus aerobic dance. J Sport Health Sci, 2013; 2:146–151.

31.

Lyons

, Baranowski

, Basen-Engquist

, et al. Testing the effects of narrative and play on physical activity among breast cancer survivors using mobile apps: Study protocol for a randomized controlled trial. BMC Cancer, 2016; 16:202.

32.

Gao

, Podlog

, Huang

. Associations among children's situational motivation, physical activity participation, and enjoyment in an active dance video game. J Sport Health Sci, 2013; 2:122–128.

33.

Staiano

, Beyl

, Hsia

, et al. Twelve weeks of dance exergaming in overweight and obese adolescent girls: Transfer effects on physical activity, screen-time, and self-efficacy. J Sport Health Sci, 2016. https://dx-doi-org.web.bisu.edu.cn/10.1016/j.jshs.2016.11.005 (accessed December 20, 2016).

34.

Cerin

, Barnett

, Baranowski

. Testing theories of dietary behavior change in youth using the mediating variable model with intervention programs. J Nutr Educ Behav, 2009; 41:309–318.

35.

Lachat

, Hawwash

, Ocke

, et al. Strengthening the Reporting of Observational Studies in Epidemiology—nutritional epidemiology (STROBE-nut): An extension of the STROBE statement. Nutr Bull, 2016; 41:240–251.

36.

Davis

, Baranowski

, Resnicow

, et al. Gimme 5 fruit and vegetables for fun and health: Process evaluation. Health Educ Behav, 2000; 27:167–176.