Abstract
Abstract
Objective:
The most productive methods of recruitment for a videogame for health (G4H) trial are not known. Success or failure of recruitment methods has been reported for a variety of clinical trials, but few specifically for G4H trials. This study's goal was to recruit 444 overweight or obese (body mass index percentile between the 84.5th–99.4th percentiles) children between the ages of 10–12 years. The article reports the results of different methods of participant recruitment.
Materials and Methods:
Participants had to agree to three fasting blood samples (baseline, immediately after, and 2 months later); be willing to wear an accelerometer for 7 days at each assessment; read and speak English fluently (because the games were in English); have no history of any condition that would affect what he/she could eat or how much physical activity he/she could get; and have an eligible home computer purchased in the last 5 years with high-speed internet. Hardware criteria reflected the types of computers upon which Diab-Nano could be effectively played. Recruitment was conducted over a 35-month period and included electronic media, print advertising, community recruitment, and an internal volunteer list. Respondents were guided to a web-based screening questionnaire that asked for source of hearing about the study.
Results:
Although diverse recruitment methods were used, slow recruitment resulted in obtaining only 45% of the recruitment goal (n = 199). Electronic media (e.g., radio, television, and internet), which reached millions of targeted parents, resulted in only 76 respondents, of whom 13 became participants; print media (e.g., magazine, newsletter/newspaper, and mail), which also reached large numbers of parents, resulted in 192 respondents, of whom 19 became participants; community recruitment (e.g., school, friend or family, doctors office, flyer, work, community program) resulted in 162 respondents, of whom 38 became participants; and the internal volunteer list resulted in 413 respondents, of whom 128 became participants.
Conclusion:
Inclusionary and exclusionary criteria and restricted access by gatekeepers could be deterrents to successful recruitment. The documented payoff of alternative comprehensive methods of recruitment should benefit other investigations in optimally allocating their recruitment resources.
Background
R
Different recruitment methods likely reach different populations and can be divided into focused recruitment campaigns that attempt to reach specific groups (e.g., members of an organization, business, and school) or those that attempt to reach a broader group throughout a community (e.g., parents of 10–12 year olds). 4 Methods can be further categorized as active or passive recruitment. 5 Broader campaigns with passive recruitment include electronic (television, radio, and e-mail) and print media (magazines, newspapers, and newsletters) that target larger less differentiated audiences. Active campaigns reach members of specific community groups (e.g., schools, community organizations, and businesses) by speaking at events or setting up information booths. While many channels exist to reach broader or more specific groups, which channels yield the greater numbers of participants is not clear. This article describes the recruitment methods for one community trial of two games for health (G4H) and reports the payoff in terms of participants from each source.
Materials and Methods
Participants were recruited for the outcome evaluation of two videogames “Escape from Diab” and “Nanoswarm: Invasion from Inner Space” (herein after referred to as G4H), designed to be played in sequence over a 2–3 month period for diabetes and obesity prevention. 6 The recruitment goal for this randomized clinical trial was 444 children between the ages of 10–12 years old with a body mass index (BMI) percentile between the 84.5th–99.4th percentiles. This age group was selected because the two videogames were designed for and tested with 10–12-year olds in a pilot study. 6
Since fasting insulin was the primary dependent variable, overweight or obese participants were desired, who would have elevated values, but not be so obese as to not be able to benefit from a low intensity behavior change intervention. Participants had to agree to three fasting blood samples (baseline, immediately after, and 2 months later); be willing to wear an accelerometer for 7 days at each assessment; read and speak English fluently (because the games were in English); have no history of any condition that would affect what he/she could eat or how much physical activity he/she could get; and have an eligible home computer with high-speed internet, that is, an operating system of the following: Microsoft Windows XP (SP3), Windows Vista (SP2), Windows 7 (SP1), and Windows 8 or 8.1; a processor equal to the following: 2.13 GHz Intel Core 2 Duo E6400 or 2.8 GHz AMD Athlon 64 X2 5600+ minimum; system memory of 2Gbytes RAM or higher and DirectXRuntimes (October 2006 version or newer); screen resolution with a minimum of 1280 × 800 and minimum 10Gbytes free space on the hard drive; and DVD optical drive or USB port for installation and controller hook up and sound card with speakers or headphone jack; and the computer must be less than 5 years old. These hardware criteria reflect the types of computers upon which Diab-Nano 6 could be effectively played.
For purposes of clarity in the ensuing statements, we use the term “respondent” for any person who used our website, but did not participate in the actual study, and the term “participant” for any person (n = 199) who actually participated in the study.
Respondents interested in participating in this study had to complete an eligibility questionnaire located on the Children's Nutrition Research Center's website or through phone call to the G4H recruiter. Interested respondents for whom we had contact information received emails that contained a link directing them to the eligibility and computer requirement questionnaires. If respondents were nonresponsive to initial e-mails, follow-up phone calls were made. After preliminary self-reported eligibility was determined, the child and guardian were scheduled for their first assessment to objectively verify the adiposity inclusionary criterion (e.g., height and weight assessment) by G4H staff.
Recruitment materials
Recruitment began in May 2014 and closed in March 2017, that is, 35 months. All G4H recruitment procedures and advertising materials (e.g., artwork and wording) were approved by Baylor College of Medicine IRB. Gameplay flyers were created with engaging artwork and wording. Tab flyers (i.e., pull tabs at the bottom of a page with project name and number to tear off and call), non-tab flyers, and packets were made to be distributed to businesses in the community (e.g., five non-tabbed flyers, two tabbed flyers, and a business card). Informational e-mails were created for distribution to gatekeepers (e.g., leaders in local businesses, schools, camps, and libraries) and posters and games (e.g., spin the wheel and identify objects for a healthy plate) were created to draw attention from parents and children when attending child-friendly community events.
Diverse channels were employed to reach as many respondents and participants as possible to meet the project's recruitment goals. Low numbers motivated continued search for new channels to reach new candidates or reach them in new ways.
Electronic media advertisement
Electronic media advertisements reached the largest numbers of people. A professionally recorded 15-second radio advertisement was run for 5 weeks with 25 advertisements per week during prime commute time for parents taking children to school and traveling to and from work in September 2014. This radio station was selected because it aired to a nine-county metropolitan area, reaching ∼128,600 women between the ages of 25–49 (by the station's estimates from a marketing firm) who potentially had children in the 10–12 year-old range. This station also sent one eDirect e-mail blast in September 2014 to opted-in e-mail addresses (i.e., customers who had given permission to this marketing company to send messages and information) meeting the demographic criteria sought for the G4H study. The eDirect eBlast was sent to 148,000 people per deployment.
A professionally made 30-second television campaign was created in February 2015 using IRB-approved video clips from the Escape from Diab videogame, text from print advertisements, and a professionally recorded voice over. This advertisement was aired eight times per week for 2 weeks on Lifetime, Disney, and ABC Family channels in the Houston area.
An online advertisement was run with on a local newspaper website in May 2015 on their lifestyle pages guaranteed to be viewed 50,000 times during its 30-day run period. They also sent an eBlast in May 2015 containing the G4H information with their Spring Bazaars and Festivals e-mail newspaper edition, which was sent to 180,000 email addresses that opted-in to receive their emails.
E-mail blasts were also distributed by a second media group on three different occasions in November 2015 to 150,000 opted-in consumers equaling 450,000 sent emails.
Print media advertisements
Print advertisements reached the next largest number of people. An advertisement was created with a trusted family publication and a resource guide, which monthly distributed magazines at ∼5500 locations within the central, north, and western regions of the metropolitan Houston area. Advertisements were run in September, November, and December of 2014 and monthly from March to November of 2015, and again March 2016 to December 2017. Apart from the print advertisements, the G4H recruitment team spent time volunteering for this publication at The Children's Festival in November 2014 and March 2015, distributing G4H flyers in goodie bags.
A 1/6 page advertisement was published in a community magazine in January 2015, which was delivered to 14,500 homes in an area with families likely to have children in the 10–12 year-old range and also have computers that would meet the Diab-Nano requirements.
In February 2015, a ¼ page advertisement was run in another family publication distributed to ∼60,000 families across 5 counties in the Houston metropolitan area.
In the May/June 2015, a ½ page advertisement and ½ page editorial about video game studies were run in another magazine marketed to moms specifically.
Internal volunteer list
Information about joining the G4H study was communicated to families who received the Children's Nutrition Research Center's (CNRC) quarterly “Nutrition and Your Child” newsletter mailing list (i.e., 1650 email copies and 2500 hard copies), and communicated by the main CNRC recruiter directly to families in the CNRC's volunteer list. The volunteer list included families who previously participated in research studies at the CNRC or had indicated interest in studies and wanted to be contacted for upcoming opportunities. The CNRC recruiter identified likely candidates (e.g., child age), gave the parent a call, explained the study, and offered participation. Since there was a 35-month recruitment window, the recruiter serially added names to the contact list, based on children passing the minimum inclusionary age criterion each month.
Community recruitment
Building relationships with children's recreation facilities near the Texas Medical Center consumed a substantial amount of recruitment time early in 2014. The recruitment team spent many afternoons and weekends at the local children's museum, the health museum, and the natural science museum's branches in the city and a suburb. In addition to actively recruiting at these facility's summer camps, the G4H recruitment team distributed flyers at a local university's summer 2014 Science Technology Engineering and Mathematics camp. Local Ys (formerly called YMCA, or Young Men's Christian Association) accepted flyers to hang in their facilities and invited the recruitment team to attend their children's festival in April 2015 to set up a booth about the study and making healthy choices. The recruitment team prepared for the Midway and Conversation Corner at the Boy Scout Fair in April 2015. By participating in the G4H Study, if eligible, scouts would be able to earn a game design badge.
About 218 public elementary and middle schools in the Greater Houston Area were contacted with information about the G4H study in the August 2014, January 2015, and August 2015. Only nine schools allowed the recruitment team to either participate in a science night or bring flyers to be passed out to fifth and sixth graders in their homework folders. About 43 charter elementary and middle schools were contacted; 10 requested informational flyers to pass out in fifth and sixth grade homework folders and 6 schools invited the recruitment team to set up a table at their science night. Twenty-eight Home School organizations were contacted: one home school organization distributed G4H information to their 4000 families in the Houston area in August 2015.
Partnering with businesses in Houston and surrounding areas to display information flyers consumed another large portion of recruitment time. Greater Houston was split into 10 regions and placed on a 10-week rotation to visit businesses and continuously distribute study flyers. During a 10-week period, 318 businesses were visited and either 2 tabbed flyers or a packet of 5 non-tabbed flyers and 2 tabbed flyers were left with them.
Analysis
Table 1 indicates the number of respondents or participants who came each quarter from each recruitment source. Respondents or participants on the web-based screening questionnaire selected how they heard about the study as they completed their eligibility questionnaire; the numbers per category in Table 1 reflects those selections. These categories were created at the beginning of the study to be generic since the specific recruitment channels (e.g., museums, festivals, and community events) were unknown during the creation of the questionnaire.
Internet, for example, email, listserv, website.
Newsletter, newspaper advertisement, or story.
Mail, for example, letter, flyer, postcard, brochure.
School, for example, flyer, word of mouth, presentation.
Other community program, for example, flyer, word of mouth, presentation.
Other—did not disclose source.
CNRC, Children's Nutrition Research Center's.
Table 2 identifies the disposition of each contact from each recruitment source over the full 35 months. The disposition categories included why they were ineligible, or if they were pending qualification due to lack of completing the computer questionnaire. If the child's BMI fell below the minimum necessary 84.5%, they were placed into the BMI <85 category, and if their BMI was above the maximum allowed 99.4%, they were placed in the BMI >99 category. If they had no home computer or their age fell out of the 10–12 year-old range, they were also disqualified. Preexisting conditions that would disqualify a child included attention-deficit hyperactivity disorder, food allergies, diabetes, autism, seizures, severe asthma, cerebral palsy, or cardiac or gastrointestinal conditions: celiac disease and irritable bowel syndrome.
Internet, for example, e-mail, listserv, website.
Magazine publications.
Newsletter, newspaper advertisement, or story.
Mail, for example, letter, flyer, postcard, brochure.
CNRC recruiter.
School, for example, flyer, word of mouth, presentation.
Friend or other family member.
Other community program, for example, flyer, word of mouth, presentation.
Other—did not disclose source.
BMI, body mass index.
The category “other” includes any disqualifying criterion not previously listed, including a sibling being in the study, being on a swim team (and not being able to wear an accelerometer for all day), or already participating in another CNRC study. “No qualify computer questionnaire” or “pending computer questionnaire” identify how many people did not have computers compatible with the game requirements. Many respondents received the computer questionnaire e-mail link, but never completed it. “Pending assessments” mean that the participant qualified for the study upon completing the eligibility questionnaire, but did not complete the baseline assessment. “Not qualified by assessment” and “qualified assessment” were final decisions based on their baseline data collection. The reason that most children were not qualified for participation was being outside of our selected BMI percentile range (84.5%–99.4%ile). These numbers could be inaccurate as the child's initially reported height and weight were all self-reported over the phone or online during the eligibility questionnaire, but later assessed and verified at baseline data collection.
Results
Media advertisement
Although electronic media (radio advertisements, eBlasts, television advertisements, and newspaper websites) reached very large numbers of people in the community many times, they were not productive of interested respondents or participants. We had 76 total respondents (i.e., 76 people completed the web-based survey), but of those, only 13 became participants. The eBlast created by one media group was sent to 148,000 opted-in e-mails meeting the demographic criteria and experienced 10,689 opens and 824 clicks on content links, but resulted in 0 participants.
Another e-mail blast deployed on three different occasions to each of 150,000 opted-in email addresses meeting our demographics resulted in 23,645 opens and 1218 click-throughs on the first deployment; another 27,886 opens and 1469 click-throughs on the second deployment; and 25,507 opens and 1703 click-throughs on the third deployment, but no participants. During the days following each deployment, hundreds of eligibility questionnaires were opened, but none was completed. This response was likely due to the e-mail blast, but because none was completed, we have no record of where they heard about the study.
Print advertisement
Monthly advertisements in a trusted family publication resulted in 159 respondents completing the eligibility questionnaire, of whom 16 qualified at baseline data collection; the community magazine had 0 respondents; the family magazine had 5 respondents complete the eligibility questionnaire; the last magazine directed at mothers had 0 respondents; 21 respondents received a newsletter or newspaper containing study information, of whom 3 qualified at baseline data collection; and 12 respondents reported receiving study information in the mail.
Internal volunteer list
Our most productive participant recruitment source was the CNRC volunteer list resulting in 413 respondents, of whom 128 qualified at baseline data collection. Most of these families had participated in research at this facility before and were familiar and comfortable with the CNRC and its staff.
Community advertisement
Information given to participants by a friend or family member resulted in 52 respondents, of whom 17 qualified at baseline; information passed out at schools resulted in 37 respondents, of whom 6 qualified at baseline data collection; 13 respondents received information from a doctor's office, of whom 2 qualified; 27 respondents saw study information on a flyer in the community, of whom 4 qualified at baseline data collection; 16 respondents saw study information at their place, of work of whom 4 qualified at baseline data collection; 11 respondents received information from other community programs, of whom 3 qualified at baseline data collection; and 16 respondents received information from other unknown sources, of whom 2 qualified at baseline data collection.
Discussion
While we mounted a comprehensive and diversified recruitment strategy, we recruited only 199 of the desired 444 participants in the possible 35-month period. Our experience was not unprecedented. A systematic review revealed that, “Among 1017 randomized clinical trial protocols from 6 research ethics boards in Switzerland, Germany, and Canada, 253 (24.9%) were discontinued and another half reached a median of only 40.9% of their target enrollment (Median 40.9; IQR = 28.5 to 59.8).” 7 Recruitment for an obesity prevention study utilized an integrated healthcare organization that provided comprehensive medical services to 3.6 million members and sent study information to 4730 randomly selected parents, followed by telephone recruitment calls. Only 7.6% of all parents initially contacted attended baseline assessments and were randomized. 8 Reports from parent participants in an Australian study suggested that they needed multiple exposures to study advertisements and possibly information from multiple sources before making a formal inquiry. 9 Our inability to recruit a desired sample size could have been due to inadequate duration of the recruitment interval, inadequacies in the channels selected for recruitment, or challenges in the stringent inclusionary/exclusionary criteria.
Large sample sizes require substantial time for recruitment. 5 A sample over 100 requires a recruitment duration between 9 and 36 months. 5 Our recruitment lasted 35 months, but the numbers trickled in per month toward the end, suggesting market saturation for purposes of recruitment. Future studies similar to ours may need to recruit in more than one metropolitan area.
In regard to recruitment channels, the CNRC's volunteer list and recruiter capitalized on having direct contact with people who participated in previous research and/or had shown interest in research. Parents gave permission to the CNRC to keep their children's date of birth and estimated BMI's from other studies on file so they could be informed of future studies for which they might qualify. The recruiter was able to identify those who might fit the criteria needed for this study and distributed their contact information to our team recruiter. Limitations of this method include enrolling participants who were not eligible due to their participation in concurrent studies, outdated information self-reported by parents, and the costs of staff to develop and maintain a recruitment database. Unfortunately, not enough children were on the volunteer list who met selection criteria, requiring us to seek other resources. Investment in a volunteer list for research participant recruitment appears to be a wise investment.
Electronic media, in general, and radio advertisements, in particular, were not successful for our study. One study recruited sixty percent of their participants in an obesity prevention study after hearing a one-minute radio advertisement about the program. 1 Another childhood obesity prevention study specifically targeting African-American females utilized radio advertisement and recruited 46% of their participants with this outlet. 4 Their voiceover was read by an African-American female in a “warm, welcoming voice, while soft music played in the background” to set this advertisement apart from others on the radio. 4 Longer radio advertisements may be worth the investment. Interview and live radio opportunities might be tried by future projects instead of the professionally made 30-second advertisements in our study. Research comparisons on effectiveness would be valuable. Investigators will need to contact local media outlets to ascertain local media development and distribution/implementation costs.
Print advertising was successful in other trials, generating the highest number of participants recruited at the most cost-effective recruitment strategy.9,10 Print media in our study generated 192 total respondents with 19 of those being participants (about 10% of our final sample). A value of print media is that, once printed, families can retain the printed resource, repeatedly view it, and have study contact information continuously available. A limitation of print media advertisements for recruitment studies is the inability to determine how many people actually saw the study information. 5 Advertisements in print media varied substantially in cost within the Houston area, likely reflecting size of reach into a market and professionalism of the advertisement.
Schools have been important sources for recruitment for video game research. 11 About 50.9% of parents recruited in one study heard about it from a flyer at their child's school. 1 Unfortunately, schools in our geographic area discourage recruitment for large-scale research, reflecting their priority on instructional time to pass standardized tests. The largest public school district had strict guidelines for participation in research that do not allow outside sources into their schools. Only 9 (of 218) schools in this district allowed us to bring flyers to be placed in their fifth and sixth grade homework folders, or set up a table at their family night.
The second largest school district allowed us to place flyers in homework folders of fourth and fifth graders in 19 out of 47 total elementary schools. Two schools invited us back to set up a table at their health fairs. Because the school principal or nurse was the gatekeeper to this resource, teachers may not have been adequately prepared to answer parents' questions regarding the materials sent home. Most home-school organizations were nonresponsive. Their main concern was their limited meeting time with families and they did not want to divert from this limited time. The primary costs of this recruitment are staff time to make contacts with the multiple levels in the schools (e.g., central administration, local principals, teachers, and attending public sessions), which are usually time-consuming for travel, waiting, and contact times.
Visitors to local community establishments were provided multiple exposures to study information. However, where the flyers were hung, or information booths were displayed, was at the discretion of the program or business and may not have always been in the most ideal spot. Hanging flyers in the community resulted in 67 total respondents with 14 of those being participants. Slow recruitment has been reported because flyers were placed on community boards covered with other flyers, which got lost in the mix, were posted over, or were removed. 12 We were not aware of the extent to which this occurred in our study. Nonresponsiveness left several community resources (e.g., summer/winter camps and churches) untapped. Staff time and transportation were the primary cost components of this method. We pursued this aggressively trying to reach all regions in the city. Investigators can select the intensity of their efforts in this regard, and thereby control this cost, but the payoff appears too low to intensively pursue this.
Protocols utilizing person recruiters were nine times more likely to achieve 50% of target enrollment, 7 and was one of two factors that favored 80% or more of target enrollment. 7 When given the opportunity to talk directly with parents, the level of interest was greatly heightened as they were able to ask questions and better understand the study. Parents seemed interested to learn about the study, but had many concerns. Only after answering their questions and talking through the study participation process did they feel comfortable enough to give their contact information for follow-up. Thus, engineering opportunities for recruiters to directly meet with parent possibilities would seem worth the costs.
Our electronic and print media recruitment channels were chosen because they were used by families meeting our demographic criteria. One explanation for the poor return on investment could be that, by choosing new outlets monthly (e.g., different magazine and different media company) and not regularly repeating advertisements, multiple exposures to potential participants were lost when that 1 month of advertising was up. The one print source that ran advertisements for consecutive months likely created familiarity among potential participants and may have generated an increased interest.
Finally, our inclusionary and exclusionary criteria may have been the largest deterrent to effective recruitment. It is not clear what percentages of possible respondents or participants were deterred by the relatively sophisticated computers needed to run the game. Our productization efforts could only meet a level of high-end graphics and processing. Given the high level of income and education among the participating families, and the large drop from number of people initiating the inclusionary screening questionnaire to those completing it, suggest that the required high level of computing was a deterrent to recruitment, especially among lower income families. Future game research should attempt to build games from the start to be played on the widest variety of digital platforms.
Two reported perceived barriers to successful recruitment of 80% or more of targeted enrollment were “long study duration” and “too restrictive eligibility criteria.” 7 Our study duration (6 months) along with requiring three blood samples also may have caused parents of eligible children to lose interest and never schedule a baseline or follow-up appointment. A marketing company suggested we shorten our consent page since it is the first page that the eligibility questionnaire respondents see, and it may have overwhelmed viewers with the time it took to read. However, this was an IRB approval requirement. Another limitation may have been a lack of knowledge among possible respondents about research and what they might gain from participation. For example, exposure to a short video about a study increased participants' knowledge and improved attitudes before study participation. 13 Future video game studies may consider sending a short video clip to accompany detailed study information to potential participants to create a positive trustworthy connection to the study, and to better inform participants about what the research study involves and what they can expect from participating. Social media should be tested for recruitment in future research 14 ; recruitment videos could be inserted in social media. Intensive interviews should be added to future protocols to assess reasons why dropouts occurred.
Conclusion
The CNRC volunteer list, which contained children interested in participating in research, along with their contact information, facilitated a high level of participation. Print advertisements were the next most consistent for generating interest in study participation, but produced small numbers of participants. Electronic media generated little interest in contrast to past trials. Community family publications should be utilized, since they were lower cost and provided a stable inflow of participants.
Footnotes
Acknowledgments
This work was funded, in part, by federal funds from the National Institutes of Health (DK091254) and is a publication of the United States Department of Agriculture (USDA/ARS) Children's Nutrition Research Center, Department of Pediatrics, Baylor College of Medicine, Houston, Texas, and had been funded, in part, with federal funds from the USDA/ARS under Cooperative Agreement No. 58-3092-5-001.
Author Disclosure Statement
No competing financial interests exist.