Strategies for recruiting participants into randomized controlled trials: A cross-program profile of the PROMISE demonstration program

1.1 Challenges to recruiting participants to field-based randomized clinical trials

For field-based randomized controlled trials to yield valid inferences of intervention effectiveness and generalizability, they must meet enrollment targets determined by appropriate power analyses on time and within budget. However, recruiting participants into trials is difficult. Although the research literature on recruitment into experimental and quasi-experimental studies is broad, when considered across areas of clinical problems, settings, and service types, it lacks depth with very few strategies determined with high certainty as effective (e.g., McDonald, 2006; Sully, 2013). For example, a recent Cochrane Collaboration meta-analysis of methods to increase recruitment into randomized and quasi-experimental trials (Treweek et al., 2018) quantified effectiveness of various recruitment strategies reported in 68 studies conducted in a variety of health care settings involving 72 comparisons with more than 74,000 participants. Unfortunately, for only two strategies did they find high-certainty evidence of effectiveness: (a) clear description of all trial procedures in initial mailings to potential participants; and (b) telephoning (or text messaging) potential participants who did not respond to initial mailings. Surprisingly, high-certainty evidence of ineffectiveness was found for the content, format, and appearance of user-tested information leaflets. Disappointingly, a small proportion of other strategies showed only moderate-certainty evidence of effectiveness (e.g., how potential participants process information, financial incentives), and none of these strategies could be recommended with confidence for use in future trials.

Consequently, faced with uncertainty about which recruitment strategies to use in field-based randomized controlled trials, researchers may be tempted to select strategies on a “trial and error” basis, which risks wasting time, money, and resources. Worse yet, some researchers may not fully appreciate the difficulty of trial recruitment, and therefore allocate insufficient resources to meet enrollment targets (e.g. Lemons, Fuchs, Gilbert & Fuchs, 2014).

In this paper, using descriptive data extracted from Mathematica’s process analysis reports for each of the six PROMISE programs, we profile these programs’ recruitment strategies. We hope to present researchers in transition, secondary education, and vocational rehabilitation with some suggestions of feasible strategies for meeting enrollment targets in future trials of field-based interventions for transition-aged youth with disabilities.

2.1 Recruitment and enrollment data sources

Mathematica, the independent evaluator of the six PROMISE programs, produced interim process analysis reports for each of the six PROMISE programs, summarizing program service implementation; demographic and clinical profiles of enrolled youth; and recruitment and enrollment outcomes. Mathematica’s findings rested upon data derived from quantifiable recruitment strategies recorded by designated program staff into electronic data bases, and from telephone interviews and visits with program staff: Arkansas (Honeycutt et al., 2018), ASPIRE Consortium (Anderson et al., 2018), California (Matulewicz et al., 2018), Maryland (Kauff et al., 2018), New York State (McCutcheon et al., 2018), and Wisconsin (Selekman et al., 2018). The reports present (a) the type, frequency, and percentage of each program’s quantifiable recruitment strategies, (b) the quarterly rates of youth enrollment across the two-year recruitment period, (c) the number of contacts needed to enroll one youth calculated in two ways—by overall average and by four rank-ordered categories; and (d) narrative descriptions about recruitment strategies unique to each program.

2.2 Characteristics of the target population

A total of 13,172 youth and families enrolled across the six PROMISE programs. Because demographic and clinical characteristics of enrollees were similar across the six programs, we summarize the following characteristics in the aggregate. Age: About 40 percent of youth were age 14, except the Maryland program (25%). Gender: Approximately two-thirds of enrolled youth were male, and one-third female. Race/ethnicity: SSA discourages researchers from analyzing the race variable in its SSI and other administrative data systems, because after 2002, it discontinued publication of data by race after altering its method for assigning new SSNs. In Mathematica’s PROMISE program process evaluation reports, about two-thirds of enrolled participants are identified as “other/unknown,” which may reflect individuals identifying as biracial or multiracial. Poverty: The study’s selection criteria of current SSI beneficiary indicates that youth and families had household incomes at or below the federal poverty level. Urbanicity: Complete residential profiles of the six programs’ enrolled participants were not available to us.

Types of disabilities: Mathematica recorded primary disabling conditions according to the SSA’s Listing of Impairments, and combined them into five categories listed in order of decreasing frequency: (a) intellectual or developmental disabilities, (b) other mental impairments, together accounting for 75 percent of enrolled youth. Remaining categories included (c) physical disabilities, (d) speech, hearing, or visual impairments, and (e) other. Primary Spoken Language: More than 85 percent of enrolled youth identified English as their preferred language, except for the California program (65%), and Spanish as the second most commonly preferred language.

2.3 Recruiter competencies with reasonable accommodations and cultural sensitivity

All programs trained and ensured staff involved in recruiting and obtaining consent and assent from youth and parents/guardians, were skilled with accommodating those with: (a) language barriers (e.g., low levels of literacy; languages other than English); (b) mental impairments (e.g., conduct disorders); (c) physical disabilities (e.g., spinal cord injuries); (d) sensory impairments (e.g., vision and hearing loss); and (e) cognitive impairments (e.g., borderline intellectual functioning). Similarly, all programs trained and ensured staff manifested cultural sensitivity as indicated by understanding one’s own culture and biases; accurately appraising the cultures of people living in one’s recruitment area; and learning how to respond to cultural beliefs that might impede recruiting and collecting data from youth and parents/guardians.

2.4 Strategies for recruiting, enrolling, and obtaining consent/assent

Youth and parents/guardians entered PROMISE programs in one of two ways: (a) they enrolled in response to invitations from direct mailings, telephone calls, or other program solicitations, or (b) in hearing about the study program in other ways, they initiated contact with program staff. Initial program contacts may have involved youth and their parents/guardians separately or together, depending on which specific recruitment method they encountered first. Each of the program’s research teams developed its own recruitment and enrollment plans, which included mixes of initial and follow-up mailings, telephone calls, emails, text messages, and in-person meetings.

3.1 Recruitment and enrollment rates by PROMISE program

All six programs met the PROMISE cooperative agreement’s enrollment requirement within the two-year recruitment window. On all metrics of recruitment and enrollment, the programs varied considerably (Table 1). First, the cumulative number of all eligible youth on the SSA lists ranged from a low of 7,828 (Maryland) to a high of 21,939 (California). Second, the percentage of eligible youth specifically targeted for recruitment ranged from a low of 51.4 percent (California, 11,271 out of 21,939) to a high of 92.6 percent (Wisconsin, 8,570 out of 9,249). Third, the percentage of all eligible youth from the SSA lists who ultimately enrolled into the study ranged from a low of 10.3 percent (New York State, 2,090 out of 20,290) to a high of 25.6 percent (Maryland, 2,006 out of 7,828), and fourth, the percentage of eligible youth specifically targeted for recruitment who enrolled into the study ranged from a low of 22.3 percent (ASPIRE Consortium, 2,051 out of 9,196) to a high of 43.2 percent (Maryland, 2,006 out of 4,644). On a geographic basis, the ASPIRE Consortium, Maryland, and Wisconsin programs recruited statewide, Arkansas recruited from four counties, and California and New York State recruited from local education agencies located in four and three regions, respectively.

Mathematica reported the number of contacts made to targeted eligible youth in two ways: (a) average number of contacts per youth by enrollment status (yes/no) (Table 3), and (b) number of contacts per youth by four rank-ordered categories (one, two to five, six to ten, and eleven or more) by enrollment status (yes/no) (Table 4). Across the six programs, the average number of contacts to targeted eligible youth who enrolled ranged from a low of 2.5 (Arkansas) to a high of 6.2 (New York State); the average number of contacts for youth who did not enroll also ranged from a low of 2.5 (Arkansas) to a high of 6.2 (Maryland) (Table 3). By rank-ordered categories, the modal number of contacts to targeted eligible youth for five of six programs was two to five (Arkansas, 71.5%, ASPIRE Consortium, 76.1%, California, 63.3%, Maryland, 53.3%, Wisconsin, 43.2%), and six to ten for one program (New York State, 41.2%) (Table 4). For five of six programs, the bulk of youth (enrolled or not) were contacted between two and ten times (77% to 96%). Only two programs reported eleven or more contacts for more than a few percent of youth (Table 4). A very low percentage of eligible youth recruited and enrolled were contacted only once: less than ten percent for three programs (ASPIRE Consortium, Maryland, New York State), and between 20 and 41 percent for the other three programs (Arkansas, California, and Wisconsin) (Table 4). Similarly, a very low percentage of eligible youth recruited but not enrolled were contacted only once.

3.3 Mix of recruitment strategies by PROMISE program

All programs used different mixes of the six quantifiable strategies to recruit eligible youth from their SSA lists (Table 2). Five of six programs reported sending an initial mailing to more than 90 percent of eligible youth, inclusive of those who enrolled and those who did not. Two states (California, Maryland) reported that percentages of youth and families receiving an initial mailing enrolled at considerably lower rates, in both absolute and relative (California, 55%, Maryland, 89%) terms, compared to those who did not enroll (California, 71%, Maryland, 99%), indicating that multiple contacts of differing strategies would be necessary to meet enrollment targets (Table 2). Follow-up mailings varied sharply across the programs ranging from less than 10 percent (ASPIRE Consortium) to greater than 50 percent (Arkansas, New York State, Wisconsin). For all six states, youth and families receiving these follow-up mailings enrolled at considerably lower rates, both in absolute and relative terms, compared to those who did not enroll (Table 2). All programs relied heavily on telephone calls; all exceeded 50 percent and three exceeded 80 percent inclusive of those who enrolled and those who did not (ASPIRE Consortium, Maryland, New York State). Email was used infrequently by all programs (less than 5%). Text messages were used only by two programs (Maryland, Wisconsin). In-person meetings between one or more project staff members and youth and families who eventually enrolled in the study clustered into two groups: relatively lower absolute and relative (less than 17%, Arkansas, New York State, Wisconsin) frequency, and relatively higher absolute and relative (ASPIRE Consortium, 100%, California, 49.1%, Maryland, 62.2%) frequency (Table 2). For in-person meetings between project staff members and youth and families in these two clusters of three states who did not enroll, absolute and relative rates of meeting in person were lower, but similar in pattern (Table 2).

Three programs offered youth and families a financial incentive to enroll. Maryland offered both the youth and the parent/guardian a gift card for signing assent and consent forms, respectively. Wisconsin provided two gift cards to the youth, the first upon completing all enrollment forms, and the second upon completing the baseline survey. ASPIRE provided a gift card to each youth upon attending its required in-person enrollment meeting.